This is an automated email from the ASF dual-hosted git repository.
tzulitai pushed a commit to branch release-2.0
in repository https://gitbox.apache.org/repos/asf/flink-statefun.git
The following commit(s) were added to refs/heads/release-2.0 by this push:
new f1c1a7b [release] Adapting the README to Stateful Functions 2.0
f1c1a7b is described below
commit f1c1a7b630a0b404140d51a0eda183fb2ccd9705
Author: Tzu-Li (Gordon) Tai <tzuli...@apache.org>
AuthorDate: Tue Apr 7 21:17:24 2020 +0800
[release] Adapting the README to Stateful Functions 2.0
---
README.md | 134 ++++++++++++++++++++++--------------------
statefun-python-sdk/README.md | 93 +++++------------------------
2 files changed, 84 insertions(+), 143 deletions(-)
diff --git a/README.md b/README.md
index edc6de9..89b4380 100755
--- a/README.md
+++ b/README.md
@@ -1,12 +1,8 @@
-<img alt="Stateful Functions" src="docs/fig/stateful_functions_logo.png"
width=400px/>
+<img alt="Stateful Functions" src="docs/fig/stateful_functions_logo.png"
width=350px/>
-Stateful Functions is an [Apache Flink](https://flink.apache.org/) library for
distributed applications and services, based on, well, you guessed it: stateful
functions.
+Stateful Functions is an [Apache Flink](https://flink.apache.org/) library
that **simplifies building distributed stateful applications**. It's based on
functions with persistent state that can interact dynamically with strong
consistency guarantees.
-The project aims to simplify the development of distributed stateful
applications by solving some of the common
-challenges in those applications: scaling, consistent state management,
reliable interaction between distributed
-services, and resource management.
-
-Stateful Functions uses Apache Flink for distributed coordination, state, and
communication.
+Stateful Functions makes it possible to combine a powerful approach to state
and composition with the elasticity, rapid scaling/scale-to-zero and rolling
upgrade capabilities of FaaS implementations like AWS Lambda and modern
resource orchestration frameworks like Kubernetes. With these features, it
addresses [two of the most cited
shortcomings](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf)
of many FaaS setups today: consistent state and efficient messaging between f
[...]
This README is meant as a brief walkthrough on the core concepts and how to
set things up
to get yourself started with Stateful Functions.
@@ -22,10 +18,11 @@ For code examples, please take a look at the
[examples](statefun-examples/).
- [Core Concepts](#core-concepts)
* [Abstraction](#abstraction)
* [Function modules and extensibility](#modules)
+ * [Runtime](#runtime)
- [Getting Started](#getting-started)
- * [Building the project](#build)
* [Running a full example](#greeter)
* [Project setup](#project-setup)
+ * [Building the Project](#build)
* [Running in the IDE](#ide-harness)
- [Deploying Applications](#deploying)
* [Deploying with a Docker image](#docker)
@@ -38,15 +35,16 @@ For code examples, please take a look at the
[examples](statefun-examples/).
### <a name="abstraction"></a>Abstraction
A Stateful Functions application consists of the following primitives:
stateful functions, ingresses,
-routers, and egresses.
+routers and egresses.
-#### Stateful functions
+<p align="center">
+ <img src="docs/fig/stateful_functions_overview.png" width="650px"/>
+</p>
-* Stateful functions are the building blocks and namesake of the Stateful
Functions framework.
-A function is a small piece of logic (currently simple Java functions) that
are invoked through a message.
+#### Stateful functions
-* Each stateful function exist as uniquely invokable _virtual instances_ of a
_function type_. Each instance
-is addressed by its type, as well as an unique id (a string) within its type.
+* A _stateful function_ is a small piece of logic/code that is invoked through
a message. Each stateful function
+exists as a uniquely invokable _virtual instance_ of a _function type_. Each
instance is addressed by its ``type``, as well as an unique ``ID`` (a string)
within its type.
* Stateful functions may be invoked from ingresses or any other stateful
function (including itself).
The caller simply needs to know the address of the target function.
@@ -61,16 +59,14 @@ at any point in time.
That state is private and local to that instance.
If you know Apache Flink’s DataStream API, you can think of stateful functions
a bit like a lightweight
-`KeyedProcessFunction`. The function type is process function transformation,
while the ID is the key. The difference
-is that functions are not assembled in a directed acyclic graph that defines
the flow of data (the streaming topology),
+`KeyedProcessFunction`. The function ``type`` is the process function
transformation, while the ``ID`` is the key. The difference
+is that functions are not assembled in a Directed Acyclic Graph (DAG) that
defines the flow of data (the streaming topology),
but rather send events arbitrarily to all other functions using addresses.
-<img alt="Stateful Functions Overview"
src="docs/fig/stateful_functions_overview.png" width="600px"/>
-
#### Ingresses and Egresses
* _Ingresses_ are the way that events initially arrive in a Stateful Functions
application.
-Ingresses can be message queues, logs, or HTTP servers - anything that
produces an event to be
+Ingresses can be message queues, logs or HTTP servers — anything that produces
an event to be
handled by the application.
* _Routers_ are attached to ingresses to determine which function instance
should handle an event initially.
@@ -79,10 +75,10 @@ handled by the application.
Egresses are optional; it is also possible that no events leave the
application and functions sink events or
directly make calls to other applications.
-### <a name="modules"></a>Modules and extensibility
+### <a name="modules"></a>Modules
-A _module_ is the entry point for adding to a Stateful Functions
-application the core building block primitives, i.e. ingresses, egresses,
routers, and stateful functions.
+A _module_ is the entry point for adding the core building block primitives to
a Stateful Functions
+application, i.e. ingresses, egresses, routers and stateful functions.
A single application may be a combination of multiple modules, each
contributing a part of the whole application.
This allows different parts of the application to be contributed by different
modules; for example,
@@ -90,36 +86,66 @@ one module may provide ingresses and egresses, while other
modules may individua
business logic as stateful functions. This facilitates working in independent
teams, but still deploying
into the same larger application.
-This extensibility is achieved by leveraging the [Java Service
Loader](https://docs.oracle.com/javase/tutorial/ext/basics/spi.html#the-serviceloader-class).
-In this context, each module is essentially a service provider.
+## <a name="runtime">Runtime
+
+The Stateful Functions runtime is designed to provide a set of properties
similar to what characterizes [serverless
functions](https://martinfowler.com/articles/serverless.html), but applied to
stateful problems.
+
+<p align="center">
+ <img src="docs/fig/stateful_functions_overview-ops.png" width="600px"/>
+</p>
+
+The runtime is built on Apache Flink<sup>®</sup>, with the following design
principles:
+
+* **Logical Compute/State Co-location:** Messaging, state access/updates and
function invocations are managed tightly together. This ensures a high-level of
consistency out-of-the-box.
+
+* **Physical Compute/State Separation:** Functions can be executed remotely,
with message and state access provided as part of the invocation request. This
way, functions can be managed like stateless processes and support rapid
scaling, rolling upgrades and other common operational patterns.
+
+* **Language Independence:** Function invocations use a simple HTTP/gRPC-based
protocol so that Functions can be easily implemented in various languages.
+
+This makes it possible to execute functions on a Kubernetes deployment, a FaaS
platform or behind a (micro)service, while providing consistent state and
lightweight messaging between functions.
## <a name="getting-started"></a>Getting Started
Follow the steps here to get started right away with Stateful Functions.
-This guide will walk you through locally building the project, running an
existing example, and setup to
-start developing and testing your own Stateful Functions application.
+This guide will walk you through setting up to
+start developing and testing your own Stateful Functions (Java) application,
and running an existing example. If you prefer to get started with Python, have
a look into the [StateFun Python
SDK](https://github.com/apache/flink-statefun/tree/master/statefun-python-sdk)
and the [Python Greeter
example](https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-python-greeter-example).
-### <a name="build"></a>Building the project
+### <a name="project-setup"></a>Project Setup
Prerequisites:
* Docker
+
* Maven 3.5.x or above
+
* Java 8 or above
-Currently, the project does not have any publicly available artifacts or
Docker images for use, so you would have to
-first build the project yourself before trying it out.
+You can quickly get started building Stateful Functions applications using the
provided quickstart Maven archetype:
```
-mvn clean install
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=statefun-quickstart \
+ -DarchetypeVersion=2.0-SNAPSHOT
```
-If you want to [deploy your applications using Docker](#docker), you should
also build the base Docker image:
+This allows you to name your newly created project. It will interactively ask
you for the `GroupId`,
+`ArtifactId` and package name. There will be a new directory with the same
name as your `ArtifactId`.
-```
-./tools/docker/build-stateful-functions.sh
-```
+We recommend you import this project into your IDE to develop and test it.
+IntelliJ IDEA supports Maven projects out of the box. If you use Eclipse, the
`m2e` plugin allows to import
+Maven projects. Some Eclipse bundles include that plugin by default, others
require you to install it manually.
+
+### <a name="build"></a>Building the Project
+
+If you want to build/package your project, go to your project directory and
run the `mvn clean package` command. You will find a JAR file that contains
your application, plus any libraries that you may have added as dependencies to
the application: `target/<artifact-id>-<version>.jar`.
+
+### <a name="ide-harness"></a>Running from the IDE
+
+To test out your application, you can directly run it in the IDE without any
further packaging or deployments.
+
+Please see the [Harness
example](statefun-examples/statefun-flink-harness-example) on how to do that.
### <a name="greeter"></a>Running a full example
@@ -149,46 +175,23 @@ docker-compose exec kafka-broker
kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--isolation-level read_committed \
--from-beginning \
- --topic greetings
-```
-
-### <a name="project-setup"></a>Project setup
-
-You can quickly get started building Stateful Functions applications using the
provided quickstart Maven archetype:
-
-```
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.flink \
- -DarchetypeArtifactId=statefun-quickstart \
- -DarchetypeVersion=2.0-SNAPSHOT
+ --topic greetings
```
-This allows you to name your newly created project. It will interactively ask
you for the groupId,
-artifactId, and package name. There will be a new directory with the same name
as your artifact id.
-
-We recommend you import this project into your IDE to develop and test it.
-IntelliJ IDEA supports Maven projects out of the box. If you use Eclipse, the
`m2e` plugin allows to import
-Maven projects. Some Eclipse bundles include that plugin by default, others
require you to install it manually.
-
-### <a name="ide-harness"></a>Running from the IDE
-
-To test out your application, you can directly run it in the IDE without any
further packaging or deployments.
-
-Please see the [Harness
example](statefun-examples/statefun-flink-harness-example) on how to do that.
-
## <a name="deploying"></a>Deploying Applications
-Stateful Functions applications can be packaged as either standalone
applications or Flink jobs that can be
+Stateful Functions applications can be packaged as either [standalone
applications](https://ci.apache.org/projects/flink/flink-statefun-docs-master/deployment-and-operations/packaging.html#images)
or [Flink
jobs](https://ci.apache.org/projects/flink/flink-statefun-docs-master/deployment-and-operations/packaging.html#flink-jar)
that can be
submitted to a Flink cluster.
### <a name="docker"></a>Deploying with a Docker image
-Below is an example Dockerfile for building an image for an application called
`statefun-example`:
+Below is an example Dockerfile for building a Stateful Functions image with an
[embedded](https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/modules.html#embedded-module)
module (Java) for an application called `statefun-example`.
```
FROM flink-statefun[:version-tag]
RUN mkdir -p /opt/statefun/modules/statefun-example
+
COPY target/statefun-example*jar /opt/statefun/modules/statefun-example/
```
@@ -206,10 +209,11 @@ simply include `statefun-flink-distribution` as a
dependency to your application
```
It includes all the runtime dependencies and configures the application's main
entry-point.
-You do not need to take any action beyond adding the dependency to your Maven
pom.
+You do not need to take any action beyond adding the dependency to your POM
file.
-Bundle the distribution with your application as a fat jar, and then submit it
as you normally would
-with any other Flink job:
+<div class="alert alert-info">
+ <strong>Attention:</strong> The distribution must be bundled in your
application fat JAR so that it is on Flink's <a
href="https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order";>user
code class loader</a>
+</div>
```
{$FLINK_DIR}/bin/flink run ./statefun-example.jar
@@ -217,7 +221,7 @@ with any other Flink job:
## <a name="contributing"></a>Contributing
-There are many possible ways to enhance the Stateful Functions API for
different types of applications. The runtime and operations will also evolve
with the developments in Apache Flink. If you find these ideas interesting or
promising, try Stateful Functions out and get involved!
+There are multiple ways to enhance the Stateful Functions API for different
types of applications; the runtime and operations will also evolve with the
developments in Apache Flink.
You can learn more about how to contribute in the [Apache Flink
website](https://flink.apache.org/contributing/how-to-contribute.html). For
code contributions, please read carefully the [Contributing
Code](https://flink.apache.org/contributing/contribute-code.html) section and
check the _Stateful Functions_ component in
[Jira](https://issues.apache.org/jira/browse/FLINK-15969?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Stateful%20Functions%22)
for an overview of ongoing commun [...]
diff --git a/statefun-python-sdk/README.md b/statefun-python-sdk/README.md
index a7dee5b..73d8355 100755
--- a/statefun-python-sdk/README.md
+++ b/statefun-python-sdk/README.md
@@ -1,98 +1,35 @@
# Apache Flink Stateful Functions
-Stateful Functions is an [Apache Flink](https://flink.apache.org/) library for
distributed applications and services, based on, well, you guessed it: stateful
functions.
+Stateful Functions is an [Apache Flink](https://flink.apache.org/) library
that **simplifies building distributed stateful applications**. It's based on
functions with persistent state that can interact dynamically with strong
consistency guarantees.
-The project aims to simplify the development of distributed stateful
applications by solving some of the common
-challenges in those applications: scaling, consistent state management,
reliable interaction between distributed
-services, and resource management.
+Stateful Functions makes it possible to combine a powerful approach to state
and composition with the elasticity, rapid scaling/scale-to-zero and rolling
upgrade capabilities of FaaS implementations like AWS Lambda and modern
resource orchestration frameworks like Kubernetes. With these features, it
addresses [two of the most cited
shortcomings](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf)
of many FaaS setups today: consistent state and efficient messaging between f
[...]
-Stateful Functions uses Apache Flink for distributed coordination, state, and
communication.
+This README is meant as a brief walkthrough on the StateFun Python SDK and how
to set things up
+to get yourself started with Stateful Functions in Python.
-This description is meant as a brief walkthrough on the core concepts and how
to set things up
-to get yourself started with Stateful Functions.
+For a fully detailed documentation, please visit the [official
docs](https://ci.apache.org/projects/flink/flink-statefun-docs-master).
-For a fully detailed documentation, please visit the [official
docs](https://ci.apache.org/projects/flink/flink-statefun-docs-release-stable).
-
-For code examples, please visit the examples in the [Github
repo](https://github.com/apache/flink-statefun/tree/master/statefun-examples).
+For code examples, please take a look at the [examples](../statefun-examples/).
## Table of Contents
-- [Core Concepts](#core-concepts)
- * [Abstraction](#abstraction)
- * [Function modules and extensibility](#modules)
- [Python SDK Overview](#sdkoverview)
- [Contributing](#contributing)
- [License](#license)
-## <a name="core-concepts"></a>Core Concepts
-
-### <a name="abstraction"></a>Abstraction
-
-A Stateful Functions application consists of the following primitives:
stateful functions, ingresses,
-routers, and egresses.
-
-#### Stateful functions
-
-* Stateful functions are the building blocks and namesake of the Stateful
Functions framework.
-A function is a small piece of logic that are invoked through a message.
-
-* Each stateful function exist as uniquely invokable _virtual instances_ of a
_function type_. Each instance
-is addressed by its type, as well as an unique id (a string) within its type.
-
-* Stateful functions may be invoked from ingresses or any other stateful
function (including itself).
-The caller simply needs to know the address of the target function.
-
-* Function instances are _virtual_, because they are not all active in memory
at the same time.
-At any point in time, only a small set of functions and their state exists as
actual objects. When a
-virtual instance receives a message, one of the objects is configured and
loaded with the state of that virtual
-instance and then processes the message. Similar to virtual memory, the state
of many functions might be “swapped out”
-at any point in time.
-
-* Each virtual instance of a function has its own state, which can be accessed
in local variables.
-That state is private and local to that instance.
-
-If you know Apache Flink’s `DataStream` API, you can think of stateful
functions a bit like a lightweight
-`KeyedProcessFunction`. The function type is process function transformation,
while the ID is the key. The difference
-is that functions are not assembled in a directed acyclic graph that defines
the flow of data (the streaming topology),
-but rather send events arbitrarily to all other functions using addresses.
-
-#### Ingresses and Egresses
-
-* _Ingresses_ are the way that events initially arrive in a Stateful Functions
application.
-Ingresses can be message queues, logs, or HTTP servers - anything that
produces an event to be
-handled by the application.
-
-* _Routers_ are attached to ingresses to determine which function instance
should handle an event initially.
-
-* _Egresses_ are a way to send events out from the application in a
standardized way.
-Egresses are optional; it is also possible that no events leave the
application and functions sink events or
-directly make calls to other applications.
-
-### <a name="modules"></a>Modules and extensibility
-
-A _module_ is the entry point for adding to a Stateful Functions
-application the core building block primitives, i.e. ingresses, egresses,
routers, and stateful functions.
-
-A single application may be a combination of multiple modules, each
contributing a part of the whole application.
-This allows different parts of the application to be contributed by different
modules; for example,
-one module may provide ingresses and egresses, while other modules may
individually contribute specific parts of the
-business logic as stateful functions. This facilitates working in independent
teams, but still deploying
-into the same larger application.
-
## <a name="sdkoverview"></a> Python SDK Overview
### Background
-The JVM based stateful functions implementation, has a `RequestReply`
extension (a protocol and an implementation) that allows calling into any HTTP
endpoint that implements that protocol.
+The JVM-based Stateful Functions implementation has a `RequestReply` extension
(a protocol and an implementation) that allows calling into any HTTP endpoint
that implements that protocol. Although it is possible to implement this
protocol independently, this is a minimal library for the Python programing
language that:
-Although it is possible to implement this protocol independently, this is a
minimal library for the Python programing language, that:
+* Allows users to define and declare their functions in a convenient way.
-* Allows users to define and declare their functions in a convenient way
-* Dispatch an invocation request sent from the JVM to the appropriate function
previously declared
+* Dispatches an invocation request sent from the JVM to the appropriate
function previously declared.
-### A Mini Tutorial
+### A Mini-Tutorial
-#### Define and Declare a function
+#### Define and Declare a Function
```
from statefun import StatefulFunctions
@@ -104,7 +41,7 @@ def greet(context, message: LoginEvent):
print("Hey " + message.user_name)
```
-This code, declares a function with a `FunctionType("demo", "greeter")` and
binds the greet Python instance to it.
+This code declares a function with a `FunctionType("demo", "greeter")` and
binds the greet Python instance to it.
#### Expose with a Request Reply Handler
@@ -118,7 +55,7 @@ handler = RequestReplyHandler(functions)
For example, using Flask:
-```
+```
@app.route('/statefun', methods=['POST'])
def handle():
response_data = handler(request.data)
@@ -153,7 +90,7 @@ functions:
#### Eager State Registration
-The request reply protocol, requires that the state names would be registered
in the module YAML file
+The request reply protocol requires that the state names would be registered
in the module YAML file
under the `states` section (see the example above). The state values could be
absent (`None` or a `google.protobuf.Any`) and they can be generally obtained
via the context parameter:
```
@@ -184,7 +121,7 @@ python3 -m unittest tests
## <a name="contributing"></a>Contributing
-There are many possible ways to enhance the Stateful Functions API for
different types of applications. The runtime and operations will also evolve
with the developments in Apache Flink. If you find these ideas interesting or
promising, try Stateful Functions out and get involved!
+There are multiple ways to enhance the Stateful Functions API for different
types of applications; the runtime and operations will also evolve with the
developments in Apache Flink.
You can learn more about how to contribute in the [Apache Flink
website](https://flink.apache.org/contributing/how-to-contribute.html). For
code contributions, please read carefully the [Contributing
Code](https://flink.apache.org/contributing/contribute-code.html) section and
check the _Stateful Functions_ component in
[Jira](https://issues.apache.org/jira/browse/FLINK-15969?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Stateful%20Functions%22)
for an overview of ongoing commun [...]
