[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r391371162
##
File path: statefun-docs/docs/sdk/modules.rst
##
@@ -0,0 +1,96 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _modules:
+
+###
+Modules
+###
+
+Stateful Function applications are composed of one or more ``Modules``.
+A module is a bundle of functions that are loaded by the runtime and available
to be messaged.
+Functions from all loaded modules are multiplexed and free to message each
other arbitrarily.
+
+Stateful Functions supports two types of modules: Embedded and Remote.
+
+.. contents:: :local:
+
+.. _embedded_module:
+
+Embedded Module
+===
+
+Embedded modules are co-located with, and embedded within, the {flink} runtime.
+
+This module type only supports JVM based languages and are defined by
implementing the ``StatefulFunctionModule`` interface.
+Embedded modules offer a single configuration method where stateful functions
are bound to the system based on their :ref:`function type `.
+Runtime configurations are available through the ``globalConfiguration``,
which is the union of all configurations in the applications
``flink-conf.yaml`` under the prefix ``statefun.module.global-config`` and any
command line arguments passed in the form ``--key value``.
+
+.. literalinclude::
../../src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
+:language: java
+:lines: 18-
+
+Embedded modules leverage `Java’s Service Provider Interfaces (SPI)
`_ for
discovery.
+This means that every JAR should contain a file
``org.apache.flink.statefun.sdk.spi.StatefulFunctionModule`` in the
``META_INF/services`` resource directory that lists all available modules that
it provides.
Review comment:
Got it. Regarding the dependency, lets discuss that on the side.
My gut feeling is to not add any extra dependencies at this point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r391370729 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. Review comment: Ok, I think that makes sense! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390706590 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. + +.. figure:: ../_static/images/concepts/statefun-app-state.svg +:width: 85% +:align: center + +One of Apache Flink's core strengths is its ability to provide fault-tolerant local state. +When inside a function, while it is performing some computation, you are always working with local state in local variables. + +Fault Tolerance +=== + +For both state and messaging, Stateful Function's is still able to provide the exactly once guaruntees users expect from a modern data processessing framework. + +.. figure:: ../_static/images/concepts/statefun-app-fault-tolerance.svg +:width: 85% +:align: center + +In the case of failure, the entire state of the world (both persisted values and messages) are rolled back to simulate completely failure free execution. + +These guaruntees are provided with no database required, instead Stateful Function's leverages Apache Flink's proven snapshotting mechanism. Review comment: ```suggestion These guarantees are provided with no database required, instead Stateful Function's leverages Apache Flink's proven snapshotting mechanism. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390711708
##
File path: statefun-docs/docs/concepts/logical.rst
##
@@ -0,0 +1,79 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _logical-functions:
+
+#
+Logical Functions
+#
+
+Stateful Function's are allocated logically, which means the system can
support an unbounded number of instances with a finite amount of resources.
+Logical instances do not use CPU, memory, or threads when not actively being
invoked, there is no theoretical upper limit on the number of instances that
can created.
+Users are encouraged to model their applications as granularly as possible,
based on what makes the most sense for their application, instead of desigining
applications around resource constraints.
+
+.. contents:: :local:
+
+.. _address:
+
+Function Address
+
+
+In a local environment, the address of an object is the same as a reference to
it.
+But in a Stateful Function's application, function instances are virtual and
their runtime location is not exposed to the user.
+Instead, an ``Address`` is used to reference a specific stateful functions in
the system..
+
+.. code-block:: proto
+
+ syntax = "proto3";
+
+ message Address {
+
+message FunctionType {
+ string namespace = 1;
+ string name = 2;
+}
+
+FunctionType function_type = 1;
+string id = 2;
+ }
+
+
+An address is made of two components, a ``FunctionType`` and ``ID``.
+A function type is similar to a class in an object-oriented language; it
declares what sort of function the address references.
+The id is a primary key, it scopes the function call to a specific instances
of the function type.
+
+When a function is being invoked, all actions - including reads and writes of
persisted values - are scoped to the current address.
+
+For example, imagine a there was a Stateful Function application to track the
inventory of a warehouse.
+One possible implementation could include an ``Inventory`` function that
tracks the number units in stock for a particular item; this would be the
function type.
+There would then be one logical instance of this type for each SKU the
warehouse manages.
+If it were clothing, there might be an instance for shirts and another for
pants; "shirt" and "pant" would be two ids.
+Each instance may be interacted with and messaged independently.
+The application is free to create as many instances as there are types of
items in inventory.
+
+Function Lifecycle
+==
+
+Logical functions are neither created nor destroyed, but always exist
throughout the lifetime of an application.
+When an application starts, each parallel worker of the framework will create
one physical object per function type.
+This object will be used to execute all logical instances of that type that
are run by that particular worker.
+The first time a message is sent to an address, it will be as if that instance
had always existed with its persisted values returning ``NULL``.
Review comment:
```suggestion
The first time a message is sent to an address, it will be as if that
instance had always existed with its persisted states being empty.
```
Not all states are initially `null`, some are empty collections for example.
Suggesting a more general term here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390713139
##
File path: statefun-docs/docs/getting_started/walkthrough.rst
##
@@ -168,7 +168,7 @@ Each time a message is processed, the function computes a
personalized message f
It reads and updates the number of times that user has been seen and sends a
greeting to the egress.
You can check the full code for the application described in this walkthrough
`here <{examples}/statefun-greeter-example>`_.
-In particular, take a look at the :ref:`module ` GreetingModule, which
is the main entry point for the full application, to see how everything gets
tied together.
+In particular, take a look at the module GreetingModule, which is the main
entry point for the full application, to see how everything gets tied together.
Review comment:
```suggestion
In particular, take a look at the module ``GreetingModule``, which is the
main entry point for the full application, to see how everything gets tied
together.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390712156
##
File path: statefun-docs/docs/concepts/logical.rst
##
@@ -0,0 +1,79 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _logical-functions:
+
+#
+Logical Functions
+#
+
+Stateful Function's are allocated logically, which means the system can
support an unbounded number of instances with a finite amount of resources.
+Logical instances do not use CPU, memory, or threads when not actively being
invoked, there is no theoretical upper limit on the number of instances that
can created.
+Users are encouraged to model their applications as granularly as possible,
based on what makes the most sense for their application, instead of desigining
applications around resource constraints.
+
+.. contents:: :local:
+
+.. _address:
+
+Function Address
+
+
+In a local environment, the address of an object is the same as a reference to
it.
+But in a Stateful Function's application, function instances are virtual and
their runtime location is not exposed to the user.
+Instead, an ``Address`` is used to reference a specific stateful functions in
the system..
+
+.. code-block:: proto
+
+ syntax = "proto3";
+
+ message Address {
+
+message FunctionType {
+ string namespace = 1;
+ string name = 2;
+}
+
+FunctionType function_type = 1;
+string id = 2;
+ }
+
+
+An address is made of two components, a ``FunctionType`` and ``ID``.
+A function type is similar to a class in an object-oriented language; it
declares what sort of function the address references.
+The id is a primary key, it scopes the function call to a specific instances
of the function type.
+
+When a function is being invoked, all actions - including reads and writes of
persisted values - are scoped to the current address.
+
+For example, imagine a there was a Stateful Function application to track the
inventory of a warehouse.
+One possible implementation could include an ``Inventory`` function that
tracks the number units in stock for a particular item; this would be the
function type.
+There would then be one logical instance of this type for each SKU the
warehouse manages.
+If it were clothing, there might be an instance for shirts and another for
pants; "shirt" and "pant" would be two ids.
+Each instance may be interacted with and messaged independently.
+The application is free to create as many instances as there are types of
items in inventory.
+
+Function Lifecycle
+==
+
+Logical functions are neither created nor destroyed, but always exist
throughout the lifetime of an application.
+When an application starts, each parallel worker of the framework will create
one physical object per function type.
+This object will be used to execute all logical instances of that type that
are run by that particular worker.
+The first time a message is sent to an address, it will be as if that instance
had always existed with its persisted values returning ``NULL``.
+
+Clearing all persisted values of a type is the same as destroying it.
Review comment:
```suggestion
Clearing all persisted states of a function instance is the same as
destroying it.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390705594 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. + +.. figure:: ../_static/images/concepts/statefun-app-state.svg +:width: 85% +:align: center + +One of Apache Flink's core strengths is its ability to provide fault-tolerant local state. Review comment: Same comment here about "local". While it is true for all Flink function / operators, in Stateful Functions it's not always true. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390710405
##
File path: statefun-docs/docs/concepts/logical.rst
##
@@ -0,0 +1,79 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _logical-functions:
+
+#
+Logical Functions
+#
+
+Stateful Function's are allocated logically, which means the system can
support an unbounded number of instances with a finite amount of resources.
+Logical instances do not use CPU, memory, or threads when not actively being
invoked, there is no theoretical upper limit on the number of instances that
can created.
+Users are encouraged to model their applications as granularly as possible,
based on what makes the most sense for their application, instead of desigining
applications around resource constraints.
+
+.. contents:: :local:
+
+.. _address:
+
+Function Address
+
+
+In a local environment, the address of an object is the same as a reference to
it.
+But in a Stateful Function's application, function instances are virtual and
their runtime location is not exposed to the user.
+Instead, an ``Address`` is used to reference a specific stateful functions in
the system..
+
+.. code-block:: proto
+
+ syntax = "proto3";
+
+ message Address {
+
+message FunctionType {
+ string namespace = 1;
+ string name = 2;
+}
+
+FunctionType function_type = 1;
+string id = 2;
+ }
+
+
+An address is made of two components, a ``FunctionType`` and ``ID``.
+A function type is similar to a class in an object-oriented language; it
declares what sort of function the address references.
+The id is a primary key, it scopes the function call to a specific instances
of the function type.
+
+When a function is being invoked, all actions - including reads and writes of
persisted values - are scoped to the current address.
Review comment:
```suggestion
When a function is being invoked, all actions - including reads and writes
of persisted states - are scoped to the current address.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390714422
##
File path: statefun-docs/docs/sdk/modules.rst
##
@@ -0,0 +1,96 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _modules:
+
+###
+Modules
+###
+
+Stateful Function applications are composed of one or more ``Modules``.
+A module is a bundle of functions that are loaded by the runtime and available
to be messaged.
+Functions from all loaded modules are multiplexed and free to message each
other arbitrarily.
+
+Stateful Functions supports two types of modules: Embedded and Remote.
+
+.. contents:: :local:
+
+.. _embedded_module:
+
+Embedded Module
+===
+
+Embedded modules are co-located with, and embedded within, the {flink} runtime.
+
+This module type only supports JVM based languages and are defined by
implementing the ``StatefulFunctionModule`` interface.
+Embedded modules offer a single configuration method where stateful functions
are bound to the system based on their :ref:`function type `.
+Runtime configurations are available through the ``globalConfiguration``,
which is the union of all configurations in the applications
``flink-conf.yaml`` under the prefix ``statefun.module.global-config`` and any
command line arguments passed in the form ``--key value``.
+
+.. literalinclude::
../../src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
+:language: java
+:lines: 18-
+
+Embedded modules leverage `Java’s Service Provider Interfaces (SPI)
`_ for
discovery.
+This means that every JAR should contain a file
``org.apache.flink.statefun.sdk.spi.StatefulFunctionModule`` in the
``META_INF/services`` resource directory that lists all available modules that
it provides.
Review comment:
There's a more straightforward way to do that now, using `@AutoService`
annotation on the module classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390707590 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. + +.. figure:: ../_static/images/concepts/statefun-app-state.svg +:width: 85% +:align: center + +One of Apache Flink's core strengths is its ability to provide fault-tolerant local state. +When inside a function, while it is performing some computation, you are always working with local state in local variables. + +Fault Tolerance +=== + +For both state and messaging, Stateful Function's is still able to provide the exactly once guaruntees users expect from a modern data processessing framework. + +.. figure:: ../_static/images/concepts/statefun-app-fault-tolerance.svg +:width: 85% +:align: center + +In the case of failure, the entire state of the world (both persisted values and messages) are rolled back to simulate completely failure free execution. + +These guaruntees are provided with no database required, instead Stateful Function's leverages Apache Flink's proven snapshotting mechanism. + +Event Egress + + +Finally, applications can output data to external systems via event egress's. + +.. figure:: ../_static/images/concepts/statefun-app-egress.svg +:width: 85% +:align: center + +Of course, functions perform arbitrary computation and can do whatever they like, which includes making RPC calls and connecting to other systems. +By using an event egress, applications can leverage pre-built interfaces and data output is tied into the systems fault tolerance to avoid data loss. Review comment: Does this sentence imply that making arbitrary RPC calls / connecting to other systems can have data loss, without using event egresses? It sort of read to me like that, it being after the `Of course, functions perform arbitrary ...` sentence. But that isn't true, as using async operations we still guarantee at-least once interactions with external systems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390704803 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. Review comment: ```suggestion Persisted State The first is that all functions have locally embedded state, known as persisted states. ``` Maybe call these persisted states instead. Otherwise, the term persisted value now points to a very specific primitive, along side persisted tables / appending buffers, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390709762 ## File path: statefun-docs/docs/concepts/logical.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _logical-functions: + +# +Logical Functions +# + +Stateful Function's are allocated logically, which means the system can support an unbounded number of instances with a finite amount of resources. +Logical instances do not use CPU, memory, or threads when not actively being invoked, there is no theoretical upper limit on the number of instances that can created. +Users are encouraged to model their applications as granularly as possible, based on what makes the most sense for their application, instead of desigining applications around resource constraints. + +.. contents:: :local: + +.. _address: + +Function Address + + +In a local environment, the address of an object is the same as a reference to it. +But in a Stateful Function's application, function instances are virtual and their runtime location is not exposed to the user. +Instead, an ``Address`` is used to reference a specific stateful functions in the system.. Review comment: ```suggestion Instead, an ``Address`` is used to reference a specific stateful function in the system.. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390709148 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. Review comment: As of now: the differences mentioned on this page are: - persisted state - fault tolerance Another one is logical instances. While that has its own dedicated page, I wonder if we should also very briefly mention it here, or have a link that directly jumps to the logical instances page. Not sure, if that breaks the "reading flow" for the user. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390706518 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. + +.. figure:: ../_static/images/concepts/statefun-app-state.svg +:width: 85% +:align: center + +One of Apache Flink's core strengths is its ability to provide fault-tolerant local state. +When inside a function, while it is performing some computation, you are always working with local state in local variables. + +Fault Tolerance +=== + +For both state and messaging, Stateful Function's is still able to provide the exactly once guaruntees users expect from a modern data processessing framework. + +.. figure:: ../_static/images/concepts/statefun-app-fault-tolerance.svg +:width: 85% +:align: center + +In the case of failure, the entire state of the world (both persisted values and messages) are rolled back to simulate completely failure free execution. Review comment: ```suggestion In the case of failure, the entire state of the world (both persisted states and messages) are rolled back to simulate completely failure free execution. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390703680 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. Review comment: ```suggestion This can be anything from a Kafka topic, to a messsage queue, to an http request - anything that can get data into the system and trigger the intitial functions to begin computation. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390710336
##
File path: statefun-docs/docs/concepts/logical.rst
##
@@ -0,0 +1,79 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _logical-functions:
+
+#
+Logical Functions
+#
+
+Stateful Function's are allocated logically, which means the system can
support an unbounded number of instances with a finite amount of resources.
+Logical instances do not use CPU, memory, or threads when not actively being
invoked, there is no theoretical upper limit on the number of instances that
can created.
+Users are encouraged to model their applications as granularly as possible,
based on what makes the most sense for their application, instead of desigining
applications around resource constraints.
+
+.. contents:: :local:
+
+.. _address:
+
+Function Address
+
+
+In a local environment, the address of an object is the same as a reference to
it.
+But in a Stateful Function's application, function instances are virtual and
their runtime location is not exposed to the user.
+Instead, an ``Address`` is used to reference a specific stateful functions in
the system..
+
+.. code-block:: proto
+
+ syntax = "proto3";
+
+ message Address {
+
+message FunctionType {
+ string namespace = 1;
+ string name = 2;
+}
+
+FunctionType function_type = 1;
+string id = 2;
+ }
+
+
+An address is made of two components, a ``FunctionType`` and ``ID``.
+A function type is similar to a class in an object-oriented language; it
declares what sort of function the address references.
+The id is a primary key, it scopes the function call to a specific instances
of the function type.
Review comment:
```suggestion
The id is a primary key, which scopes the function call to a specific
instance of the function type.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390709575 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values Review comment: Can we make this title, and "Fault Tolerance" one level below the other titles like "Event Ingress" / "Event Egress" / "Stateful Functions"? The reason for that is because these are sub-details further explaining "Stateful Functions", and it striked me a bit odd that all these sections are the same level. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs]
Refactor statefun documentation for multi-language SDKs
URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390712287
##
File path: statefun-docs/docs/concepts/logical.rst
##
@@ -0,0 +1,79 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _logical-functions:
+
+#
+Logical Functions
+#
+
+Stateful Function's are allocated logically, which means the system can
support an unbounded number of instances with a finite amount of resources.
+Logical instances do not use CPU, memory, or threads when not actively being
invoked, there is no theoretical upper limit on the number of instances that
can created.
+Users are encouraged to model their applications as granularly as possible,
based on what makes the most sense for their application, instead of desigining
applications around resource constraints.
+
+.. contents:: :local:
+
+.. _address:
+
+Function Address
+
+
+In a local environment, the address of an object is the same as a reference to
it.
+But in a Stateful Function's application, function instances are virtual and
their runtime location is not exposed to the user.
+Instead, an ``Address`` is used to reference a specific stateful functions in
the system..
+
+.. code-block:: proto
+
+ syntax = "proto3";
+
+ message Address {
+
+message FunctionType {
+ string namespace = 1;
+ string name = 2;
+}
+
+FunctionType function_type = 1;
+string id = 2;
+ }
+
+
+An address is made of two components, a ``FunctionType`` and ``ID``.
+A function type is similar to a class in an object-oriented language; it
declares what sort of function the address references.
+The id is a primary key, it scopes the function call to a specific instances
of the function type.
+
+When a function is being invoked, all actions - including reads and writes of
persisted values - are scoped to the current address.
+
+For example, imagine a there was a Stateful Function application to track the
inventory of a warehouse.
+One possible implementation could include an ``Inventory`` function that
tracks the number units in stock for a particular item; this would be the
function type.
+There would then be one logical instance of this type for each SKU the
warehouse manages.
+If it were clothing, there might be an instance for shirts and another for
pants; "shirt" and "pant" would be two ids.
+Each instance may be interacted with and messaged independently.
+The application is free to create as many instances as there are types of
items in inventory.
+
+Function Lifecycle
+==
+
+Logical functions are neither created nor destroyed, but always exist
throughout the lifetime of an application.
+When an application starts, each parallel worker of the framework will create
one physical object per function type.
+This object will be used to execute all logical instances of that type that
are run by that particular worker.
+The first time a message is sent to an address, it will be as if that instance
had always existed with its persisted values returning ``NULL``.
+
+Clearing all persisted values of a type is the same as destroying it.
+If an instance has no state and is not actively running, then it occupies no
CPU, no threads, and no memory.
+
+An instance with data stored in one or more of its persisted values only
occupies the resources necessary to store that data.
Review comment:
```suggestion
An instance with data stored in one or more of its persisted states only
occupies the resources necessary to store that data.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390710157 ## File path: statefun-docs/docs/concepts/logical.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _logical-functions: + +# +Logical Functions +# + +Stateful Function's are allocated logically, which means the system can support an unbounded number of instances with a finite amount of resources. +Logical instances do not use CPU, memory, or threads when not actively being invoked, there is no theoretical upper limit on the number of instances that can created. +Users are encouraged to model their applications as granularly as possible, based on what makes the most sense for their application, instead of desigining applications around resource constraints. + +.. contents:: :local: + +.. _address: + +Function Address + + +In a local environment, the address of an object is the same as a reference to it. +But in a Stateful Function's application, function instances are virtual and their runtime location is not exposed to the user. +Instead, an ``Address`` is used to reference a specific stateful functions in the system.. + +.. code-block:: proto + + syntax = "proto3"; Review comment: Why use protobuf to illustrate this? It seems unrelated, also might confuse users that they need to construct `Address` as a protobuf message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390705435 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. Review comment: Also not sure of the mention "locally" here. It seems to be a bit dangerous to use, considering that remote request-reply protocol functions do not really have their state available locally. How about something along the lines of: `The first is that all functions have *something*-scoped state, known as persisted states.` I'm afraid while I feel unsure about it, I also don't have much better suggestions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390708228 ## File path: statefun-docs/docs/concepts/logical.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _logical-functions: + +# +Logical Functions +# + +Stateful Function's are allocated logically, which means the system can support an unbounded number of instances with a finite amount of resources. +Logical instances do not use CPU, memory, or threads when not actively being invoked, there is no theoretical upper limit on the number of instances that can created. Review comment: ```suggestion Logical instances do not use CPU, memory, or threads when not actively being invoked, so there is no theoretical upper limit on the number of instances that can created. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs
tzulitai commented on a change in pull request #54: [FLINK-16515][docs] Refactor statefun documentation for multi-language SDKs URL: https://github.com/apache/flink-statefun/pull/54#discussion_r390706263 ## File path: statefun-docs/docs/concepts/index.rst ## @@ -0,0 +1,97 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _concepts: + + +Concepts + + +.. toctree:: + :hidden: + + logical + +Stateful Functions provides a framework for building event drivent applications. +Here, we explain important aspects of Stateful Function’s architecture. + +.. contents:: :local: + +Event Ingress += + +Stateful Function applications sit squarely in the event driven space, so the natural place to start is with getting events into the system. + +.. figure:: ../_static/images/concepts/statefun-app-ingress.svg +:width: 85% +:align: center + +In stateful functions, the component that ingests records into the system is called an event ingress. +This can be anything from a Kafka topic, to a messsage queue, to an http request. +Anything that can get data into the system and trigger the intitial functions to begin compution. + +Stateful Functions +== + +At the core of the diagram are the namesake stateful functions. + +.. figure:: ../_static/images/concepts/statefun-app-functions.svg +:width: 85% +:align: center + +Think of these as the building blocks for your service. +They can message each other arbitrarily, which is one way in which this framework moves away from the traditional stream processing view of the world. +Instead of building up a static dataflow DAG, these functions can communicate with each other in arbitrary, potentially cyclic, even round trip ways. + +If you are familiar with actor programming, this does share certain similarities in its ability to dynamically message between components. +However, there are a number of significant differences. + +Persisted Values + + +The first is that all functions have locally embedded state, known as persisted values. + +.. figure:: ../_static/images/concepts/statefun-app-state.svg +:width: 85% +:align: center + +One of Apache Flink's core strengths is its ability to provide fault-tolerant local state. +When inside a function, while it is performing some computation, you are always working with local state in local variables. + +Fault Tolerance +=== + +For both state and messaging, Stateful Function's is still able to provide the exactly once guaruntees users expect from a modern data processessing framework. Review comment: ```suggestion For both state and messaging, Stateful Functions is still able to provide the exactly-once guarantees users expect from a modern data processessing framework. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
