[
https://issues.apache.org/jira/browse/FLINK-20265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-20265:
----------------------------------------
Description:
Currently, users declare the states a function will access with a module YAML
definition file. The modules are loaded once when starting a StateFun cluster,
meaning that the state specifications remain static throughout the cluster's
execution lifetime.
We propose that state specifications should be declared by the function
themselves via the language SDKs, instead of being declared in the module YAMLs.
The state specifications, now living in the functions, can be made discoverable
by the StateFun runtime through the invocation request-reply protocol.
Brief simplified sketch of the extended protocol:
- StateFun dispatches an invocation request, with states [A, B].
- Function receives request, but since it requires states [A, B, C, D], it
responds with a {{IncompleteInvocationContext}} response that indicates state
values for [C, D] is missing.
- StateFun receives this response, and registers new Flink state handles for
[C, D].
- Finally, a new invocation request with the same input messages, but "patched"
with new states to contain all values for [A, B, C, D] is resent to the
function.
This JIRA only targets updating the Protobuf messages {{ToFunction}} and
{{FromFunction}} to fulfill the extended protocol, and support handling
{{IncompleteInvocationContext}} responses in the request dispatcher.
Updating SDKs should be separate subtask JIRAs.
was:
Currently, users declare the states a function will access with a module YAML
definition file. The modules are loaded once when starting a StateFun cluster,
meaning that the state specifications remain static throughout the cluster's
execution lifetime.
We propose that state specifications should be declared by the function
themselves via the language SDKs, instead of being declared in the module YAMLs.
The state specifications, now living in the functions, can be made discoverable
by the StateFun runtime through the invocation request-reply protocol.
Brief simplified sketch of the extended protocol:
- StateFun dispatches an invocation request, with states [A, B].
- Function receives request, but since it requires states [A, B, C, D], it
responds with a {{IncompleteInvocationContext}} response that indicates state
values for [C, D] is missing.
- StateFun receives this response, and registers new Flink state handles for
[C, D].
- Finally, a new invocation request with the same input messages, but "patched"
with new states to contain all values for [A, B, C, D] is resent to the
function.
> Extend invocation protocol to allow functions to indicate incomplete
> invocation context
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-20265
> URL: https://issues.apache.org/jira/browse/FLINK-20265
> Project: Flink
> Issue Type: Sub-task
> Components: Stateful Functions
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Major
> Fix For: statefun-2.3.0
>
>
> Currently, users declare the states a function will access with a module YAML
> definition file. The modules are loaded once when starting a StateFun
> cluster, meaning that the state specifications remain static throughout the
> cluster's execution lifetime.
> We propose that state specifications should be declared by the function
> themselves via the language SDKs, instead of being declared in the module
> YAMLs.
> The state specifications, now living in the functions, can be made
> discoverable by the StateFun runtime through the invocation request-reply
> protocol.
> Brief simplified sketch of the extended protocol:
> - StateFun dispatches an invocation request, with states [A, B].
> - Function receives request, but since it requires states [A, B, C, D], it
> responds with a {{IncompleteInvocationContext}} response that indicates state
> values for [C, D] is missing.
> - StateFun receives this response, and registers new Flink state handles for
> [C, D].
> - Finally, a new invocation request with the same input messages, but
> "patched" with new states to contain all values for [A, B, C, D] is resent to
> the function.
> This JIRA only targets updating the Protobuf messages {{ToFunction}} and
> {{FromFunction}} to fulfill the extended protocol, and support handling
> {{IncompleteInvocationContext}} responses in the request dispatcher.
> Updating SDKs should be separate subtask JIRAs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
