[
https://issues.apache.org/jira/browse/FLINK-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai closed FLINK-16244.
---------------------------------------
Fix Version/s: statefun-1.1
Resolution: Fixed
Merged to master via f367b40e9096e89a7afe04c93b067bb7b3f25a1e
> Add Asynchronous operations to state lazily.
> --------------------------------------------
>
> Key: FLINK-16244
> URL: https://issues.apache.org/jira/browse/FLINK-16244
> Project: Flink
> Issue Type: Task
> Components: Stateful Functions
> Reporter: Igal Shilman
> Assignee: Igal Shilman
> Priority: Major
> Labels: pull-request-available
> Fix For: statefun-1.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently AsyncSink would add eagerly a registered async operation.
> An alternative approach would be to keep the async operations in an in memory
> map and only
> write them to the underlying map on snapshotState().
> The rational behind this approach is the assumption that most async
> operations complete between two consecutive checkpoints, and therefore adding
> and removing them from the underlying state backend (rocksdb by default) is
> wasteful.
> An implementation outline suggestion:
> 1. Add a LazyAsyncOperations class that keeps both an in memory map
> and a MapStateHandle
> this map would support add(), remove() and also flush()
> 2. Use that class in AsyncSink and in AsyncMessageDecorator
> 3. call flush() from FunctionGroupOperator#snapshotState()
> note that a special care should be taken in flash() as the current key needs
> to be set
> in the keyedStateBackend.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
