[
https://issues.apache.org/jira/browse/FLINK-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614961#comment-16614961
]
Kostas Kloudas commented on FLINK-10343:
----------------------------------------
Well the discussion is not about the implementation. It is about the APIs here
and what is exposed at which level.
In this context, I would say the Functions should be as they are, so that they
are safe for any user to use.
Now, caching the keyed state before pushing to persistent storage,* if
implemented*, it should be the responsibility of the
state backend, and *not* the operator. The operator should be just the
orchestrator between the state, the udfs, and the timers.
So it is responsible only for calling the right method at the right time.
Nothing more.
Given the above, I would say that the issue now becomes should we (or not)
implement and support a caching state backend.
This sounds like a good idea, and there were some thoughts about this in the
past. But this is a bigger discussion with the participation of also other
members of the community that are more familiar with the difficulties of such a
task (e.g. support
of incremental checkpoints, savepoints, etc).
> Expose setCurrentKey method to streamRuntimeContext
> ---------------------------------------------------
>
> Key: FLINK-10343
> URL: https://issues.apache.org/jira/browse/FLINK-10343
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.7.0
> Reporter: aitozi
> Assignee: aitozi
> Priority: Major
> Fix For: 1.7.0
>
>
> when we use reducing state / aggregating keyed state and so on , we have to
> read value from state backend and update the value with userFunction and then
> put back to state backend. If we can just cache certain data in heap with a
> map, and update once in snapshot method with
> {code:java}
> snapshot() {
> for(Map.Entry<String, String> entry : map.entrySet()){
> setCurrentKey(entry.getKey());
> valueState.update(entry.getValue()); // put value back to state backend
> }}
> {code}
> we just have to expose the setCurrentKey to userFunction and the will enable
> the ability to cache partitial keyedState in memory by userself.
> what's your opinion [[email protected]] [~azagrebin] ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
