[
https://issues.apache.org/jira/browse/FLINK-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743686#comment-14743686
]
Stephan Ewen commented on FLINK-2283:
-------------------------------------
Aggregating is a group operation, group operations need windows. That is one of
the fundamental concepts in streaming, no?
Programs run forever (or at least years, which is infinitely for all practical
matters). Any form of gathering state needs a clear specification when state is
discarded, otherwise it is not practically usable. This "expiry" is exactly the
window.
We have a user on the mailing list right now that is running a Flink
installation and has a setup where continuously new keys arrive. The state
grows infinitely, the program is unusable.
This particular case would need something like a "trigger each time, evict
every hour" window or so. Why not treat it like that and tool around it to make
it very easily usable (i.e., with pre-defined policies)?
> Make grouped reduce/fold/aggregations stateful using Partitioned state
> ----------------------------------------------------------------------
>
> Key: FLINK-2283
> URL: https://issues.apache.org/jira/browse/FLINK-2283
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 0.10
> Reporter: Gyula Fora
> Assignee: Márton Balassi
> Priority: Minor
>
> Currently the inner state of the grouped aggregations are not persisted as an
> operator state.
> These operators should be reimplemented to use the newly introduced
> partitioned state abstractions which will make them fault tolerant and
> scalable for the future.
> A suggested implementation would be to use a stateful mapper to implement the
> desired behaviour.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
