[
https://issues.apache.org/jira/browse/FLINK-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743706#comment-14743706
]
Gyula Fora commented on FLINK-2283:
-----------------------------------
I understand your point but what I am trying to say that this operator is that
this operator with it's current specification is not really a window operation.
(because infinite window is not really a window)
State discard is clearly specified as never happening in this case, so there is
no exiration like you would assume for a window operation. Increasing number of
keys is a problem no matter how this is implemented, but this should be handled
by the state backend and this should not be a real issue as inactive keys can
be lazily retrieved when necessary.
So I think there are 2 questions here:
- Do we want to have an operator with the semantics of the current rolling
group-reduce?
- If we do, how do we implement it?
For the first I don't see why not as the state limitation can be overcome by
the state backend. For the second, why would we chose a windowed implementation
when it only adds complexity over the trivial map implementation.
> Make grouped reduce/fold/aggregations stateful using Partitioned state
> ----------------------------------------------------------------------
>
> Key: FLINK-2283
> URL: https://issues.apache.org/jira/browse/FLINK-2283
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 0.10
> Reporter: Gyula Fora
> Assignee: Márton Balassi
> Priority: Minor
>
> Currently the inner state of the grouped aggregations are not persisted as an
> operator state.
> These operators should be reimplemented to use the newly introduced
> partitioned state abstractions which will make them fault tolerant and
> scalable for the future.
> A suggested implementation would be to use a stateful mapper to implement the
> desired behaviour.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
