[
https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14538269#comment-14538269
]
ASF GitHub Bot commented on FLINK-951:
--------------------------------------
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/570#issuecomment-100996059
I had a look at the pull request and I like very much what it tries to do.
The problem right now is that I can hardly say without investing a lot of
time whether this is in good shape to merge. This pull request does a at least
two very big things at the same time:
- Move iteration synchronization to the JobManager
- Unify aggregators and accumulators into one.
With all the example / testcase adjustments, this becomes a lot to review.
The description of the pull request also does not make it easy, since many
questions and decisions that arise are not explained:
- What interface do the unified aggregators/accumulators follow: The
aggregators, or the accumulators.
- How is the blocking superstep synchronization currently done. With actor
ask?
- How is the aggregator/accumulator unification achieved, when aggregators
are created per superstep, and accumulators once?
This is a lot for a very delicate and critical mechanism. I think if we
want to merge this, we would need more details on how things were changed (what
is the concept behind the changed, not just what are the code diffs).
We may need to break it into multiple self-contained changes that we can
individually review and merge, to make sure that it gets properly checked and
will work robustly.
> Reworking of Iteration Synchronization, Accumulators and Aggregators
> --------------------------------------------------------------------
>
> Key: FLINK-951
> URL: https://issues.apache.org/jira/browse/FLINK-951
> Project: Flink
> Issue Type: Improvement
> Components: Iterations, Optimizer
> Affects Versions: 0.9
> Reporter: Markus Holzemer
> Assignee: Markus Holzemer
> Labels: refactoring
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I just realized that there is no real Jira issue for the task I am currently
> working on.
> I am currently reworking a few things regarding Iteration Synchronization,
> Accumulators and Aggregators. Currently the synchronization at the end of one
> superstep is done through channel events. That makes it hard to track the
> current status of iterations. That is why I am changing this synchronization
> to use RPC calls with the JobManager, so that the JobManager manages the
> current status of all iterations.
> Currently we use Accumulators outside of iterations and Aggregators inside of
> iterations. Both have a similiar function, but a bit different interfaces and
> handling. I want to unify these two concepts. I propose that we stick in the
> future to Accumulators only. Aggregators therefore are removed and
> Accumulators are extended to cover the usecases Aggregators were used fore
> before. The switch to RPC for iterations makes it possible to also send the
> current Accumulator values at the end of each superstep, so that the
> JobManager (and thereby the webinterface) will be able to print intermediate
> accumulation results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
