[
https://issues.apache.org/jira/browse/FLINK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736598#comment-14736598
]
ASF GitHub Bot commented on FLINK-1297:
---------------------------------------
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/605#issuecomment-138863134
Hi @tammymendt. Thanks for the pull request! The accumulators work a little
bit different now because they are now accumulated on a per-task basis and
reported to the job manager in regular intervals.
The `clone()` method in `OperatorStatistics` doesn't create a deep copy of
the object, i.e. some references are reused. That causes problems when merging
the accumulators because runtime accumulators are modified while merging
accumulators for sending them to the job manager.
I could make the test pass by a nasty deep copy using Java serialization.
However, I didn't managed to make a proper copy using the provided interfaces.
I think you can probably do that faster because you know the code very well.
> Add support for tracking statistics of intermediate results
> -----------------------------------------------------------
>
> Key: FLINK-1297
> URL: https://issues.apache.org/jira/browse/FLINK-1297
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Reporter: Alexander Alexandrov
> Assignee: Alexander Alexandrov
> Fix For: 0.10
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> One of the major problems related to the optimizer at the moment is the lack
> of proper statistics.
> With the introduction of staged execution, it is possible to instrument the
> runtime code with a statistics facility that collects the required
> information for optimizing the next execution stage.
> I would therefore like to contribute code that can be used to gather basic
> statistics for the (intermediate) result of dataflows (e.g. min, max, count,
> count distinct) and make them available to the job manager.
> Before I start, I would like to hear some feedback form the other users.
> In particular, to handle skew (e.g. on grouping) it might be good to have
> some sort of detailed sketch about the key distribution of an intermediate
> result. I am not sure whether a simple histogram is the most effective way to
> go. Maybe somebody would propose another lightweight sketch that provides
> better accuracy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
