[
https://issues.apache.org/jira/browse/FLINK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497982#comment-14497982
]
ASF GitHub Bot commented on FLINK-1297:
---------------------------------------
Github user aalexandrov commented on the pull request:
https://github.com/apache/flink/pull/605#issuecomment-93719337
:+1: thanks for the great work! I'll review that (probably over the
weekend) and will appreciate if some of the core Flink committers (@sewen,
@rmetzger, @fhueske) can also make a pass over the code.
One more caveat from me: this implements only the runtime aspect of the
statistics collecting logic.
A second PR which allows to configure the points where statistics should be
tracked in a programmatic way as part of the DataBag API shoud follow.
@tammymendt and me were discussing as syntax along the lines of:
```scala
A = // some dataflow assembly code
A.withStatistics( "statsForX", keySelectorFn )
env.execute()
// grab the statistics after the execution is done
env.getAccumulator("statsForX")
```
Once this is in place we will play around and implement some ideas on
incremental optimization.
> Add support for tracking statistics of intermediate results
> -----------------------------------------------------------
>
> Key: FLINK-1297
> URL: https://issues.apache.org/jira/browse/FLINK-1297
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Reporter: Alexander Alexandrov
> Assignee: Alexander Alexandrov
> Fix For: 0.9
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> One of the major problems related to the optimizer at the moment is the lack
> of proper statistics.
> With the introduction of staged execution, it is possible to instrument the
> runtime code with a statistics facility that collects the required
> information for optimizing the next execution stage.
> I would therefore like to contribute code that can be used to gather basic
> statistics for the (intermediate) result of dataflows (e.g. min, max, count,
> count distinct) and make them available to the job manager.
> Before I start, I would like to hear some feedback form the other users.
> In particular, to handle skew (e.g. on grouping) it might be good to have
> some sort of detailed sketch about the key distribution of an intermediate
> result. I am not sure whether a simple histogram is the most effective way to
> go. Maybe somebody would propose another lightweight sketch that provides
> better accuracy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
