[
https://issues.apache.org/jira/browse/FLINK-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels resolved FLINK-1621.
---------------------------------------
Resolution: Fixed
Assignee: Maximilian Michels
Resolved by implementing a GroupCombine operator which, together with the
GroupReduce operator, enables a form of the generalized combine found in Google
Dataflow.
> Create a generalized combine function
> -------------------------------------
>
> Key: FLINK-1621
> URL: https://issues.apache.org/jira/browse/FLINK-1621
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Affects Versions: 0.9
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Labels: features, performance
> Fix For: 0.9
>
>
> Flink allows combiners which accept a type {{I}} and combine the values of
> this type into type {{O}}. In Google Dataflow, combiners are more
> generalized. They accept an Input {{I}}, produce an intermediate combine
> value of {{T}}, and finally an output {{O}}. Flink's combiners are like the
> {{SimpleCombineFn}} in Google Dataflow.
> Right now, we translate the {{KeyedCombineFn}} into a {{SortPartition}}
> followed by a {{MapPartition}} to emulate the Combiner's behavior.
> Rudimentary performance tests showed that this behavior causes a significant
> increase in run time compared to the proper Combine implementation.
> Let's implement a more generalized Combiner to create a better mapping from
> Google Dataflow to Flink.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
