[
https://issues.apache.org/jira/browse/BEAM-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055416#comment-16055416
]
Jingsong Lee commented on BEAM-2477:
------------------------------------
*Local combine*: Cloud Dataflow/Flink Batch optimizes Combine operations (such
as Count and Sum) by performing partial combining locally before sending the
data to the main grouping operation. Graph optimizations in
https://cloud.google.com/blog/big-data/2017/05/after-lambda-exactly-once-processing-in-cloud-dataflow-part-2-ensuring-low-latency
*Incremental aggregation*: Similar to Flink's concept,
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#windowfunction-with-incremental-aggregation
While the GroupByKey will keep the details of elements until the window closes.
(AFAIK in Flink Runner)
> BeamAggregationRel should use Combine.perKey instead of GroupByKey
> ------------------------------------------------------------------
>
> Key: BEAM-2477
> URL: https://issues.apache.org/jira/browse/BEAM-2477
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Labels: dsl_sql_merge
>
> Their semantics are the same, but the efficiency of implementation is quite
> different, and at the runner level there is a lot of optimization for
> `Combine.perKey`.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
