[
https://issues.apache.org/jira/browse/FLINK-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363036#comment-14363036
]
ASF GitHub Bot commented on FLINK-1618:
---------------------------------------
GitHub user gyfora opened a pull request:
https://github.com/apache/flink/pull/485
[FLINK-1618] [streaming] Parallel time reduce
This commit introduces a new critical feature for the windowing api, which
allows parallel discretization and reduce over Time windows.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mbalassi/flink FLINK-1618
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/485.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #485
----
commit d52a04f8e66f110406adb8976be00822155009db
Author: Gyula Fora <[email protected]>
Date: 2015-03-16T08:52:41Z
[FLINK-1618] [streaming] Parallel time reduce
----
> Add parallel time discretisation for time-window transformations
> -----------------------------------------------------------------
>
> Key: FLINK-1618
> URL: https://issues.apache.org/jira/browse/FLINK-1618
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Reporter: Gyula Fora
> Assignee: Gyula Fora
>
> Currently discretizers for all windowing policies including time are executed
> with parallelism 1 when they define global windows. (for instance: sum of the
> last 10 minutes)
> While this is necessary for arbitrary policies like delta based or
> user-defined policies. Some discretizers such as Time can be implemented in a
> distributed fashion.
> Distributed time discretisers (and other types) can be implemented in the
> following way:
> -The discretisers should create StreamWindow s with incrementally increasing
> ID-s starting from the same value so that it is possible to merge them after
> the transformation
> - The partitioner for each discretizer should send the number of partitions
> created to the merger (the merger should be aware of the number of
> partitioners present to wait for all the information)
> - Based on all the partitioning info the merger can merge the windows
> properly afterwards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
