[
https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363513#comment-14363513
]
ASF GitHub Bot commented on FLINK-1671:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/487
[FLINK-1671] Add different execution modes to APIs and optimizer.
This allows users to specify how programs should be executed:
- PIPELINED: Uses pipelining for shuffling, except where the data
exchange is deadlock prone.
- BATCH: Used batch for shuffles, broadcasts, rebalancing.
- BATCH_FORCED: Uses batch everywhere, effectively prohibits chaining
(debug only, IMHO)
- PIPELINED_FORCED: Uses pipelining everywhere, also in deadlock prone
situations.
I fixed and improved a lot of comments on-the-fly while implementing this.
Because of that, the PR is slightly bigger.
NOTE: This is only the API, and optimizer implementation of selecting the
proper data exchange mode. The runtime implementation of blocking data exchange
is a separate pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink modes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/487.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #487
----
commit 62a41f82598a75af76956f971bf05a994a073b7b
Author: Stephan Ewen <[email protected]>
Date: 2015-03-10T14:14:09Z
[optimizer] Remove obsolete and unused fields from OptimizerNode
commit 11a7449714771532fd8b101af07ef3e0bd289a21
Author: Stephan Ewen <[email protected]>
Date: 2015-03-10T17:22:20Z
[FLINK-1671] [jobmanager] Rename JobManager's ExecutionMode to
JobManagerMode
This is done to avoid name conflicts with the overloaded type name
ExecutionMode.
commit 6f9ab0b9d3370ca2dd18de371d1fae025e60561e
Author: Stephan Ewen <[email protected]>
Date: 2015-03-10T17:36:52Z
[FLINK-1671] [tests] Rename "ExecutionMode" in MultiProgramTestBase to
"TestExecutionMode"
... to avoid name clashes with the program execution mode.
commit 537b06602b194e708d61203e5a8b43ecaf96024e
Author: Stephan Ewen <[email protected]>
Date: 2015-03-10T19:25:01Z
[FLINK-1671] [core] Add execution mode to execution config
commit 566a2bbc38e6b06a43b4efe99834b9aadec062d6
Author: Stephan Ewen <[email protected]>
Date: 2015-03-11T22:35:11Z
[FLINK-1671] [optimizer] Add data exchange mode to optimizer classes
----
> Add execution modes for programs
> --------------------------------
>
> Key: FLINK-1671
> URL: https://issues.apache.org/jira/browse/FLINK-1671
> Project: Flink
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, there is a single way that programs get executed: Pipelined. With
> the new code for batch shuffles (https://github.com/apache/flink/pull/471),
> we have much more flexibility and I would like to expose that.
> I suggest to add more execution modes that can be chosen on the
> `ExecutionEnvironment`:
> - {{BATCH}} A mode where every shuffle is executed in a batch way, meaning
> preceding operators must be done before successors start. Only for the batch
> programs (d'oh).
> - {{PIPELINED}} This is the mode corresponding to the current execution
> mode. It pipelines where possible and batches, where deadlocks would
> otherwise happen. Initially, I would make this the default (be close to the
> current behavior). Only available for batch programs.
> - {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining
> shuffles and fall back to batch shuffles upon failure and recovery, or once
> it sees that not enough slots are available to bring up all operators at once
> (requirement for pipelining).
> - {{STREAMING}} This is the default and only way for streaming programs.
> All communication is pipelined, and the special streaming checkpointing code
> is activated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
