GitHub user uce opened a pull request: https://github.com/apache/flink/pull/471
[FLINK-1350] [runtime] Add blocking result partition variant - **Renames** runtime intermediate result classes (after an offline discussion with @StephanEwen I realized that calling subpartitions *queue* was rather confusing): a) Removes "Intermediate" prefix b) Queue => Subpartition c) Iterator => View Documentation is coming up soon in FLINK-1373. - **[FLINK-1350](https://issues.apache.org/jira/browse/FLINK-1350)**: Adds a *spillable result subpartition variant* for **BLOCKING results**, which writes data to memory first and starts to spill (asynchronously) if not enough memory is available to produce the result in-memory only. Receiving tasks of BLOCKING results are only deployed after *all* partitions have been fully produced. PIPELINED and BLOCKING results can not be mixed. @StephanEwen, what is your opinion on these two points? We could also start deploying receivers of blocking results as soon as a single partition is finished and then send the update RPC calls later. The current solution is simpler and results in less RPC calls (single per parallel producer). - **[FLINK-1359](https://issues.apache.org/jira/browse/FLINK-1359)**: Adds **simple state tracking** to result partitions with notifications after partitions/subpartitions have been consumed. In normal operation, each partition has to be consumed at least once before it can be released. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/incubator-flink blocking_results-flink-1350 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/471.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #471 ---- commit cbe71d72378dc3a8546dd0322f8712e7352b7a08 Author: Ufuk Celebi <u...@apache.org> Date: 2015-01-06T16:11:08Z [FLINK-1350] [runtime] Add blocking result partition variant - Renames runtime intermediate result classes: a) Removes "Intermediate" prefix b) Queue => Subpartition c) Iterator => View - [FLINK-1350] Adds a spillable result subpartition variant for BLOCKING results, which writes data to memory first and starts to spill (asynchronously) if not enough memory is available to produce the result in-memory only. Receiving tasks of BLOCKING results are only deployed after *all* partitions have been fully produced. PIPELINED and BLOCKING results can not be mixed. - [FLINK-1359] Adds simple state tracking to result partitions with notifications after partitions/subpartitions have been consumed. Each partition has to be consumed at least once before it can be released. Currently there is no notion of historic intermediate results, i.e. results are released as soon as they are consumed. commit 878cf9cecf41877eb82aa3a34f3266c2d4760bcd Author: Ufuk Celebi <u...@apache.org> Date: 2015-03-09T14:29:06Z [FLINK-1350] [runtime] Send final task state update only after unregistering task commit 315d96ed93a2b52ab9f0ac6bdf6727f497488478 Author: Ufuk Celebi <u...@apache.org> Date: 2015-03-09T15:17:25Z [FLINK-1350] [runtime] Deploy receivers of blocking results only after all producers finished commit 66f7c4c0286e6b380107d5b7a22568ffbb3a8011 Author: Ufuk Celebi <u...@apache.org> Date: 2015-03-09T15:47:17Z [runtime] Update Netty version to 4.0.26.Final commit fccfeca84f9bc990f11139cce3cd381d3f04dbd4 Author: Ufuk Celebi <u...@apache.org> Date: 2015-03-10T09:50:46Z [tests] Increase number of network buffers in ProcessFailureBatchRecoveryITCase This is necessary, because the network thread has an extra buffer pool to read data into from spilled partitions. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---