[
https://issues.apache.org/jira/browse/FLINK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354633#comment-14354633
]
ASF GitHub Bot commented on FLINK-1350:
---------------------------------------
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/471
[FLINK-1350] [runtime] Add blocking result partition variant
- **Renames** runtime intermediate result classes (after an offline
discussion with @StephanEwen I realized that calling subpartitions *queue* was
rather confusing):
a) Removes "Intermediate" prefix
b) Queue => Subpartition
c) Iterator => View
Documentation is coming up soon in FLINK-1373.
- **[FLINK-1350](https://issues.apache.org/jira/browse/FLINK-1350)**: Adds
a *spillable result subpartition variant* for **BLOCKING results**, which
writes data to memory first and starts to spill (asynchronously) if not enough
memory is available to produce the result in-memory only.
Receiving tasks of BLOCKING results are only deployed after *all*
partitions have been fully produced. PIPELINED and BLOCKING results can not be
mixed.
@StephanEwen, what is your opinion on these two points? We could also start
deploying receivers of blocking results as soon as a single partition is
finished and then send the update RPC calls later. The current solution is
simpler and results in less RPC calls (single per parallel producer).
- **[FLINK-1359](https://issues.apache.org/jira/browse/FLINK-1359)**: Adds
**simple state tracking** to result partitions with notifications after
partitions/subpartitions have been consumed. In normal operation, each
partition has to be consumed at least once before it can be released.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/incubator-flink
blocking_results-flink-1350
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/471.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #471
----
commit cbe71d72378dc3a8546dd0322f8712e7352b7a08
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-01-06T16:11:08Z
[FLINK-1350] [runtime] Add blocking result partition variant
- Renames runtime intermediate result classes:
a) Removes "Intermediate" prefix
b) Queue => Subpartition
c) Iterator => View
- [FLINK-1350] Adds a spillable result subpartition variant for BLOCKING
results, which writes data to memory first and starts to spill
(asynchronously) if not enough memory is available to produce the
result in-memory only.
Receiving tasks of BLOCKING results are only deployed after *all*
partitions have been fully produced. PIPELINED and BLOCKING results can
not
be mixed.
- [FLINK-1359] Adds simple state tracking to result partitions with
notifications after partitions/subpartitions have been consumed. Each
partition has to be consumed at least once before it can be released.
Currently there is no notion of historic intermediate results, i.e.
results
are released as soon as they are consumed.
commit 878cf9cecf41877eb82aa3a34f3266c2d4760bcd
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-03-09T14:29:06Z
[FLINK-1350] [runtime] Send final task state update only after
unregistering task
commit 315d96ed93a2b52ab9f0ac6bdf6727f497488478
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-03-09T15:17:25Z
[FLINK-1350] [runtime] Deploy receivers of blocking results only after all
producers finished
commit 66f7c4c0286e6b380107d5b7a22568ffbb3a8011
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-03-09T15:47:17Z
[runtime] Update Netty version to 4.0.26.Final
commit fccfeca84f9bc990f11139cce3cd381d3f04dbd4
Author: Ufuk Celebi <u...@apache.org>
Date: 2015-03-10T09:50:46Z
[tests] Increase number of network buffers in
ProcessFailureBatchRecoveryITCase
This is necessary, because the network thread has an extra buffer pool to
read
data into from spilled partitions.
----
> Add blocking intermediate result partitions
> -------------------------------------------
>
> Key: FLINK-1350
> URL: https://issues.apache.org/jira/browse/FLINK-1350
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Reporter: Ufuk Celebi
> Assignee: Ufuk Celebi
>
> The current state of runtime support for intermediate results (see
> https://github.com/apache/incubator-flink/pull/254 and FLINK-986) only
> supports pipelined intermediate results (with back pressure), which are
> consumed as they are being produced.
> The next variant we need to support are blocking intermediate results
> (without back pressure), which are fully produced before being consumed. This
> is for example desirable in situations, where we currently may run into
> deadlocks when running pipelined.
> I will start working on this on top of my pending pull request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
