Ufuk Celebi created FLINK-1833:
----------------------------------
Summary: Refactor partition availability notification in
ExecutionGraph
Key: FLINK-1833
URL: https://issues.apache.org/jira/browse/FLINK-1833
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime, JobManager
Affects Versions: master
Reporter: Ufuk Celebi
The mechanism to notify the JobManager about available result partitions is
hard to understand. The are two parts to this:
1) JobManager
- The deployment of receivers happens in the Execution class although it is by
now totally unrelated to the state of a specific execution. I propose to move
this to the respective IntermediateResultPartition.
- The deployment information for a receiver is spread across different
components: when creating the TaskDeploymentDescriptor and the "caching" of
partition infos at the consuming vertex. This is very hard to follow and
results in unnecessary messages being sent (which are discarded at the TM).
2) TaskManager
- Pipelined results notify where you would expect it in the ResultPartition,
but blocking results don't have an extra message and are implicitly
piggy-backed to the final state transition, after which the job manager deploys
receivers if all blocking partitions of a result have been produced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
