Zhu Zhu created FLINK-14163:
-------------------------------
Summary: A Task should be deployed only after all its partitions
have completed the registration
Key: FLINK-14163
URL: https://issues.apache.org/jira/browse/FLINK-14163
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu
Fix For: 1.10.0
Currently {{Execution#producedPartitions}} is assigned after the partitions
have completed the registration to shuffle master in
{{Execution#registerProducedPartitions(...)}}.
But the task deployment process (in {{Execution#deploy()) will create
{{ResultPartitionDeploymentDescriptor}} directly from
{{Execution#producedPartitions}} without checking whether it's assigned.
This may lead to a task deployed without its result partitions. And eventually
cause the job to hang.
It is not problematic at the moment when using Flink default shuffle master
{{NettyShuffleMaster}} since it returns a completed future on registration.
However, if the behavior is changed or if users are using a customized
{{ShuffleMaster}}, it may cause problems.
Besides that, {{Execution#producedPartitions}} is also used for
* generating downstream task input descriptor
* retrieve {{ResultPartitionID}} for partition releasing
To avoid issues to happen, we may need to change all the usages of
{{Execution#producedPartitions} to a callback way, e.g. change
{{Execution#producedPartitions} from {{Map<IntermediateResultPartitionID,
ResultPartitionDeploymentDescriptor>}} to
{{CompletableFuture<Map<IntermediateResultPartitionID,
ResultPartitionDeploymentDescriptor>>}} and adjust all its usages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
