[
https://issues.apache.org/jira/browse/FLINK-22677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375217#comment-17375217
]
Zhu Zhu commented on FLINK-22677:
---------------------------------
Problems below could happen if enabling partition registration is async:
1. task is deployed before its produced partitions are registered. this can
result in a bad {{ResultPartitionDeploymentDescriptor}}
2. consumer task is deployed before its consumed partitions are registered, if
the consumer and producer are in the same region
3. partitions can be leaked in {{JobMasterPartitionTracker}} if the producer
task is failed/canceled before registration is done
4. partition can be leaked in the shuffle service if the producer task is
failed/canceled before registration is done
To solve this problem, I'd propose to:
1. DefaultScheduler deploys a task only after all its partitions have completed
registration. This solve problem #1 and #2. #2 can be solved because we now
always deploy in topologically order, which means when a task is deployed, all
its upstream tasks are deployed and their partitions will be registered already.
2. Check the producer state on partition registration completion. If the
producer is not {{SCHEDULED}}, directly ask the {{ShuffleMaster}} to release
the partition and do not track it
A PR is opened as proposed above.
> Scheduler should invoke ShuffleMaster#registerPartitionWithProducer by a real
> asynchronous fashion
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-22677
> URL: https://issues.apache.org/jira/browse/FLINK-22677
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.14.0
> Reporter: Jin Xing
> Assignee: Zhu Zhu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> Current scheduler enforces a synchronous registration though the API of
> ShuffleMaster#registerPartitionWithProducer returns a CompletableFuture. In
> scenario of remote shuffle service, the talk between ShuffleMaster and remote
> cluster tends to be expensive. A synchronous registration risks to block main
> thread potentially and might cause negative side effects like heartbeat
> timeout. Additionally, expensive synchronous invokes to remote could
> bottleneck the throughput for applying shuffle resource, especially for batch
> jobs with complicated DAGs;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
