wuyi created SPARK-32518:
----------------------------
Summary: CoarseGrainedSchedulerBackend.maxNumConcurrentTasks
should consider all kinds of resources
Key: SPARK-32518
URL: https://issues.apache.org/jira/browse/SPARK-32518
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.0
Reporter: wuyi
Currently, CoarseGrainedSchedulerBackend.maxNumConcurrentTasks only considers
the CPU for the max concurrent tasks. This can cause the application to hang
when a barrier stage requires extra custom resources but the cluster doesn't
have enough corresponding resources. Because, without the checking for other
custom resources in maxNumConcurrentTasks, the barrier stage can be submitted
to the TaskSchedulerImpl. But the TaskSchedulerImpl can not launch tasks for
the barrier stage due to the insufficient task slots calculated by
calculateAvailableSlots(which does check all kinds of resources).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
