Ngone51 commented on a change in pull request #28260:
URL: https://github.com/apache/spark/pull/28260#discussion_r411439057
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -588,13 +594,33 @@ private[spark] class TaskSchedulerImpl(
-1
}
// Skip the barrier taskSet if the available slots are less than the
number of pending tasks.
- if (taskSet.isBarrier && numBarrierSlotsAvailable < taskSet.numTasks) {
- // Skip the launch process.
- // TODO SPARK-24819 If the job requires more slots than available
(both busy and free
- // slots), fail the job on submit.
- logInfo(s"Skip current round of resource offers for barrier stage
${taskSet.stageId} " +
- s"because the barrier taskSet requires ${taskSet.numTasks} slots,
while the total " +
- s"number of available slots is $numBarrierSlotsAvailable.")
+ if (taskSet.isBarrier) {
+ val enoughSlots = numBarrierSlotsAvailable >= taskSet.numTasks
Review comment:
TODO: should be possible maximum slots instead of current available
slots.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
