dongjoon-hyun commented on code in PR #58054:
URL: https://github.com/apache/spark/pull/58054#discussion_r3825245402
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -2143,6 +2151,23 @@ private[spark] class DAGScheduler(
return
}
+ // While the executors are held there are zero free slots by construction,
so the terminal
+ // gang admission check below would fail any pipelined job outright. Defer
the submission
+ // and re-check on a timer instead, like a barrier job: the shortfall is
not transient
+ // scarcity, and the job should wait for the resume.
+ if (hasPipelined && executorsHeld) {
Review Comment:
Correcting the reply above: the warm-up-budget fix it describes was
subsequently removed in 10732ea, together with the deferral itself -- the PR
now excludes pipelined-shuffle jobs from the hold's scope, as documented on the
finding-29 thread. This finding was what tipped that decision: the deferral
needed to be correct across the asynchronous post-resume warm-up, and each fix
there (including this one) narrowed the window rather than closing the class.
With the deferral gone there is no re-post to race the resume; a pipelined job
submitted while held fails its gang admission immediately and is resubmitted
after the resume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
