jerrypeng commented on code in PR #59523:
URL: https://github.com/apache/spark/pull/59523#discussion_r4408573848
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -2665,19 +2687,17 @@ private[spark] class DAGScheduler(
return
}
- // Gang admission for a pipelined job: the runnable stage graph is one
pipelined group (a
- // materialized prefix's stages never run), so check up front (before any
stage is created)
- // that the cluster can run the entire group concurrently. If it cannot
fit, fail the job now
- // -- no partial scheduler state, and no member ever left running while a
sibling waits on
- // slots (true all-or-nothing gang admission). Inert for a regular job (no
pipelined
- // dependency).
- if (hasPipelined && rejectUnadmittablePipelinedGroup(jobId, finalRDD,
partitions, listener)) {
- // Clear the slot-check retry count an earlier deferral may have left.
- barrierJobIdToNumTasksCheckFailures.remove(jobId)
- return
- }
var finalStage: ResultStage = null
try {
+ if (hasPipelined) {
+ checkPipelinedCacheSupported(finalRDD)
+ // Validate cache semantics before counting tasks: a cached consumer
could suppress
Review Comment:
Agreed. I combined the hasPipelined blocks, moved the cache rationale above
its call, and restored the retry-counter cleanup comment. The validation order
is unchanged.
A regression verifies that malformed admission configuration fails only the
job, leaves the context running, and permits successful resubmission after
correction. This exception behavior was already introduced by the PR. The
prepared description addition documents it and remains unpublished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
