venkata91 opened a new pull request, #59083: URL: https://github.com/apache/spark/pull/59083
### What changes were proposed in this pull request? - Adds a best-effort guard in `SparkContext.holdExecutors()`, beside the pipelined-job check, that refuses the hold while an active shuffle's output is not reliably stored and the external shuffle service is off. - Reads `MapOutputTrackerMaster.hasUnreliablyStoredShuffle`; skipped when the shuffle service is on, since all output is then off-executor. - Stacked on #58437 (SPARK-59138); its commits appear here until that merges. ### Why are the changes needed? - `executorHoldSupported` admits a deployment when `ShuffleDataIO` reports reliable storage, but an individual shuffle can still fall back to local-disk output. - Draining the executors then discards that only copy and forces a recompute. This makes the hold storage-aware, matching the DA scale-down in #58687 (SPARK-59806). ### Does this PR introduce _any_ user-facing change? - No. `holdExecutors()` is an unreleased API; this only refines when it returns false. ### How was this patch tested? - New `SparkContextSuite` test "SPARK-59807: holdExecutors is refused while an unreliably-stored shuffle is active": under reliable `ShuffleDataIO` with the service off, an unreliable shuffle makes the hold return false; unregistering it lets the hold proceed. ### Was this patch authored or co-authored using generative AI tooling? - Generated-by: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
