zahed1994 opened a new pull request, #58380: URL: https://github.com/apache/spark/pull/58380
### What changes were proposed in this pull request? When `spark.dynamicAllocation.enabled=true` and `spark.speculation=true`, straggling tasks requiring speculative execution can stall indefinitely if all remaining active executors reside on the same host. In `ExecutorAllocationManager.scala`, `maxNumExecutorsNeededPerResourceProfile` calculates `maxNeeded` strictly based on `(running + pendingTasks + pendingSpeculative) / tasksPerExecutor`. If the current active executor count equals `maxNeeded`, `ExecutorAllocationManager` calculates target executors as equal to the active count and requests no new executors. At the same time, Spark's task scheduler avoids launching speculative task copies on an executor on the same host where the task is already running slow. Consequently, no active executor can run the speculative task and no new executor is requested, causing speculative tasks to stall indefinitely. This PR updates `maxNumExecutorsNeededPerResourceProfile` in `ExecutorAllocationManager.scala` to allocate an additional target executor when `pendingSpeculative > 0` and `maxNeeded` equals the current active executor count. This allows `ExecutorAllocationManager` to request an extra executor from the cluster manager (YARN / K8s / Standalone) on a distinct host to execute the speculative task. ### Why are the changes needed? Without this change, applications using Dynamic Allocation and speculation can hang indefinitely when remaining executors reside on the same slow host. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Added unit test `SPARK-49485: request additional executor when speculative tasks equal maxNeeded` in `ExecutorAllocationManagerSuite.scala`. - Verified cleanly via `core/compile` and `core/scalastyle`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
