j1wonpark opened a new pull request, #57651: URL: https://github.com/apache/spark/pull/57651
### What changes were proposed in this pull request? This PR skips pending-task bookkeeping for `TaskStart` and `SpeculativeTaskSubmitted` events when the corresponding stage attempt has already completed. Running-task accounting remains unchanged for late `TaskStart` events so that matching `TaskEnd` events are handled correctly. Regression tests were added for regular task starts, speculative task starts, and speculative task submissions after `StageCompleted`. ### Why are the changes needed? After `StageCompleted` removes the stage metadata, a late `TaskStart` event can recreate task-index state. Processing the matching `TaskEnd` can then produce a negative pending task count. When dynamic allocation uses `minExecutors=0`, this can leave the executor target at zero and prevent later stages from requesting executors. ### Does this PR introduce _any_ user-facing change? Yes. Late task events from completed stages no longer prevent subsequent stages from requesting executors when dynamic allocation is enabled. ### How was this patch tested? Added regression tests covering late regular task starts, speculative task starts, and speculative task submissions. The three new tests failed against `upstream/master` and passed with this patch. ```bash build/sbt 'core/testOnly org.apache.spark.ExecutorAllocationManagerSuite' ``` All 40 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5.6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
