miaoever opened a new pull request, #59229: URL: https://github.com/apache/spark/pull/59229
### What changes were proposed in this pull request? Update `ExecutorAllocationListener` so an explicitly killed, unfinished regular task is counted as pending again and restarts the scheduler backlog timer. The listener records task indices completed by speculative attempts so that a later `TaskKilled` event for the corresponding regular attempt remains a no-op, preserving the behavior required by SPARK-30511. Killed speculative attempts and events from completed stages also remain no-ops. ### Why are the changes needed? When a regular task starts while the executor target is zero, it can drain the scheduler queue and clear the backlog timer. If that task is killed before succeeding, the scheduler requeues it, but the listener currently ignores `TaskKilled`. The task is therefore not counted as pending and the timer is not restarted. If the available executor is then lost, dynamic allocation may not request a replacement and the stage can stall. See [SPARK-59972](https://issues.apache.org/jira/browse/SPARK-59972). ### Does this PR introduce _any_ user-facing change? Yes. Dynamic allocation now requests an executor for a requeued unfinished regular task after its previous attempt is explicitly killed. ### How was this patch tested? Added `ExecutorAllocationManagerSuite` regression tests covering: - an unfinished regular task being killed; - a speculative attempt being killed; - a regular attempt being killed after a speculative copy succeeds; and - a killed-task event arriving after stage completion. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
