zahed1994 commented on PR #58380: URL: https://github.com/apache/spark/pull/58380#issuecomment-5819035552
Thanks @sunchao, good catch on the stage retry path. I've updated the PR accordingly in commit 19e31aab941: - **Retain running speculative IDs across `onStageCompleted`**: Removed `stageAttemptToSpeculativeTaskIds -= stageAttempt` from `onStageCompleted` (mirroring `stageAttemptToNumRunningTask`). Running speculative copies remain counted in `runningSpeculativeTasksPerResourceProfile`, preventing the target from dropping (3→2) when a stage completes early for retry. - **Cleanup on `onTaskEnd`**: When a speculative task finishes in `onTaskEnd`, its `taskId` is removed. Once the set is empty, the entry is removed from `stageAttemptToSpeculativeTaskIds` and `removeStageFromResourceProfileIfUnused` is triggered so the resource profile mapping cleans up cleanly. - **Regression test**: Added `"SPARK-49485: preserve running speculative attempts across stage retries"` in `ExecutorAllocationManagerSuite` verifying target retention at 3 across `StageCompleted` until `TaskEnd`. All 51 tests in `ExecutorAllocationManagerSuite` and `scalastyle` pass cleanly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
