dongjoon-hyun opened a new pull request, #59092: URL: https://github.com/apache/spark/pull/59092
### What changes were proposed in this pull request? This PR aims to fix two `Resubmitted` accounting bugs in `AppStatusListener.onTaskEnd`. When an executor is lost, `TaskSetManager.executorLost` posts `SparkListenerTaskEnd` with `Resubmitted` reason for the already-successful shuffle map tasks of that executor (without external shuffle service or on decommission) if the task set is not zombie. This event reuses the original finished `TaskInfo` and has no matching `SparkListenerTaskStart`. Since SPARK-41187 (apache/spark#38702), `onTaskEnd` uses `activeDelta = 0` for `Resubmitted`, but the following places still handle it like a normal task end. 1. The speculation summary decrements `numActiveTasks` unconditionally. This PR uses `activeDelta` like the other active task counters. 2. `ExecutorStageSummary.taskTime` and `ExecutorSummary.totalDuration` add `event.taskInfo.duration` again although the duration of the successful attempt was already counted. This PR skips them for `Resubmitted` like the executor-level task metrics, which are already skipped for `Resubmitted`. ### Why are the changes needed? To show correct values. - When a successful speculative task is resubmitted, `numActiveTasks` of the speculation summary becomes negative (e.g. `-1`), and nothing corrects it later because only `onTaskStart` and `onTaskEnd` update it. This value is stored in `SpeculationStageSummaryWrapper` and shown via the REST API `speculationSummary` field and the `Speculation Summary` table of the stage page. - For every `Resubmitted` task end, the duration of the successful attempt is counted twice in `ExecutorStageSummary.taskTime` (`Task Time` of `Aggregated Metrics by Executor` table in the stage page) and `ExecutorSummary.totalDuration` (`Task Time (GC Time)` in the executors page and `totalDuration_seconds_total` in the Prometheus metrics). Note that the old `ExecutorsListener` of Apache Spark 2.2 ignored `Resubmitted` task ends including the duration, but SPARK-20646 (apache/spark#19678) kept only the task metrics part when porting it to `AppStatusListener`. These were found during the review of apache/spark#58968. ### Does this PR introduce _any_ user-facing change? Yes, this fixes the incorrect values in the Spark UI, REST API, and Prometheus metrics. The negative `numActiveTasks` of the speculation summary exists since Apache Spark [v3.3.0](https://github.com/apache/spark/releases/tag/v3.3.0) (2022-06-09), and the double-counted task duration exists since Apache Spark [v2.3.0](https://github.com/apache/spark/releases/tag/v2.3.0) (2018-02-22). ### How was this patch tested? Pass the CIs with the newly added test case. I also manually verified that the new test case fails without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
