lan1994 opened a new issue, #57494: URL: https://github.com/apache/spark/issues/57494
**Version** spark 3.5+ (k8s) **Description** I observed a behavior difference between Spark on Kubernetes and YARN for executor heartbeat-timeout scenarios. In heartbeat timeout handling, HeartbeatReceiver triggers both: 1. sc.killAndReplaceExecutor(executorId) (which uses countFailures=true) 2. RemoveExecutor(executorId, ExecutorProcessLost(timeout)) However, in Kubernetes backend, doKillExecutors() historically removes executors eagerly with ExecutorKilled. This can win the race against the later timeout-based RemoveExecutor(...ExecutorProcessLost...), so the final loss reason becomes ExecutorKilled. Since ExecutorKilled is treated as non-app-caused failure (exitCausedByApp=false), task failures may not be counted towards spark.task.maxFailures, and fast-fail is less likely to happen. **Impact** 1. Heartbeat-timeout task failures may bypass failure counting. 2. Jobs can retry for a long time instead of fast-failing. 3. Behavior diverges from YARN under similar failure conditions. **Suggested fix direction** In K8s doKillExecutors, preserve countFailures semantics from killExecutors: Keep ExecutorKilled for explicit driver-initiated non-counting kills. For replacement kills (countFailures=true), avoid eagerly converting to ExecutorKilled; keep a non-ExecutorKilled reason (or allow timeout reason to land) so task-failure counting semantics are preserved. If this direction is acceptable to maintainers, I can contribute a PR with a minimal fix and corresponding unit tests. **Additional question:** Could maintainers advise how to obtain an Apache JIRA account for new contributors? <img width="583" height="279" alt="Image" src="https://github.com/user-attachments/assets/06ad189c-2cf7-42bc-8e42-38d12e135ef5"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
