wangyum opened a new pull request, #57525: URL: https://github.com/apache/spark/pull/57525
### What changes were proposed in this pull request? This PR adds an executor-side app-ID verification that runs immediately after the executor fetches the driver's Spark config via `RetrieveSparkAppConfig`, before `SparkEnv` is created. The new `CoarseGrainedExecutorBackend.verifyAppId` compares the executor's `--app-id` launch argument (assigned by the cluster manager) against `spark.app.id` from the driver's `SparkConf` (set in `SparkContext`), and throws `SparkException` on mismatch. The check is invoked in both code paths that retrieve the driver config: - `CoarseGrainedExecutorBackend.run()` — covers Standalone and YARN (YARN delegates to the shared `run()` via `YarnCoarseGrainedExecutorBackend`). - `KubernetesExecutorBackend.run()` — separate implementation, so the call is added there explicitly. The verification is unconditional (always on). It only fires when `spark.app.id` is present in the driver properties **and** differs from the executor's app ID — a condition that never occurs in normal operation. ### Why are the changes needed? When a driver hits a fatal error (e.g., OOM) that kills the RPC dispatcher thread, `SparkContext.stop()` releases the driver's RPC port while the driver process (and in YARN cluster mode, the ApplicationMaster) can remain in a zombie state — still heartbeating to the resource manager but effectively dead. During this window, the freed port can be rebound by another driver on the same host, and newly launched executors can connect to the wrong driver and register to the wrong application, risking data corruption. ### Does this PR introduce _any_ user-facing change? Yes. In the rare case of an app-ID mismatch, the executor will now exit immediately at startup with a `SparkException` instead of registering to the wrong driver. The error message is: ``` Executor app ID <executorAppId> does not match driver app ID <driverAppId>. This likely means the executor connected to the wrong driver. Exiting. ``` No config or API changes. No impact on normal operation. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GLM 5.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
