jon-gao-db opened a new pull request, #57941: URL: https://github.com/apache/spark/pull/57941
### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
