psavalle opened a new pull request, #59713: URL: https://github.com/apache/spark/pull/59713
### What changes were proposed in this pull request? The PR ties Python worker factories to the session lifecycle: - Adds the session artifact UUID and driver/executor ownership to each factory’s cache key. - On executor session eviction, stops and removes matching task-worker factories before deleting session files. - Cleans driver-side planner factories when the session’s `ArtifactManager` closes. - Ensures workers finishing after their factory was removed are closed instead of returned to a stale pool. - Stops daemon monitor threads and worker processes when a factory is shut down. This prevents reuse of daemons whose working directory was deleted and avoids per-session process/thread leaks. ### Why are the changes needed? This addresses: - [SPARK-54055](https://issues.apache.org/jira/browse/SPARK-54055): Spark Connect sessions leak pyspark UDF daemon processes and threads - [SPARK-50118](https://issues.apache.org/jira/browse/SPARK-50118): Spark removes working directory while Python UDF runs. There was an attempt in the past to fix this, but it was eventually reverted. Spark caches a separate `PythonWorkerFactory` for each combination of Python configuration and session artifact UUID. Each factory may own a long-live Python daemon, reusable Python workers, an idle-worker monitor thread. For session-isolated workloads, those processes are launched with the session’s artifact directory as their working directory, for example: ``` <SparkFiles root>/<session UUID>/ ``` We can hit the following scenario: 1. An executor runs Python tasks for an isolated Spark session. 2. Spark creates and caches a `PythonWorkerFactory`, its daemon starts inside the session directory. 3. The executor’s isolated-session cache later evicts that session. 4. `IsolatedSessionState.cleanup()` deletes the session directory and classloader resources. 5. The corresponding Python worker factory is not removed or stopped. The daemon therefore remains alive with its current working directory pointing to a directory that has been deleted. If the same session is used again, Spark may reuse that factory and daemon ,which can lead to issues. Even if the session is never used again, every evicted session can leave behind resources (daemon/worker processes, monitor thread) forever. ### Does this PR introduce _any_ user-facing change? This fixes bugs/leaks. ### How was this patch tested? Unit and integration tests. ### Was this patch authored or co-authored using generative AI tooling? `Generated-by: codex-cli 0.161.0` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
