dbtsai commented on PR #56907:
URL: https://github.com/apache/spark/pull/56907#issuecomment-4858648485
## Review feedback (follow-up)
### Timeout path can orphan the child JVM
(`python/pyspark/sql/connect/session.py:1523`)
On the 120s start-up timeout path, `proc.terminate()` sends SIGTERM only to
the detached daemon Python process, which was started with
`start_new_session=True` (a new session / process-group leader). That daemon
launches a child JVM (SparkContext) inside `local_server.py:main()`'s
`builder.getOrCreate()`, and its SIGTERM handler + `spark.stop()` cleanup are
only wired up *after* `getOrCreate()` returns.
If the timeout fires while the daemon is still inside `getOrCreate()` —
exactly the slow/hung-startup case that triggers the timeout — terminating just
the daemon leader leaves the JVM orphaned with no cleanup. The leaked JVM keeps
consuming memory/CPU and may later finish binding the requested port, confusing
subsequent reuse attempts. This is most likely to hit precisely when the host
is already resource-constrained.
Suggested fix: signal the whole process group rather than just the daemon
PID, and escalate to kill if it does not exit, so the orphaned JVM is reaped.
On POSIX, kill the session/group created by `start_new_session=True`; on the
daemon side, install the SIGTERM handler / cleanup before or around the JVM
launch.
```python
try:
if os.name == "posix":
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
else:
proc.terminate()
try:
proc.wait(timeout=10)
except Exception:
if os.name == "posix":
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
else:
proc.kill()
except OSError:
pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
