Jonathan Gao created SPARK-58977:
------------------------------------
Summary: TransformWithState in PySpark state server reports
spurious daemon crash during early shutdown
Key: SPARK-58977
URL: https://issues.apache.org/jira/browse/SPARK-58977
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark
Affects Versions: 4.0.0
Reporter: Jonathan Gao
This is a follow-up to SPARK-58751 and
https://github.com/apache/spark/pull/57941.
TransformWithState in PySpark starts a JVM state-server listener that blocks in
ServerSocketChannel.accept() while waiting for the Python worker to connect.
If the runner is stopped before that connection is established, the shutdown
path interrupts and closes the channel. Java NIO wakes accept() with
ClosedByInterruptException or AsynchronousCloseException. These are expected
shutdown signals, but the pre-connect path does not handle them, so the
listener exception is wrapped as:
TransformWithStateInPySpark state server daemon thread exited unexpectedly
(crashed)
Expected behavior:
The listener should terminate cleanly, preserve interrupt status for
interrupt-driven shutdown, and transition the processor handle to CLOSED.
Proposed fix:
- Interrupt the listener before closing the channel.
- Handle the expected NIO interruption and closed-channel exceptions around
accept().
- Preserve interrupt status only for the interrupt-driven path.
- Add mocked exception-hierarchy tests and real loopback ServerSocketChannel
race tests.
Pull request: https://github.com/apache/spark/pull/58217
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
