[
https://issues.apache.org/jira/browse/SPARK-58977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned SPARK-58977:
------------------------------------
Assignee: Jonathan Gao
> TransformWithState in PySpark state server reports spurious daemon crash
> during early shutdown
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-58977
> URL: https://issues.apache.org/jira/browse/SPARK-58977
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Jonathan Gao
> Assignee: Jonathan Gao
> Priority: Major
> Labels: pull-request-available
>
> This is a follow-up to SPARK-58751 and
> https://github.com/apache/spark/pull/57941.
> TransformWithState in PySpark starts a JVM state-server listener that blocks
> in ServerSocketChannel.accept() while waiting for the Python worker to
> connect.
> If the runner is stopped before that connection is established, the shutdown
> path interrupts and closes the channel. Java NIO wakes accept() with
> ClosedByInterruptException or AsynchronousCloseException. These are expected
> shutdown signals, but the pre-connect path does not handle them, so the
> listener exception is wrapped as:
> TransformWithStateInPySpark state server daemon thread exited unexpectedly
> (crashed)
> Expected behavior:
> The listener should terminate cleanly, preserve interrupt status for
> interrupt-driven shutdown, and transition the processor handle to CLOSED.
> Proposed fix:
> - Interrupt the listener before closing the channel.
> - Handle the expected NIO interruption and closed-channel exceptions around
> accept().
> - Preserve interrupt status only for the interrupt-driven path.
> - Add mocked exception-hierarchy tests and real loopback ServerSocketChannel
> race tests.
> Pull request: https://github.com/apache/spark/pull/58217
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
