L. C. Hsieh created SPARK-61720:
-----------------------------------
Summary: Share the Python worker idle timeout handling between the
Python runner readers
Key: SPARK-61720
URL: https://issues.apache.org/jira/browse/SPARK-61720
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
The handling of the Python worker idle timeout exists in three copies: in
BasePythonRunner.ReaderInputStream (the synchronous read path), in the socket
input stream of the pipelined read path, and in
PythonPlannerRunner.WorkerInputStream on the driver. Each copy logs the
timeout, kills the worker if killOnIdleTimeout is set, and fails the read once
the killed worker has closed the connection. A fix in one of them has to be
repeated in the others by hand.
This moves the logic into one shared helper used by all three readers, without
changing behavior. The pipelined read path also gets a test for the idle
timeout, which it did not have.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]