L. C. Hsieh created SPARK-61720:
-----------------------------------

             Summary: Share the Python worker idle timeout handling between the 
Python runner readers
                 Key: SPARK-61720
                 URL: https://issues.apache.org/jira/browse/SPARK-61720
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


The handling of the Python worker idle timeout exists in three copies: in 
BasePythonRunner.ReaderInputStream (the synchronous read path), in the socket 
input stream of the pipelined read path, and in 
PythonPlannerRunner.WorkerInputStream on the driver. Each copy logs the 
timeout, kills the worker if killOnIdleTimeout is set, and fails the read once 
the killed worker has closed the connection. A fix in one of them has to be 
repeated in the others by hand.

This moves the logic into one shared helper used by all three readers, without 
changing behavior. The pipelined read path also gets a test for the idle 
timeout, which it did not have.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to