L. C. Hsieh created SPARK-61765:
-----------------------------------

             Summary: Report the Python worker idle timeout when the read from 
the killed worker fails
                 Key: SPARK-61765
                 URL: https://issues.apache.org/jira/browse/SPARK-61765
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


After the idle timeout kills a Python worker 
(spark.sql.execution.pyspark.udf.killOnIdleTimeout), the read from the worker 
does not always return the end of the stream. The connection of a killed worker 
can be reset, for example when the worker had unread input, and the read then 
throws an IOException (Connection reset).

IdleTimeoutHandler.throwIfKilled only looks at the value a read returned, so in 
that case the task fails with "Python worker exited unexpectedly (crashed)", 
together with the hint to enable the fault handler, instead of "Python worker 
process terminated due to idle timeout (timeout: N seconds)". This affects the 
three streams that use the handler: BasePythonRunner.ReaderInputStream, the 
pipelined socket stream, and PythonPlannerRunner.WorkerInputStream.

A read (or a write on the reading thread) that fails after the handler killed 
the worker should be reported as the idle timeout, with the I/O failure as the 
cause. A failure with no kill should be reported as today.

This was pointed out in the review of SPARK-61720.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to