L. C. Hsieh created SPARK-61765:
-----------------------------------
Summary: Report the Python worker idle timeout when the read from
the killed worker fails
Key: SPARK-61765
URL: https://issues.apache.org/jira/browse/SPARK-61765
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
After the idle timeout kills a Python worker
(spark.sql.execution.pyspark.udf.killOnIdleTimeout), the read from the worker
does not always return the end of the stream. The connection of a killed worker
can be reset, for example when the worker had unread input, and the read then
throws an IOException (Connection reset).
IdleTimeoutHandler.throwIfKilled only looks at the value a read returned, so in
that case the task fails with "Python worker exited unexpectedly (crashed)",
together with the hint to enable the fault handler, instead of "Python worker
process terminated due to idle timeout (timeout: N seconds)". This affects the
three streams that use the handler: BasePythonRunner.ReaderInputStream, the
pipelined socket stream, and PythonPlannerRunner.WorkerInputStream.
A read (or a write on the reading thread) that fails after the handler killed
the worker should be reported as the idle timeout, with the I/O failure as the
cause. A failure with no kill should be reported as today.
This was pointed out in the review of SPARK-61720.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]