[
https://issues.apache.org/jira/browse/SPARK-61775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-61775:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Do not wait forever for the read to end after the idle timeout killed the
> Python worker
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61775
> URL: https://issues.apache.org/jira/browse/SPARK-61775
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> When the Python worker idle timeout kills a worker
> (spark.sql.execution.pyspark.udf.idleTimeoutSeconds with
> spark.sql.execution.pyspark.udf.killOnIdleTimeout),
> BasePythonRunner.IdleTimeoutHandler keeps waiting for the read from the
> worker to end: the read has to return the end of the stream or fail
> (SPARK-61765) before the idle timeout error is thrown. Every further idle
> timeout after the kill only logs "Waiting for Python worker process to
> terminate after idle timeout" and tries the read again.
> If the connection to the killed worker is never closed, the read never ends
> and the task hangs, although the worker has already been killed. This can
> happen, for example, when another process still holds the worker's end of the
> socket open.
> The handler should bound this wait and fail the task with the idle timeout
> error once the killed worker is known to be gone, or after some time, instead
> of waiting forever.
> One thing to check first: for a worker forked by the Python daemon, whether
> ProcessHandle.isAlive() is reliable between the kill and the daemon reaping
> the process, since a check like "timed out again after the kill and the
> process is no longer alive" depends on it.
> This was raised in the review of SPARK-61765
> (https://github.com/apache/spark/pull/60540).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]