[
https://issues.apache.org/jira/browse/SPARK-60305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60305:
--------------------------------
Affects Version/s: 4.2.0
4.1.3
4.0.4
4.3.0
4.4.0
(was: 5.0.0)
> Interrupting a Python Arrow UDF task can free its input buffer while it is
> being written
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-60305
> URL: https://issues.apache.org/jira/browse/SPARK-60305
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> Since SPARK-44705, the default (non-pipelined) Python runner serializes the
> task input on the task thread into a DirectByteBufferOutputStream inside
> BasePythonRunner.ReaderInputStream. The Arrow-based inputs (PythonArrowInput,
> ColumnarArrowPythonInput) write that stream through
> Channels.newChannel(dataOut), via ArrowStreamWriter and WriteChannel. The
> returned channel is an AbstractInterruptibleChannel: each write chunk runs
> between begin() and end(), and closing the channel closes the wrapped stream.
> If another thread interrupts the task thread while it is inside such a write,
> for example Task.kill(interruptThread = true) when a speculative attempt or a
> cancelled job is killed, Thread.interrupt() runs the channel's interruptor on
> the interrupting thread. That closes the DataOutputStream and therefore the
> DirectByteBufferOutputStream, which frees its direct buffer while the task
> thread is still copying into it (or copying the old buffer in grow()). This
> can crash the executor, silently corrupt memory, or fail with an NPE /
> internal error instead of a clean task kill.
> Affected: Arrow-based Python UDFs and related operators (pandas/Arrow UDFs,
> Arrow-optimized Python UDFs, mapInPandas/mapInArrow, applyInPandas, ...) on
> the default non-pipelined path. The pickled BatchEvalPython path writes the
> stream directly and is not affected.
> Fix: hand the writer a stream whose close() does not close the underlying
> buffer, and free the buffer only where the runner closes it itself.
> Reported by Dongjoon Hyun in the review of
> https://github.com/apache/spark/pull/59332 (SPARK-60113), which fixes the
> same issue on the pipelined writer path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
