[
https://issues.apache.org/jira/browse/SPARK-61195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141713#comment-18141713
]
Huang Kuan Hao commented on SPARK-61195:
----------------------------------------
Thanks for filing this. One catch: the listener-order argument works for
TRANSFORM, but not for {{PipedRDD}}.
{{PipedRDD}} creates its parent iterator inside the stdin writer thread
({{firstParent[T].iterator(split, context)}}, {{PipedRDD.scala:149}}), so the
scan registers its close listener on that thread, after the writer thread has
started. A task completion listener added when the thread starts would
therefore run after the reader is closed. With a {{mapPartitions}} between the
scan and {{pipe("cat")}}, the {{mapPartitions}} function runs on the {{stdin
writer for List(cat)}} thread.
So {{PipedRDD}} probably needs {{addTaskPreCompletionListener}}, like the
pipelined Python writer, which uses it for the same listener-ordering issue.
The writer also needs a way to stop. It writes through a {{PrintWriter}}, which
swallows {{IOException}} once the process is gone, so the loop doesn't notice
and keeps reading the parent iterator. Without a check like
{{out.checkError()}} or a flag set by the listener, joining the thread could
mean waiting for the rest of the partition to be read.
I reproduced this on Spark 4.1.3 in local mode with Comet turned off,
{{spark.memory.offHeap.enabled=true}} and a 4-million-row Parquet table.
{{spark.sql.columnVector.offheap.enabled}} falls back to that setting.
{{spark.read.parquet(p).rdd.pipe("head -1").count()}} in a loop: SIGSEGV in
{{OffHeapColumnVector.isNullAt}} on the stdin writer thread.
{{SELECT TRANSFORM(id, s) USING 'cat' AS (a, b) FROM t LIMIT 1}} in a loop: the
same crash on the TRANSFORM feed thread.
I ran into this while testing DataFusion Comet, whose native plans hit the same
race. Happy to open a PR for this if that helps.
> Stop the threads that read a task's input on their own before its readers are
> closed
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61195
> URL: https://issues.apache.org/jira/browse/SPARK-61195
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Gengliang Wang
> Priority: Major
>
> Some operators read their input on a thread of their own, which the task does
> not wait for: the writer thread of script transformation ({{TRANSFORM}}), the
> stdin writer thread of {{PipedRDD}}, and the writer thread of pipelined
> Python UDFs ({{spark.python.udf.pipelined.enabled}}). Such a thread can still
> read a columnar batch after its task completed, while the completion listener
> of {{DataSourceRDD}} closes the reader that returned the batch, for example
> under a {{LIMIT}} or when the script exits early.
> The readers of native data sources read the buffers of a batch at their
> addresses (SPARK-60753). So a reader that is closed before the end of its
> stream cannot free the buffers of its last batch at once: it leaves them to a
> Cleaner, which frees them once the reader and the batch are unreachable, at a
> garbage collection that this off-heap memory does not trigger. With
> {{spark.sql.columnVector.offheap.enabled}}, the vectorized Parquet reader
> instead frees its {{OffHeapColumnVector}}s at close, so such a thread can
> read freed memory there.
> Script transformation also does not destroy its process when its task
> completes, unlike {{PipedRDD}}. Under a {{LIMIT}}, a script that keeps
> writing output that nobody reads any more blocks once the pipe is full, and
> so does the writer thread, which then keeps the thread, the process and the
> batch alive.
> Proposal: have these operators stop their thread, and wait for it to end,
> from a task completion listener that they register when they start the
> thread. Listeners run in reverse order of registration, so this one runs
> before {{DataSourceRDD}} closes the readers. Script transformation should
> destroy its process without interrupting its writer thread: the interrupt
> makes {{proc.waitFor()}} in the {{finally}} block of the writer throw
> {{InterruptedException}}, which {{NonFatal}} does not catch, so that
> {{SparkUncaughtExceptionHandler}} exits the executor. The failures that the
> writer gets once its task completed should not be logged as errors.
> {{NativePartitionReader.close()}} can then free the buffers of its last batch
> at once, without the Cleaner.
> Discussion:
> https://github.com/apache/spark/pull/59961#discussion_r4488238438,
> https://github.com/apache/spark/pull/59961#discussion_r4500198812 and
> https://github.com/apache/spark/pull/59961#discussion_r4500198806
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]