Gengliang Wang created SPARK-62031:
--------------------------------------
Summary: Free the last batch of a native data source scan that
stops early without waiting for a GC
Key: SPARK-62031
URL: https://issues.apache.org/jira/browse/SPARK-62031
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Since SPARK-60753, the column vectors of a native data source scan read the
buffers of a batch at their addresses. A reader that is closed before the end
of its stream cannot tell whether a thread still reads the batch that it
returned last, so it leaves the buffers of that batch to a Cleaner, which frees
them once the reader and the batch are unreachable, at a garbage collection
that this off-heap memory does not trigger. Under {{DataSourceRDD}}, this is
every scan whose task ends before the stream does, as with {{LIMIT}},
{{show()}} or {{take()}}, and in failed or killed tasks: each holds up to one
batch per partition until a garbage collection.
Only three operators of Spark read the input of a task on a thread that the
task does not wait for when it completes: the stdin writer thread of
{{PipedRDD}}, the writer thread of script transformation ({{TRANSFORM}}), and
the writer thread of the R runner. The task of the writer of pipelined Python
UDFs waits for it before its completion listeners run (SPARK-60113).
Proposal: these three operators register their thread for the task before they
start it, and the reader frees the buffers of its last batch when it is closed
in a task without such a thread. In a task with such a thread, and outside of a
task, where the reader does not know its consumers, it leaves them to the
Cleaner as before. The operators are otherwise unchanged.
SPARK-61195 proposes to make these operators stop their threads and wait for
them, after which the Cleaner can be removed. That changes how they shut down,
and can follow later.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]