Gengliang Wang created SPARK-62031:
--------------------------------------

             Summary: Free the last batch of a native data source scan that 
stops early without waiting for a GC
                 Key: SPARK-62031
                 URL: https://issues.apache.org/jira/browse/SPARK-62031
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Since SPARK-60753, the column vectors of a native data source scan read the 
buffers of a batch at their addresses. A reader that is closed before the end 
of its stream cannot tell whether a thread still reads the batch that it 
returned last, so it leaves the buffers of that batch to a Cleaner, which frees 
them once the reader and the batch are unreachable, at a garbage collection 
that this off-heap memory does not trigger. Under {{DataSourceRDD}}, this is 
every scan whose task ends before the stream does, as with {{LIMIT}}, 
{{show()}} or {{take()}}, and in failed or killed tasks: each holds up to one 
batch per partition until a garbage collection.

Only three operators of Spark read the input of a task on a thread that the 
task does not wait for when it completes: the stdin writer thread of 
{{PipedRDD}}, the writer thread of script transformation ({{TRANSFORM}}), and 
the writer thread of the R runner. The task of the writer of pipelined Python 
UDFs waits for it before its completion listeners run (SPARK-60113).

Proposal: these three operators register their thread for the task before they 
start it, and the reader frees the buffers of its last batch when it is closed 
in a task without such a thread. In a task with such a thread, and outside of a 
task, where the reader does not know its consumers, it leaves them to the 
Cleaner as before. The operators are otherwise unchanged.

SPARK-61195 proposes to make these operators stop their threads and wait for 
them, after which the Cleaner can be removed. That changes how they shut down, 
and can follow later.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to