[ 
https://issues.apache.org/jira/browse/SPARK-62031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62031:
-----------------------------------
    Labels: pull-request-available  (was: )

> Free the last batch of a native data source scan that stops early without 
> waiting for a GC
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62031
>                 URL: https://issues.apache.org/jira/browse/SPARK-62031
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Gengliang Wang
>            Priority: Major
>              Labels: pull-request-available
>
> Since SPARK-60753, the column vectors of a native data source scan read the 
> buffers of a batch at their addresses. A reader that is closed before the end 
> of its stream cannot tell whether a thread still reads the batch that it 
> returned last, so it leaves the buffers of that batch to a Cleaner, which 
> frees them once the reader and the batch are unreachable, at a garbage 
> collection that this off-heap memory does not trigger. Under 
> {{DataSourceRDD}}, this is every scan whose task ends before the stream does, 
> as with {{LIMIT}}, {{show()}} or {{take()}}, and in failed or killed tasks: 
> each holds up to one batch per partition until a garbage collection.
> Only three operators of Spark read the input of a task on a thread that the 
> task does not wait for when it completes: the stdin writer thread of 
> {{PipedRDD}}, the writer thread of script transformation ({{TRANSFORM}}), and 
> the writer thread of the R runner. The task of the writer of pipelined Python 
> UDFs waits for it before its completion listeners run (SPARK-60113).
> Proposal: these three operators register their thread for the task before 
> they start it, and the reader frees the buffers of its last batch when it is 
> closed in a task without such a thread. In a task with such a thread, and 
> outside of a task, where the reader does not know its consumers, it leaves 
> them to the Cleaner as before. The operators are otherwise unchanged.
> SPARK-61195 proposes to make these operators stop their threads and wait for 
> them, after which the Cleaner can be removed. That changes how they shut 
> down, and can follow later.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to