[
https://issues.apache.org/jira/browse/SPARK-60441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60441:
-----------------------------------
Labels: pull-request-available (was: )
> UnsafeSorterSpillReader keeps its 1 MB record buffer until the task ends
> ------------------------------------------------------------------------
>
> Key: SPARK-60441
> URL: https://issues.apache.org/jira/browse/SPARK-60441
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Qing Fu
> Priority: Major
> Labels: pull-request-available
>
> SPARK-47146 (3.4.3, 3.5.2, 4.0.0) made {{UnsafeSorterSpillReader}} register a
> task completion listener that closes the reader, so that an abandoned
> reader's stream (and its read-ahead thread) is closed at the end of the task.
> The listener's lambda calls {{close()}} on the reader, so the task keeps
> every reader it ever created, and with it the reader's record buffer, until
> the task ends, including readers that were read to the end and closed.
> The record buffer is allocated at 1 MB up front ({{new byte[1024 * 1024]}}).
> {{UnsafeExternalSorter.getIterator}} creates a spill reader per call, and a
> sort-merge join whose matches spilled ({{ExternalAppendOnlyUnsafeRowArray}})
> calls it for every streamed row, so a task retains about 1 MB of heap per
> streamed row with spilled matches. Each reader also allocates its 1 MB up
> front, however short its records are.
> Proposed:
> - register a small holder that references only the stream, which the reader
> closes and the listener closes at the end of the task, so that a reader the
> caller has dropped can be garbage collected;
> - start the record buffer at 1 KB and grow it by doubling, as most readers
> created by {{getIterator}} read a few records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
