Qing Fu created SPARK-60441:
-------------------------------
Summary: UnsafeSorterSpillReader keeps its 1 MB record buffer
until the task ends
Key: SPARK-60441
URL: https://issues.apache.org/jira/browse/SPARK-60441
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Qing Fu
SPARK-47146 (3.4.3, 3.5.2, 4.0.0) made {{UnsafeSorterSpillReader}} register a
task completion listener that closes the reader, so that an abandoned reader's
stream (and its read-ahead thread) is closed at the end of the task. The
listener's lambda calls {{close()}} on the reader, so the task keeps every
reader it ever created, and with it the reader's record buffer, until the task
ends, including readers that were read to the end and closed.
The record buffer is allocated at 1 MB up front ({{new byte[1024 * 1024]}}).
{{UnsafeExternalSorter.getIterator}} creates a spill reader per call, and a
sort-merge join whose matches spilled ({{ExternalAppendOnlyUnsafeRowArray}})
calls it for every streamed row, so a task retains about 1 MB of heap per
streamed row with spilled matches. Each reader also allocates its 1 MB up
front, however short its records are.
Proposed:
- register a small holder that references only the stream, which the reader
closes and the listener closes at the end of the task, so that a reader the
caller has dropped can be garbage collected;
- start the record buffer at 1 KB and grow it by doubling, as most readers
created by {{getIterator}} read a few records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
