[
https://issues.apache.org/jira/browse/SPARK-60998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60998:
-----------------------------------
Labels: pull-request-available (was: )
> LongToUnsafeRowMap reserves a full page (up to 64 MiB) of execution memory
> per map, causing OOM with many shuffled hash joins in one task
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60998
> URL: https://issues.apache.org/jira/browse/SPARK-60998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Since SPARK-54116 (4.2.0), LongToUnsafeRowMap allocates its memory through
> MemoryConsumer.allocatePage. allocatePage rounds every request up to the
> consumer's page size, which is spark.buffer.pageSize, or a value derived from
> the executor memory and cores of up to 64 MiB. Before SPARK-54116 the map
> started with a 1 MiB page and acquired execution memory for exactly what it
> allocated.
> So each LongToUnsafeRowMap now reserves at least one full page, typically 64
> MiB, even when it holds only a few rows. Page growth (2 MiB, 4 MiB, ...) and
> deserialization are rounded up the same way.
> When one task keeps many of these maps alive at once, for example a stage
> with a long chain of shuffled hash joins without AQE, the task runs out of
> execution memory long before the maps hold that much data:
> {code} [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 2048 bytes of memory, got
> 0. Memory used by task ... grouped by consumer:
> org.apache.spark.sql.execution.joins.LongToUnsafeRowMap@...: 64.0 MiB
> org.apache.spark.sql.execution.joins.LongToUnsafeRowMap@...: 64.0 MiB ...
> {code}
> The same query passes with the 1 MiB initial page used before SPARK-54116.
> This affects both on-heap and off-heap memory modes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
