L. C. Hsieh created SPARK-60998:
-----------------------------------
Summary: LongToUnsafeRowMap reserves a full page (up to 64 MiB) of
execution memory per map, causing OOM with many shuffled hash joins in one task
Key: SPARK-60998
URL: https://issues.apache.org/jira/browse/SPARK-60998
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
Since SPARK-54116 (4.2.0), LongToUnsafeRowMap allocates its memory through
MemoryConsumer.allocatePage. allocatePage rounds every request up to the
consumer's page size, which is spark.buffer.pageSize, or a value derived from
the executor memory and cores of up to 64 MiB. Before SPARK-54116 the map
started with a 1 MiB page and acquired execution memory for exactly what it
allocated.
So each LongToUnsafeRowMap now reserves at least one full page, typically 64
MiB, even when it holds only a few rows. Page growth (2 MiB, 4 MiB, ...) and
deserialization are rounded up the same way.
When one task keeps many of these maps alive at once, for example a stage with
a long chain of shuffled hash joins without AQE, the task runs out of execution
memory long before the maps hold that much data:
{code} [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 2048 bytes of memory, got
0. Memory used by task ... grouped by consumer:
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap@...: 64.0 MiB
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap@...: 64.0 MiB ... {code}
The same query passes with the 1 MiB initial page used before SPARK-54116. This
affects both on-heap and off-heap memory modes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
