This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new dcfe32e0b98e [SPARK-61051][SQL] Fix LongHashedRelation build failure
when the dense key array does not fit in execution memory
dcfe32e0b98e is described below
commit dcfe32e0b98eecccd75174c2d30e4b2cd4a01e4d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 19:17:17 2026 -0800
[SPARK-61051][SQL] Fix LongHashedRelation build failure when the dense key
array does not fit in execution memory
### What changes were proposed in this pull request?
This PR makes `LongToUnsafeRowMap.optimize()` keep the map in sparse mode
when it cannot allocate the dense array, instead of failing the task. It
catches the `SparkOutOfMemoryError` with the `UNABLE_TO_ACQUIRE_MEMORY`
condition thrown by `allocateArray` and returns, as `optimize()` did before
SPARK-54116.
`MemoryConsumer.allocateArray` releases any partially acquired page in
`throwOom` before it throws, so nothing leaks when the conversion is skipped.
Other `SparkOutOfMemoryError`s still fail the task, as in Spark 4.1. For
example, `SPILL_OUT_OF_MEMORY` means that another consumer's spill failed with
an `IOException`. By then, `TaskMemoryManager` may have already acquired part
of the request, and that part stays with the task. The consumer whose spill
failed may also be unusable.
### Why are the changes needed?
Dense mode is only a lookup optimization. In
[v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11) and
earlier, `optimize()` reserved the memory with `ensureAcquireMemory` and
returned on `SparkException`, so a map that could not get the extra memory
stayed sparse.
SPARK-54116 replaced that with `allocateArray`, which throws
`SparkOutOfMemoryError`, and `optimize()` no longer catches it. The dense array
is allocated while the sparse array is still alive, and it can be as large as
the sparse array. So a task that has enough execution memory to build the map
now fails with `UNABLE_TO_ACQUIRE_MEMORY` at the end of the build, where Spark
4.1 succeeded. This affects `LongHashedRelation`s built with the task's memory
manager, such as the build side of [...]
The regression is in
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11),
[v4.2.1](https://github.com/apache/spark/releases/tag/v4.2.1) (2026-10-21) and
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
### Does this PR introduce _any_ user-facing change?
Yes. A task whose execution memory is too small for the dense conversion no
longer fails with `SparkOutOfMemoryError`. The hashed relation stays in sparse
mode, as in Spark 4.1.
### How was this patch tested?
Pass the CIs with a newly added test case in `HashedRelationSuite`, which
runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. The
test builds a map whose sparse array and page fit in a small execution-memory
budget while the dense array does not. It checks that `optimize()` keeps the
map sparse and usable without leaking memory. It also checks that `optimize()`
still throws `SPILL_OUT_OF_MEMORY` when another consumer's spill fails.
The test fails without this fix:
```
$ build/sbt "sql/testOnly
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite"
...
[info] Cause: org.apache.spark.memory.SparkOutOfMemoryError:
[UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 507912 bytes of memory, got 262144.
...
[info] *** 2 TESTS FAILED ***
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This patch had conflicts when merged, resolved by
Committer: Dongjoon Hyun <[email protected]>
Closes #60191 from dongjoon-hyun/SPARK-61051.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ef3f5a3fd0a8f9b5592aff6724d89827fdabff98)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/execution/joins/HashedRelation.scala | 10 +++-
.../sql/execution/joins/HashedRelationSuite.scala | 55 +++++++++++++++++++++-
2 files changed, 62 insertions(+), 3 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index 801243fc9be0..b388d2aff22b 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -868,7 +868,15 @@ private[execution] final class LongToUnsafeRowMap(
// Convert to dense mode if it does not require more memory or could fit
within L1 cache
// SPARK-16740: Make sure range doesn't overflow if minKey has a large
negative value
if (range >= 0 && (range < array.size() || range < 1024)) {
- val denseArray = allocateArray((range + 1).toInt)
+ val denseArray = try {
+ allocateArray((range + 1).toInt)
+ } catch {
+ case e: SparkOutOfMemoryError if e.getCondition ==
"UNABLE_TO_ACQUIRE_MEMORY" =>
+ // There is not enough memory to convert, so keep the map sparse.
`allocateArray`
+ // releases any memory it acquired before throwing. Other errors,
such as a failed
+ // spill of another consumer, still fail the task.
+ return
+ }
denseArray.zeroOut()
var i = 0
while (i < array.size()) {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index 50fa9aa899d5..a6ae728e7ccd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.joins
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream,
ObjectInputStream, ObjectOutputStream}
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, IOException,
ObjectInputStream, ObjectOutputStream}
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
@@ -27,7 +27,7 @@ import org.apache.spark.SparkConf
import org.apache.spark.SparkException
import org.apache.spark.internal.config.{BUFFER_PAGESIZE,
MEMORY_OFFHEAP_ENABLED, MEMORY_OFFHEAP_SIZE}
import org.apache.spark.internal.config.Kryo._
-import org.apache.spark.memory.{SparkOutOfMemoryError, TaskMemoryManager,
UnifiedMemoryManager}
+import org.apache.spark.memory.{MemoryConsumer, SparkOutOfMemoryError,
TaskMemoryManager, UnifiedMemoryManager}
import org.apache.spark.network.util.ByteUnit
import org.apache.spark.serializer.{JavaSerializer, KryoSerializer}
import org.apache.spark.sql.catalyst.InternalRow
@@ -865,6 +865,57 @@ abstract class HashedRelationSuite extends
SharedSparkSession {
map.free()
}
}
+
+ test("SPARK-61051: LongToUnsafeRowMap stays sparse when the dense array does
not fit") {
+ // The sparse array (1 MiB) and the page (1 MiB) fit, but the dense array
(about 0.5 MiB)
+ // does not, because it is allocated while the sparse array is still alive.
+ val memorySize = ByteUnit.MiB.toBytes(2) + ByteUnit.KiB.toBytes(256)
+ val smallUmm = new UnifiedMemoryManager(
+ new SparkConf()
+ .set(MEMORY_OFFHEAP_ENABLED, useOffHeapMemoryMode)
+ .set(MEMORY_OFFHEAP_SIZE, memorySize)
+ .set(BUFFER_PAGESIZE, ByteUnit.MiB.toBytes(1)),
+ memorySize,
+ memorySize / 2,
+ 1)
+ val smallMm = new TaskMemoryManager(smallUmm, 0)
+ val unsafeProj = UnsafeProjection.create(Seq(BoundReference(0, LongType,
false)))
+ // Capacity 1 << 16 gives a sparse array of 1 << 17 longs, which 32 keys
do not grow.
+ val map = new LongToUnsafeRowMap(smallMm, 1 << 16)
+ val keys = (0 until 32).map(_ * 2048L)
+ keys.foreach(k => map.append(k, unsafeProj(InternalRow(k))))
+ val used = map.getUsed
+ assert(used === ByteUnit.MiB.toBytes(2))
+
+ try {
+ map.optimize()
+ } catch {
+ // SparkOutOfMemoryError is fatal to ScalaTest and would abort the whole
suite.
+ case e: SparkOutOfMemoryError =>
+ map.free()
+ fail("optimize() should keep the map sparse when the dense array does
not fit", e)
+ }
+ // The map keeps its sparse array and the failed allocation does not leak
memory.
+ assert(map.getUsed === used)
+ assert(smallUmm.executionMemoryUsed === used)
+ val resultRow = new UnsafeRow(1)
+ keys.foreach(k => assert(map.getValue(k, resultRow).getLong(0) === k))
+ assert(map.getValue(1L, resultRow) === null)
+
+ // A failed spill of another consumer is not a lack of memory, so
optimize() does not hide it.
+ val failingConsumer = new MemoryConsumer(smallMm,
smallMm.getTungstenMemoryMode) {
+ override def spill(size: Long, trigger: MemoryConsumer): Long = {
+ throw new IOException("Injected spill failure")
+ }
+ }
+ val remaining = memorySize - used
+ assert(failingConsumer.acquireMemory(remaining) === remaining)
+ val e = intercept[SparkOutOfMemoryError](map.optimize())
+ assert(e.getCondition === "SPILL_OUT_OF_MEMORY")
+ failingConsumer.freeMemory(remaining)
+ map.free()
+ assert(smallUmm.executionMemoryUsed === 0)
+ }
}
class HashedRelationOnHeapSuite extends HashedRelationSuite {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]