This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new a1fca621d0e4 [SPARK-61051][SQL] Fix LongHashedRelation build failure 
when the dense key array does not fit in execution memory
a1fca621d0e4 is described below

commit a1fca621d0e4ad785f33ee84192712e07c051b08
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 19:17:17 2026 -0800

    [SPARK-61051][SQL] Fix LongHashedRelation build failure when the dense key 
array does not fit in execution memory
    
    ### What changes were proposed in this pull request?
    
    This PR makes `LongToUnsafeRowMap.optimize()` keep the map in sparse mode 
when it cannot allocate the dense array, instead of failing the task. It 
catches the `SparkOutOfMemoryError` with the `UNABLE_TO_ACQUIRE_MEMORY` 
condition thrown by `allocateArray` and returns, as `optimize()` did before 
SPARK-54116.
    
    `MemoryConsumer.allocateArray` releases any partially acquired page in 
`throwOom` before it throws, so nothing leaks when the conversion is skipped.
    
    Other `SparkOutOfMemoryError`s still fail the task, as in Spark 4.1. For 
example, `SPILL_OUT_OF_MEMORY` means that another consumer's spill failed with 
an `IOException`. By then, `TaskMemoryManager` may have already acquired part 
of the request, and that part stays with the task. The consumer whose spill 
failed may also be unusable.
    
    ### Why are the changes needed?
    
    Dense mode is only a lookup optimization. In 
[v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11) and 
earlier, `optimize()` reserved the memory with `ensureAcquireMemory` and 
returned on `SparkException`, so a map that could not get the extra memory 
stayed sparse.
    
    SPARK-54116 replaced that with `allocateArray`, which throws 
`SparkOutOfMemoryError`, and `optimize()` no longer catches it. The dense array 
is allocated while the sparse array is still alive, and it can be as large as 
the sparse array. So a task that has enough execution memory to build the map 
now fails with `UNABLE_TO_ACQUIRE_MEMORY` at the end of the build, where Spark 
4.1 succeeded. This affects `LongHashedRelation`s built with the task's memory 
manager, such as the build side of [...]
    
    The regression is in 
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11), 
[v4.2.1](https://github.com/apache/spark/releases/tag/v4.2.1) (2026-10-21) and 
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A task whose execution memory is too small for the dense conversion no 
longer fails with `SparkOutOfMemoryError`. The hashed relation stays in sparse 
mode, as in Spark 4.1.
    
    ### How was this patch tested?
    
    Pass the CIs with a newly added test case in `HashedRelationSuite`, which 
runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. The 
test builds a map whose sparse array and page fit in a small execution-memory 
budget while the dense array does not. It checks that `optimize()` keeps the 
map sparse and usable without leaking memory. It also checks that `optimize()` 
still throws `SPILL_OUT_OF_MEMORY` when another consumer's spill fails.
    
    The test fails without this fix:
    
    ```
    $ build/sbt "sql/testOnly 
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite 
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite"
    ...
    [info]   Cause: org.apache.spark.memory.SparkOutOfMemoryError: 
[UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 507912 bytes of memory, got 262144.
    ...
    [info] *** 2 TESTS FAILED ***
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This patch had conflicts when merged, resolved by
    Committer: Dongjoon Hyun <[email protected]>
    
    Closes #60191 from dongjoon-hyun/SPARK-61051.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit ef3f5a3fd0a8f9b5592aff6724d89827fdabff98)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/execution/joins/HashedRelation.scala | 10 +++-
 .../sql/execution/joins/HashedRelationSuite.scala  | 55 +++++++++++++++++++++-
 2 files changed, 62 insertions(+), 3 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index 801243fc9be0..b388d2aff22b 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -868,7 +868,15 @@ private[execution] final class LongToUnsafeRowMap(
     // Convert to dense mode if it does not require more memory or could fit 
within L1 cache
     // SPARK-16740: Make sure range doesn't overflow if minKey has a large 
negative value
     if (range >= 0 && (range < array.size() || range < 1024)) {
-      val denseArray = allocateArray((range + 1).toInt)
+      val denseArray = try {
+        allocateArray((range + 1).toInt)
+      } catch {
+        case e: SparkOutOfMemoryError if e.getCondition == 
"UNABLE_TO_ACQUIRE_MEMORY" =>
+          // There is not enough memory to convert, so keep the map sparse. 
`allocateArray`
+          // releases any memory it acquired before throwing. Other errors, 
such as a failed
+          // spill of another consumer, still fail the task.
+          return
+      }
       denseArray.zeroOut()
       var i = 0
       while (i < array.size()) {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index 50fa9aa899d5..a6ae728e7ccd 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -17,7 +17,7 @@
 
 package org.apache.spark.sql.execution.joins
 
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream, 
ObjectInputStream, ObjectOutputStream}
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, IOException, 
ObjectInputStream, ObjectOutputStream}
 
 import scala.collection.mutable
 import scala.collection.mutable.ArrayBuffer
@@ -27,7 +27,7 @@ import org.apache.spark.SparkConf
 import org.apache.spark.SparkException
 import org.apache.spark.internal.config.{BUFFER_PAGESIZE, 
MEMORY_OFFHEAP_ENABLED, MEMORY_OFFHEAP_SIZE}
 import org.apache.spark.internal.config.Kryo._
-import org.apache.spark.memory.{SparkOutOfMemoryError, TaskMemoryManager, 
UnifiedMemoryManager}
+import org.apache.spark.memory.{MemoryConsumer, SparkOutOfMemoryError, 
TaskMemoryManager, UnifiedMemoryManager}
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.serializer.{JavaSerializer, KryoSerializer}
 import org.apache.spark.sql.catalyst.InternalRow
@@ -865,6 +865,57 @@ abstract class HashedRelationSuite extends 
SharedSparkSession {
       map.free()
     }
   }
+
+  test("SPARK-61051: LongToUnsafeRowMap stays sparse when the dense array does 
not fit") {
+    // The sparse array (1 MiB) and the page (1 MiB) fit, but the dense array 
(about 0.5 MiB)
+    // does not, because it is allocated while the sparse array is still alive.
+    val memorySize = ByteUnit.MiB.toBytes(2) + ByteUnit.KiB.toBytes(256)
+    val smallUmm = new UnifiedMemoryManager(
+      new SparkConf()
+        .set(MEMORY_OFFHEAP_ENABLED, useOffHeapMemoryMode)
+        .set(MEMORY_OFFHEAP_SIZE, memorySize)
+        .set(BUFFER_PAGESIZE, ByteUnit.MiB.toBytes(1)),
+      memorySize,
+      memorySize / 2,
+      1)
+    val smallMm = new TaskMemoryManager(smallUmm, 0)
+    val unsafeProj = UnsafeProjection.create(Seq(BoundReference(0, LongType, 
false)))
+    // Capacity 1 << 16 gives a sparse array of 1 << 17 longs, which 32 keys 
do not grow.
+    val map = new LongToUnsafeRowMap(smallMm, 1 << 16)
+    val keys = (0 until 32).map(_ * 2048L)
+    keys.foreach(k => map.append(k, unsafeProj(InternalRow(k))))
+    val used = map.getUsed
+    assert(used === ByteUnit.MiB.toBytes(2))
+
+    try {
+      map.optimize()
+    } catch {
+      // SparkOutOfMemoryError is fatal to ScalaTest and would abort the whole 
suite.
+      case e: SparkOutOfMemoryError =>
+        map.free()
+        fail("optimize() should keep the map sparse when the dense array does 
not fit", e)
+    }
+    // The map keeps its sparse array and the failed allocation does not leak 
memory.
+    assert(map.getUsed === used)
+    assert(smallUmm.executionMemoryUsed === used)
+    val resultRow = new UnsafeRow(1)
+    keys.foreach(k => assert(map.getValue(k, resultRow).getLong(0) === k))
+    assert(map.getValue(1L, resultRow) === null)
+
+    // A failed spill of another consumer is not a lack of memory, so 
optimize() does not hide it.
+    val failingConsumer = new MemoryConsumer(smallMm, 
smallMm.getTungstenMemoryMode) {
+      override def spill(size: Long, trigger: MemoryConsumer): Long = {
+        throw new IOException("Injected spill failure")
+      }
+    }
+    val remaining = memorySize - used
+    assert(failingConsumer.acquireMemory(remaining) === remaining)
+    val e = intercept[SparkOutOfMemoryError](map.optimize())
+    assert(e.getCondition === "SPILL_OUT_OF_MEMORY")
+    failingConsumer.freeMemory(remaining)
+    map.free()
+    assert(smallUmm.executionMemoryUsed === 0)
+  }
 }
 
 class HashedRelationOnHeapSuite extends HashedRelationSuite {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to