This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 07aeda94f26e [SPARK-61078][SQL] Skip the throwaway allocations in 
`LongToUnsafeRowMap`'s deserialization constructor
07aeda94f26e is described below

commit 07aeda94f26e20f097aa07b1ad84af225aa3dbd2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 22:52:28 2026 -0800

    [SPARK-61078][SQL] Skip the throwaway allocations in `LongToUnsafeRowMap`'s 
deserialization constructor
    
    ### What changes were proposed in this pull request?
    
    This PR makes the no-arg constructor of `LongToUnsafeRowMap`, which 
`JavaSerializer` and `KryoSerializer` use to create the map before calling 
`readExternal` / `read(kryo, in)`, skip the allocations in `init()`.
    
    - The primary constructor becomes private and takes an `allocateOnInit` 
flag. A public auxiliary constructor keeps the existing `(mm, capacity, 
ignoresDuplicatedKey = false)` signature and passes `allocateOnInit = true`, so 
all existing callers are unchanged.
    - The no-arg constructor passes `allocateOnInit = false`, so `array` and 
`page` stay `null` until `read()` allocates them with the serialized sizes.
    - `read()` frees the existing `array` and `page` only when they are not 
`null`. It still frees them when `read()` is called on a map built with the 
regular constructor.
    
    ### Why are the changes needed?
    
    The primary constructor always runs `init()`, which allocates a 2-long 
array and an initial page of `INITIAL_PAGE_SIZE_BYTES` (1 MiB). The no-arg 
constructor goes through it too, so a deserialized `LongToUnsafeRowMap` 
allocates both in its constructor, and `read()` frees them right away before 
allocating the real ones. Nothing ever uses these allocations.
    
    The cost is small. A broadcast `LongHashedRelation` is deserialized once 
per executor, not once per task. The 1 MiB page is at the pooling threshold of 
`HeapMemoryAllocator`, so after `freePage` the array goes back to the JVM-wide 
pool, and a later allocation of the same size, such as the next 
deserialization, can reuse it until it is garbage-collected. This PR is a small 
cleanup that removes the unused allocations.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. It only removes the unused allocation when a `LongToUnsafeRowMap` is 
deserialized.
    
    ### How was this patch tested?
    
    Pass the CIs with a newly added test case in `HashedRelationSuite`, which 
runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. It 
checks that a map created the way `JavaSerializer` (the no-arg constructor) and 
`KryoSerializer` (`Kryo.newInstance`) create it has not allocated any memory 
before `read()`.
    
    The test fails without this fix because the map already holds the 16-byte 
array and the 1 MiB page:
    
    ```
    $ build/sbt "sql/testOnly 
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite 
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite"
    ...
    [info] - SPARK-61078: LongToUnsafeRowMap does not allocate memory before 
deserialization *** FAILED *** (10 milliseconds)
    [info]   1048592 did not equal 0 (HashedRelationSuite.scala:467)
    ...
    [info] *** 2 TESTS FAILED ***
    ```
    
    The existing test cases cover `read()` with `null` `array` and `page`: 
`LongToUnsafeRowMap` and `SPARK-24809` (Java serialization, including 
serializing a deserialized map again) and `Spark-14521` (`KryoSerializer`). 
`LongToUnsafeRowMap with random keys` covers `read()` on a map built with the 
regular constructor.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60195 from dongjoon-hyun/SPARK-61078.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit b06b99c6d45a175e63d2271f43aa5a3d1dfd0853)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/execution/joins/HashedRelation.scala  | 21 ++++++++++++++-------
 .../sql/execution/joins/HashedRelationSuite.scala   |  8 ++++++++
 2 files changed, 22 insertions(+), 7 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index b388d2aff22b..396ef237a05d 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -533,10 +533,11 @@ private[joins] object UnsafeHashedRelation {
  *
  * see 
http://java-performance.info/implementing-world-fastest-java-int-to-int-hash-map/
  */
-private[execution] final class LongToUnsafeRowMap(
+private[execution] final class LongToUnsafeRowMap private (
     val mm: TaskMemoryManager,
     capacity: Int,
-    ignoresDuplicatedKey: Boolean = false)
+    ignoresDuplicatedKey: Boolean,
+    allocateOnInit: Boolean)
   extends MemoryConsumer(mm, LongToUnsafeRowMap.MIN_PAGE_SIZE_BYTES, 
mm.getTungstenMemoryMode)
   with Externalizable with KryoSerializable {
 
@@ -573,7 +574,11 @@ private[execution] final class LongToUnsafeRowMap(
   // The number of unique keys.
   private var numKeys = 0L
 
-  // needed by serializer
+  def this(mm: TaskMemoryManager, capacity: Int, ignoresDuplicatedKey: Boolean 
= false) =
+    this(mm, capacity, ignoresDuplicatedKey, allocateOnInit = true)
+
+  // needed by serializer. `read()` allocates `array` and `page` with the 
serialized sizes,
+  // so skip the allocations in `init()`.
   def this() = {
     this(
       new TaskMemoryManager(
@@ -583,7 +588,9 @@ private[execution] final class LongToUnsafeRowMap(
           Runtime.getRuntime.maxMemory / 2,
           1),
         0),
-      0)
+      0,
+      ignoresDuplicatedKey = false,
+      allocateOnInit = false)
   }
 
   private def init(): Unit = {
@@ -599,7 +606,7 @@ private[execution] final class LongToUnsafeRowMap(
     }
   }
 
-  init()
+  if (allocateOnInit) init()
 
   def spill(size: Long, trigger: MemoryConsumer): Long = 0L
 
@@ -976,11 +983,11 @@ private[execution] final class LongToUnsafeRowMap(
 
     val length = readLong().toInt
     mask = length - 2
-    freeArray(array)
+    if (array != null) freeArray(array)
     array = allocateArray(length)
     readData(readBuffer, array.memoryBlock.getBaseObject, 
array.memoryBlock.getBaseOffset, length)
     val pageLength = readLong().toInt
-    freePage(page)
+    if (page != null) freePage(page)
     // Use Long multiplication: pageLength can be up to 1 << 30 (8 GiB page / 
8),
     // and `Int * Int` overflows at that bound, leading to a 0-byte 
allocatePage
     // and a subsequent cursor that runs past the page's end.
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index a6ae728e7ccd..23c41f4ed77b 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -461,6 +461,14 @@ abstract class HashedRelationSuite extends 
SharedSparkSession {
     assert(java.util.Arrays.equals(os.toByteArray, os2.toByteArray))
   }
 
+  test("SPARK-61078: LongToUnsafeRowMap does not allocate memory before 
deserialization") {
+    // JavaSerializer creates the map with the no-arg constructor, and 
KryoSerializer creates it
+    // with `Kryo.newInstance`. Then `read()` allocates `array` and `page`.
+    assert(new LongToUnsafeRowMap().getUsed === 0)
+    val kryo = new KryoSerializer(new SparkConf).newKryo()
+    assert(kryo.newInstance(classOf[LongToUnsafeRowMap]).getUsed === 0)
+  }
+
   test("SPARK-31511: Make BytesToBytesMap iterators thread-safe") {
     val ser = sparkContext.env.serializer.newInstance()
     val key = Seq(BoundReference(0, LongType, false))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to