This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 07aeda94f26e [SPARK-61078][SQL] Skip the throwaway allocations in
`LongToUnsafeRowMap`'s deserialization constructor
07aeda94f26e is described below
commit 07aeda94f26e20f097aa07b1ad84af225aa3dbd2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 22:52:28 2026 -0800
[SPARK-61078][SQL] Skip the throwaway allocations in `LongToUnsafeRowMap`'s
deserialization constructor
### What changes were proposed in this pull request?
This PR makes the no-arg constructor of `LongToUnsafeRowMap`, which
`JavaSerializer` and `KryoSerializer` use to create the map before calling
`readExternal` / `read(kryo, in)`, skip the allocations in `init()`.
- The primary constructor becomes private and takes an `allocateOnInit`
flag. A public auxiliary constructor keeps the existing `(mm, capacity,
ignoresDuplicatedKey = false)` signature and passes `allocateOnInit = true`, so
all existing callers are unchanged.
- The no-arg constructor passes `allocateOnInit = false`, so `array` and
`page` stay `null` until `read()` allocates them with the serialized sizes.
- `read()` frees the existing `array` and `page` only when they are not
`null`. It still frees them when `read()` is called on a map built with the
regular constructor.
### Why are the changes needed?
The primary constructor always runs `init()`, which allocates a 2-long
array and an initial page of `INITIAL_PAGE_SIZE_BYTES` (1 MiB). The no-arg
constructor goes through it too, so a deserialized `LongToUnsafeRowMap`
allocates both in its constructor, and `read()` frees them right away before
allocating the real ones. Nothing ever uses these allocations.
The cost is small. A broadcast `LongHashedRelation` is deserialized once
per executor, not once per task. The 1 MiB page is at the pooling threshold of
`HeapMemoryAllocator`, so after `freePage` the array goes back to the JVM-wide
pool, and a later allocation of the same size, such as the next
deserialization, can reuse it until it is garbage-collected. This PR is a small
cleanup that removes the unused allocations.
### Does this PR introduce _any_ user-facing change?
No. It only removes the unused allocation when a `LongToUnsafeRowMap` is
deserialized.
### How was this patch tested?
Pass the CIs with a newly added test case in `HashedRelationSuite`, which
runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. It
checks that a map created the way `JavaSerializer` (the no-arg constructor) and
`KryoSerializer` (`Kryo.newInstance`) create it has not allocated any memory
before `read()`.
The test fails without this fix because the map already holds the 16-byte
array and the 1 MiB page:
```
$ build/sbt "sql/testOnly
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite"
...
[info] - SPARK-61078: LongToUnsafeRowMap does not allocate memory before
deserialization *** FAILED *** (10 milliseconds)
[info] 1048592 did not equal 0 (HashedRelationSuite.scala:467)
...
[info] *** 2 TESTS FAILED ***
```
The existing test cases cover `read()` with `null` `array` and `page`:
`LongToUnsafeRowMap` and `SPARK-24809` (Java serialization, including
serializing a deserialized map again) and `Spark-14521` (`KryoSerializer`).
`LongToUnsafeRowMap with random keys` covers `read()` on a map built with the
regular constructor.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60195 from dongjoon-hyun/SPARK-61078.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit b06b99c6d45a175e63d2271f43aa5a3d1dfd0853)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/execution/joins/HashedRelation.scala | 21 ++++++++++++++-------
.../sql/execution/joins/HashedRelationSuite.scala | 8 ++++++++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index b388d2aff22b..396ef237a05d 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -533,10 +533,11 @@ private[joins] object UnsafeHashedRelation {
*
* see
http://java-performance.info/implementing-world-fastest-java-int-to-int-hash-map/
*/
-private[execution] final class LongToUnsafeRowMap(
+private[execution] final class LongToUnsafeRowMap private (
val mm: TaskMemoryManager,
capacity: Int,
- ignoresDuplicatedKey: Boolean = false)
+ ignoresDuplicatedKey: Boolean,
+ allocateOnInit: Boolean)
extends MemoryConsumer(mm, LongToUnsafeRowMap.MIN_PAGE_SIZE_BYTES,
mm.getTungstenMemoryMode)
with Externalizable with KryoSerializable {
@@ -573,7 +574,11 @@ private[execution] final class LongToUnsafeRowMap(
// The number of unique keys.
private var numKeys = 0L
- // needed by serializer
+ def this(mm: TaskMemoryManager, capacity: Int, ignoresDuplicatedKey: Boolean
= false) =
+ this(mm, capacity, ignoresDuplicatedKey, allocateOnInit = true)
+
+ // needed by serializer. `read()` allocates `array` and `page` with the
serialized sizes,
+ // so skip the allocations in `init()`.
def this() = {
this(
new TaskMemoryManager(
@@ -583,7 +588,9 @@ private[execution] final class LongToUnsafeRowMap(
Runtime.getRuntime.maxMemory / 2,
1),
0),
- 0)
+ 0,
+ ignoresDuplicatedKey = false,
+ allocateOnInit = false)
}
private def init(): Unit = {
@@ -599,7 +606,7 @@ private[execution] final class LongToUnsafeRowMap(
}
}
- init()
+ if (allocateOnInit) init()
def spill(size: Long, trigger: MemoryConsumer): Long = 0L
@@ -976,11 +983,11 @@ private[execution] final class LongToUnsafeRowMap(
val length = readLong().toInt
mask = length - 2
- freeArray(array)
+ if (array != null) freeArray(array)
array = allocateArray(length)
readData(readBuffer, array.memoryBlock.getBaseObject,
array.memoryBlock.getBaseOffset, length)
val pageLength = readLong().toInt
- freePage(page)
+ if (page != null) freePage(page)
// Use Long multiplication: pageLength can be up to 1 << 30 (8 GiB page /
8),
// and `Int * Int` overflows at that bound, leading to a 0-byte
allocatePage
// and a subsequent cursor that runs past the page's end.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index a6ae728e7ccd..23c41f4ed77b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -461,6 +461,14 @@ abstract class HashedRelationSuite extends
SharedSparkSession {
assert(java.util.Arrays.equals(os.toByteArray, os2.toByteArray))
}
+ test("SPARK-61078: LongToUnsafeRowMap does not allocate memory before
deserialization") {
+ // JavaSerializer creates the map with the no-arg constructor, and
KryoSerializer creates it
+ // with `Kryo.newInstance`. Then `read()` allocates `array` and `page`.
+ assert(new LongToUnsafeRowMap().getUsed === 0)
+ val kryo = new KryoSerializer(new SparkConf).newKryo()
+ assert(kryo.newInstance(classOf[LongToUnsafeRowMap]).getUsed === 0)
+ }
+
test("SPARK-31511: Make BytesToBytesMap iterators thread-safe") {
val ser = sparkContext.env.serializer.newInstance()
val key = Seq(BoundReference(0, LongType, false))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]