dongjoon-hyun opened a new pull request, #60195: URL: https://github.com/apache/spark/pull/60195
[SPARK-61078][SQL] Skip the throwaway allocations in `LongToUnsafeRowMap`'s deserialization constructor ### What changes were proposed in this pull request? This PR makes the no-arg constructor of `LongToUnsafeRowMap`, which `JavaSerializer` and `KryoSerializer` use to create the map before calling `readExternal` / `read(kryo, in)`, skip the allocations in `init()`. - The primary constructor becomes private and takes an `allocateOnInit` flag. A public auxiliary constructor keeps the existing `(mm, capacity, ignoresDuplicatedKey = false)` signature and passes `allocateOnInit = true`, so all existing callers are unchanged. - The no-arg constructor passes `allocateOnInit = false`, so `array` and `page` stay `null` until `read()` allocates them with the serialized sizes. - `read()` frees the existing `array` and `page` only when they are not `null`. It still frees them when `read()` is called on a map built with the regular constructor. ### Why are the changes needed? The primary constructor always runs `init()`, which allocates a 2-long array and an initial page of `INITIAL_PAGE_SIZE_BYTES` (1 MiB). The no-arg constructor goes through it too, so every deserialized `LongToUnsafeRowMap` allocates 1 MiB and 16 bytes in its constructor (an on-heap `long[]`, because the no-arg constructor's `TaskMemoryManager` is on-heap). `read()` then frees both right away and allocates the real ones. This happens on executors for every broadcast `LongHashedRelation` and only adds allocation and GC work. In the released versions since SPARK-54116 ([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11), this page was also rounded up to the page size of the no-arg constructor's memory manager, which is derived from the JVM heap and reaches 64 MiB once the heap is larger than about 1 GiB. SPARK-60998 removed the rounding, but the 1 MiB throwaway page remains. ### Does this PR introduce _any_ user-facing change? No. It only removes the unused allocation when a `LongToUnsafeRowMap` is deserialized. ### How was this patch tested? Pass the CIs with a newly added test case in `HashedRelationSuite`, which runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. It checks that a map created the way `JavaSerializer` (the no-arg constructor) and `KryoSerializer` (`Kryo.newInstance`) create it has not allocated any memory before `read()`. The test fails without this fix because the map already holds the 16-byte array and the 1 MiB page: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite" ... [info] - SPARK-61078: LongToUnsafeRowMap does not allocate memory before deserialization *** FAILED *** (10 milliseconds) [info] 1048592 did not equal 0 (HashedRelationSuite.scala:467) ... [info] *** 2 TESTS FAILED *** ``` The existing test cases cover `read()` with `null` `array` and `page`: `LongToUnsafeRowMap` and `SPARK-24809` (Java serialization, including serializing a deserialized map again) and `Spark-14521` (`KryoSerializer`). `LongToUnsafeRowMap with random keys` covers `read()` on a map built with the regular constructor. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
