dongjoon-hyun opened a new pull request, #60191: URL: https://github.com/apache/spark/pull/60191
### What changes were proposed in this pull request? This PR makes `LongToUnsafeRowMap.optimize()` keep the map in sparse mode when it cannot allocate the dense array, instead of failing the task. It catches the `SparkOutOfMemoryError` thrown by `allocateArray` and returns, as `optimize()` did before SPARK-54116. `MemoryConsumer.allocateArray` releases any partially acquired page in `throwOom` before it throws, so nothing leaks when the conversion is skipped. ### Why are the changes needed? Dense mode is only a lookup optimization. In [v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11) and earlier, `optimize()` reserved the memory with `ensureAcquireMemory` and returned on `SparkException`, so a map that could not get the extra memory stayed sparse. SPARK-54116 replaced that with `allocateArray`, which throws `SparkOutOfMemoryError`, and `optimize()` no longer catches it. The dense array is allocated while the sparse array is still alive, and it can be as large as the sparse array. So a task that has enough execution memory to build the map now fails with `UNABLE_TO_ACQUIRE_MEMORY` at the end of the build, where Spark 4.1 succeeded. This affects `LongHashedRelation`s built with the task's memory manager, such as the build side of a shuffled hash join on a long key. The regression is in [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11), [v4.2.1](https://github.com/apache/spark/releases/tag/v4.2.1) (2026-10-21) and [v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27). ### Does this PR introduce _any_ user-facing change? Yes. A task whose execution memory is too small for the dense conversion no longer fails with `SparkOutOfMemoryError`. The hashed relation stays in sparse mode, as in Spark 4.1. ### How was this patch tested? Pass the CIs with a newly added test case in `HashedRelationSuite`, which runs in both `HashedRelationOnHeapSuite` and `HashedRelationOffHeapSuite`. The test builds a map whose sparse array and page fit in a small execution-memory budget while the dense array does not. It then checks that `optimize()` keeps the map sparse and usable without leaking memory. The test fails without this fix: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite" ... [info] Cause: org.apache.spark.memory.SparkOutOfMemoryError: [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 507912 bytes of memory, got 262144. ... [info] *** 2 TESTS FAILED *** ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
