This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 3dbf1dc18976 [SPARK-61082][SQL] Fix UnsafeHashedRelation build failure 
when many small relations are alive in one task
3dbf1dc18976 is described below

commit 3dbf1dc189766a74b053d60cd340c2633074999a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 7 12:07:52 2026 -0800

    [SPARK-61082][SQL] Fix UnsafeHashedRelation build failure when many small 
relations are alive in one task
    
    ### What changes were proposed in this pull request?
    
    This PR lets `BytesToBytesMap` start with a small data page and grow its 
pages by doubling, and uses it in `UnsafeHashedRelation`, the hash relation for 
join keys other than a single `LONG` (for example a `STRING` key or multiple 
key columns).
    
    - `BytesToBytesMap`: a new constructor takes `initialPageSizeBytes` after 
`pageSizeBytes`. The first data page has that size (at most `pageSizeBytes`), 
and each later page doubles until the pages add up to `pageSizeBytes`. The last 
of them is cut to what is left, and all pages after them have `pageSizeBytes`. 
A record larger than the next page still gets a page of its own size, and 
`reset()` starts again from the initial page size. The existing constructors 
pass `pageSizeBytes` as the [...]
    - `MemoryConsumer`: a new `allocatePage(size, required)` requests `size` 
bytes and, like `allocatePage(required)`, accepts a smaller page of at least 
`required` bytes. `allocatePage(required)` now calls it with `max(pageSize, 
required)`. `BytesToBytesMap` needs both values so that a partial memory grant 
still works when the requested page size grows.
    - `UnsafeHashedRelation`: both building (`apply`) and deserializing 
(`read`) use a 1 MiB initial page, 
`UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES`, the same size as 
`LongToUnsafeRowMap.INITIAL_PAGE_SIZE_BYTES` from SPARK-60998.
    
    For a large relation with a 64 MiB page size, the pages are 1, 2, 4, 8, 16, 
32 and 1 MiB, then 64 MiB each. Because the growing pages end exactly at one 
page size, they do not move the sizes at which a relation needs another full 
page. The relation needs 7 more pages than before, which is small next to the 
limit of 8192 pages per task.
    
    The growth is opt-in because changing every `BytesToBytesMap` would also 
change when hash aggregation falls back to sort-based aggregation. A smaller 
fixed page size in `UnsafeHashedRelation` alone is not enough either: each 
relation would still reserve a whole page, and a large relation would need many 
more pages.
    
    ### Why are the changes needed?
    
    `BytesToBytesMap` allocates its data pages through 
`MemoryConsumer.allocatePage`, which rounds every request up to `max(pageSize, 
required)`. `UnsafeHashedRelation` passes 
`SparkEnv.get.memoryManager.pageSizeBytes`, which is `spark.buffer.pageSize` or 
a value of up to 64 MiB derived from the executor memory and cores. So every 
relation reserves a full page, even when it holds only a few rows.
    
    - A shuffled hash join keeps its relation until the task ends, and the 
relation cannot spill. A task with a chain of shuffled hash joins on such keys 
runs out of execution memory long before the relations hold that much data:
    
      ```
      [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 2048 bytes of memory, got 0.
      Memory used by task 0 grouped by consumer:
        org.apache.spark.unsafe.map.BytesToBytesMap5aa9a000: 64.0 MiB
        org.apache.spark.unsafe.map.BytesToBytesMap6bc9c3f9: 64.0 MiB
        org.apache.spark.unsafe.map.BytesToBytesMap5240e37f: 64.0 MiB
        org.apache.spark.unsafe.map.BytesToBytesMap31cf8e3c: 64.0 MiB
      ```
    
      This is the output of the new test `SPARK-61082: many 
UnsafeHashedRelations fit in one task when each holds little data` without this 
fix: 4 relations of 10 rows each take the whole 256 MiB, and the 5th cannot 
allocate its 2 KiB pointer array.
    
    - A broadcast hash join builds the relation on the driver and deserializes 
it on each executor with an on-heap memory manager that is not bounded by 
execution memory. So every broadcast relation allocates a full page as a 
`long[]` on the heap, 64 MiB even for a single row.
    
    SPARK-60998 fixed the same problem for `LongToUnsafeRowMap`, which is used 
for a single `LONG` key. For `UnsafeHashedRelation`, this is not a regression: 
it has rounded its pages up to the page size since Spark 2.0, when it started 
to build a `BytesToBytesMap` directly (SPARK-14052).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Queries that keep many `UnsafeHashedRelation`s in one task, such as a 
chain of shuffled hash joins on non-`LONG` keys, no longer fail with 
`UNABLE_TO_ACQUIRE_MEMORY` when the relations are small. Each relation reserves 
memory close to what it stores, and a small broadcast relation uses much less 
driver and executor heap. For small relations, the `data size of build side` 
metric of `ShuffledHashJoin` and the `data size` metric of `BroadcastExchange` 
are smaller accordingly. Hash a [...]
    
    ### How was this patch tested?
    
    New tests:
    
    - `HashedRelationSuite`, which runs in both `HashedRelationOnHeapSuite` and 
`HashedRelationOffHeapSuite`:
      - `SPARK-61082: many UnsafeHashedRelations fit in one task when each 
holds little data`: 16 relations of 10 rows each on a `STRING` key, in 256 MiB 
of execution memory. The test first checks that the page size from `SparkEnv` 
is larger than 16 MiB, so that it covers the bug.
      - `SPARK-61082: UnsafeHashedRelation does not round its page up to the 
page size`: a relation with one row and its copies deserialized by the Java and 
the Kryo serializer each reserve less than 2 MiB, instead of about 64 MiB, and 
their maps grow their pages up to the `SparkEnv` page size.
    - `AbstractBytesToBytesMapSuite`, which runs in both 
`BytesToBytesMapOnHeapSuite` and `BytesToBytesMapOffHeapSuite`:
      - `dataPagesGrowFromInitialPageSize`: with a 1 KiB initial page and a 6 
KiB page size, the pages are 1, 2 and 3 KiB, then 6 KiB. After `reset()`, the 
first page is 1 KiB again, and a record larger than the next page size (2 KiB) 
gets a page of exactly its size. A map whose initial page size is larger than 
its page size starts with a page of the page size.
      - `initialCapacityBoundsChecking` also checks that the initial page size 
must be positive.
    
    Both `HashedRelationSuite` tests fail without the fix in both memory modes. 
The existing suites for `BytesToBytesMap`, `TaskMemoryManager`, hash 
aggregation and joins also pass.
    
    ```
    build/sbt "core/testOnly 
org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite 
org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite 
org.apache.spark.memory.TaskMemoryManagerSuite"
    build/sbt "sql/testOnly 
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite 
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite 
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMapSuite 
org.apache.spark.sql.execution.UnsafeKVExternalSorterSuite"
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60196 from dongjoon-hyun/SPARK-61082.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    
    (cherry picked from commit 874fd21c4dfaac3ed24e2dae07fd14455a9a827a)
    Signed-off-by: Dongjoon Hyun <[email protected]>
    
    
    skip-checks: true
---
 .../org/apache/spark/memory/MemoryConsumer.java    | 13 +++-
 .../apache/spark/unsafe/map/BytesToBytesMap.java   | 83 ++++++++++++++++++++--
 .../unsafe/map/AbstractBytesToBytesMapSuite.java   | 55 ++++++++++++++
 .../spark/sql/execution/joins/HashedRelation.scala | 21 +++++-
 .../sql/execution/joins/HashedRelationSuite.scala  | 67 +++++++++++++++++
 5 files changed, 231 insertions(+), 8 deletions(-)

diff --git a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java 
b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
index 3ec3c5cdf084..696d24a7b652 100644
--- a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
+++ b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
@@ -128,8 +128,17 @@ public abstract class MemoryConsumer {
    * @throws SparkOutOfMemoryError
    */
   protected MemoryBlock allocatePage(long required) {
-    MemoryBlock page =
-      taskMemoryManager.allocatePageWithMinimum(Math.max(pageSize, required), 
required, this);
+    return allocatePage(Math.max(pageSize, required), required);
+  }
+
+  /**
+   * Allocate a memory block of `size` bytes. A smaller block may be returned, 
but it has at least
+   * `required` bytes, which must not exceed `size`.
+   *
+   * @throws SparkOutOfMemoryError
+   */
+  protected MemoryBlock allocatePage(long size, long required) {
+    MemoryBlock page = taskMemoryManager.allocatePageWithMinimum(size, 
required, this);
     if (page == null || page.size() < required) {
       throwOom(page, required);
     }
diff --git 
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index b9ef89292c2a..850ff34219c5 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -170,11 +170,30 @@ public final class BytesToBytesMap extends MemoryConsumer 
{
   @Nullable private final KeyOperationsFactory keyOperationsFactory;
 
   /**
-   * The size of the data pages that hold key and value data. Map entries 
cannot span multiple
-   * pages, so this limits the maximum entry size.
+   * The maximum size of the regular data pages that hold key and value data. 
A map with a smaller
+   * initial page size grows its pages up to this size. Map entries cannot 
span multiple pages, so
+   * a record larger than a page gets a page of its own size.
    */
   private final long pageSizeBytes;
 
+  /**
+   * The size of the first data page, at most {@link #pageSizeBytes}.
+   */
+  private final long initialPageSizeBytes;
+
+  /**
+   * The size of the next data page while the map grows. It starts at
+   * {@link #initialPageSizeBytes} and doubles with every new page, up to 
{@link #pageSizeBytes}.
+   */
+  private long nextPageSizeBytes;
+
+  /**
+   * The total size requested for the data pages so far. While it is below
+   * {@link #pageSizeBytes}, a new page is at most what is left to reach it, 
so the growing pages
+   * add up to exactly one page size before the pages of {@link 
#pageSizeBytes} begin.
+   */
+  private long requestedDataPagesBytes;
+
   /**
    * Number of keys defined in the map.
    */
@@ -240,6 +259,26 @@ public final class BytesToBytesMap extends MemoryConsumer {
       double loadFactor,
       long pageSizeBytes,
       @Nullable KeyOperationsFactory keyOperationsFactory) {
+    this(
+      taskMemoryManager,
+      blockManager,
+      serializerManager,
+      initialCapacity,
+      loadFactor,
+      pageSizeBytes,
+      pageSizeBytes,
+      keyOperationsFactory);
+  }
+
+  private BytesToBytesMap(
+      TaskMemoryManager taskMemoryManager,
+      BlockManager blockManager,
+      SerializerManager serializerManager,
+      int initialCapacity,
+      double loadFactor,
+      long pageSizeBytes,
+      long initialPageSizeBytes,
+      @Nullable KeyOperationsFactory keyOperationsFactory) {
     super(taskMemoryManager, pageSizeBytes, 
taskMemoryManager.getTungstenMemoryMode());
     this.taskMemoryManager = taskMemoryManager;
     this.blockManager = blockManager;
@@ -259,10 +298,38 @@ public final class BytesToBytesMap extends MemoryConsumer 
{
       throw new IllegalArgumentException("Page size " + pageSizeBytes + " 
cannot exceed " +
         TaskMemoryManager.MAXIMUM_PAGE_SIZE_BYTES);
     }
+    if (initialPageSizeBytes <= 0) {
+      throw new IllegalArgumentException("Initial page size must be greater 
than 0");
+    }
+    this.initialPageSizeBytes = Math.min(initialPageSizeBytes, pageSizeBytes);
+    this.nextPageSizeBytes = this.initialPageSizeBytes;
     this.initialCapacity = initialCapacity;
     allocate(initialCapacity);
   }
 
+  /**
+   * Creates a map whose first data page has `initialPageSizeBytes` (at most 
`pageSizeBytes`), and
+   * whose later pages double in size until they add up to `pageSizeBytes`. 
The last of them is cut
+   * to what is left, and all pages after them have `pageSizeBytes`. A map 
holding little data then
+   * does not reserve a full page. A record larger than the next page gets a 
page of its size.
+   */
+  public BytesToBytesMap(
+      TaskMemoryManager taskMemoryManager,
+      int initialCapacity,
+      long pageSizeBytes,
+      long initialPageSizeBytes) {
+    this(
+      taskMemoryManager,
+      SparkEnv.get() != null ? SparkEnv.get().blockManager() :  null,
+      SparkEnv.get() != null ? SparkEnv.get().serializerManager() :  null,
+      initialCapacity,
+      // In order to re-use the longArray for sorting, the load factor cannot 
be larger than 0.5.
+      0.5,
+      pageSizeBytes,
+      initialPageSizeBytes,
+      null);
+  }
+
   public BytesToBytesMap(
       TaskMemoryManager taskMemoryManager,
       int initialCapacity,
@@ -1037,18 +1104,24 @@ public final class BytesToBytesMap extends 
MemoryConsumer {
    * @return whether there is enough space to allocate the new page.
    */
   private boolean acquireNewPage(long required) {
+    final long remainingGrowthBytes = pageSizeBytes - requestedDataPagesBytes;
+    final long size = Math.max(
+      remainingGrowthBytes > 0 ? Math.min(nextPageSizeBytes, 
remainingGrowthBytes) : pageSizeBytes,
+      required);
     final MemoryBlock page;
     try {
-      page = allocatePage(required);
+      page = allocatePage(size, required);
     } catch (SparkOutOfMemoryError e) {
       return false;
     }
     // Retaining exact-fit minimum-retry pages would consume one page-table 
slot per map entry.
-    if (required < pageSizeBytes && page.size() == required &&
+    if (required < size && page.size() == required &&
         isPageAllocationFromMinimumRetry(page)) {
       freePage(page);
       return false;
     }
+    requestedDataPagesBytes += size;
+    nextPageSizeBytes = Math.min(pageSizeBytes, nextPageSizeBytes * 2);
     currentPage = page;
     dataPages.add(currentPage);
     UnsafeAlignedOffset.putSize(currentPage.getBaseObject(), 
currentPage.getBaseOffset(), 0);
@@ -1194,6 +1267,8 @@ public final class BytesToBytesMap extends MemoryConsumer 
{
     canGrowArray = true;
     currentPage = null;
     pageCursor = 0;
+    nextPageSizeBytes = initialPageSizeBytes;
+    requestedDataPagesBytes = 0;
     if (longArray != null) {
       freeArray(longArray);
     }
diff --git 
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
 
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
index 0fee9b91f4eb..6612e4cb6755 100644
--- 
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
+++ 
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
@@ -51,6 +51,7 @@ import org.apache.spark.serializer.SerializerInstance;
 import org.apache.spark.serializer.SerializerManager;
 import org.apache.spark.storage.*;
 import org.apache.spark.unsafe.Platform;
+import org.apache.spark.unsafe.UnsafeAlignedOffset;
 import org.apache.spark.unsafe.array.ByteArrayMethods;
 import org.apache.spark.util.Utils;
 import org.apache.spark.internal.config.package$;
@@ -772,6 +773,8 @@ public abstract class AbstractBytesToBytesMapSuite {
     assertThrows(IllegalArgumentException.class,
       () -> new BytesToBytesMap(taskMemoryManager, 1,
               TaskMemoryManager.MAXIMUM_PAGE_SIZE_BYTES + 1));
+    assertThrows(IllegalArgumentException.class,
+      () -> new BytesToBytesMap(taskMemoryManager, 1, PAGE_SIZE_BYTES, 0));
   }
 
   @Test
@@ -817,6 +820,58 @@ public abstract class AbstractBytesToBytesMapSuite {
     }
   }
 
+  @Test
+  public void dataPagesGrowFromInitialPageSize() {
+    final BytesToBytesMap map = new BytesToBytesMap(taskMemoryManager, 1, 
6144, 1024);
+    try {
+      // The pages double from 1 KiB, and the last growing page is cut to the 
3 KiB left, so the
+      // growing pages add up to exactly one page size before the pages of 6 
KiB begin.
+      assertEquals(1024, appendUntilNewDataPage(map, 1));
+      assertEquals(2048, appendUntilNewDataPage(map, 1));
+      assertEquals(3072, appendUntilNewDataPage(map, 1));
+      assertEquals(6144, appendUntilNewDataPage(map, 1));
+      assertEquals(6144, appendUntilNewDataPage(map, 1));
+
+      // reset() starts again from the initial page size.
+      map.reset();
+      assertEquals(1024, appendUntilNewDataPage(map, 1));
+      // A record larger than the next page size, 2 KiB, gets a page of its 
own size: the record
+      // count of the page, the two lengths, the key, the value and the 
pointer to the next value.
+      final int largeValueWords = 300;
+      assertEquals(3L * UnsafeAlignedOffset.getUaoSize() + 8 + largeValueWords 
* 8L + 8,
+        appendUntilNewDataPage(map, largeValueWords));
+    } finally {
+      map.free();
+    }
+
+    // An initial page size larger than the page size is capped at the page 
size.
+    final BytesToBytesMap cappedMap = new BytesToBytesMap(taskMemoryManager, 
1, 1024, 4096);
+    try {
+      assertEquals(1024, appendUntilNewDataPage(cappedMap, 1));
+    } finally {
+      cappedMap.free();
+    }
+  }
+
+  /**
+   * Appends records with a value of `valueWords` longs until the map 
allocates a new data page, and
+   * returns the size of that page. All records share one key, so the pointer 
array does not grow
+   * and every increase of the memory consumption is a new data page.
+   */
+  private static long appendUntilNewDataPage(BytesToBytesMap map, int 
valueWords) {
+    final long[] key = new long[]{0};
+    final long[] value = new long[valueWords];
+    final long previousMemory = map.getTotalMemoryConsumption();
+    while (true) {
+      assertTrue(map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8).append(
+        key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 
valueWords * 8));
+      final long memory = map.getTotalMemoryConsumption();
+      if (memory > previousMemory) {
+        return memory - previousMemory;
+      }
+    }
+  }
+
   @Test
   public void avoidDeadlock() throws InterruptedException {
     memoryManager.limit(PAGE_SIZE_BYTES);
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index 396ef237a05d..ae5e3cda25c8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -222,6 +222,9 @@ private[joins] class UnsafeHashedRelation(
 
   override def estimatedSize: Long = binaryMap.getTotalMemoryConsumption
 
+  // Visible for testing: the page size that the map grows its data pages up 
to.
+  private[joins] def mapPageSizeBytes: Long = binaryMap.getPageSizeBytes
+
   // re-used in 
get()/getValue()/getWithKeyIndex()/getValueWithKeyIndex()/valuesWithKeyIndex()
   var resultRow = new UnsafeRow(numFields)
 
@@ -417,7 +420,8 @@ private[joins] class UnsafeHashedRelation(
     binaryMap = new BytesToBytesMap(
       taskMemoryManager,
       (nKeys * 1.5 + 1).toInt, // reduce hash collision
-      pageSizeBytes)
+      pageSizeBytes,
+      UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
 
     var i = 0
     var keyBuffer = new Array[Byte](1024)
@@ -452,6 +456,18 @@ private[joins] class UnsafeHashedRelation(
 
 private[joins] object UnsafeHashedRelation {
 
+  /**
+   * The size of the first data page of the map. Later pages double in size 
until they add up to
+   * the page size, and then all pages have the page size.
+   *
+   * Otherwise `BytesToBytesMap` rounds its first page up to the page size 
(`spark.buffer.pageSize`
+   * or a default of up to 64 MiB), so even a few rows reserve a full page. A 
task that keeps many
+   * relations alive at once, e.g. a stage with a chain of shuffled hash 
joins, then runs out of
+   * execution memory long before the relations hold that much data. Each 
broadcast relation would
+   * also allocate a full page on the heap of the driver and of every executor.
+   */
+  val INITIAL_PAGE_SIZE_BYTES: Long = 1L << 20
+
   def apply(
       input: Iterator[InternalRow],
       key: Seq[Expression],
@@ -469,7 +485,8 @@ private[joins] object UnsafeHashedRelation {
       taskMemoryManager,
       // Only 70% of the slots can be used before growing, more capacity help 
to reduce collision
       (sizeEstimate * 1.5 + 1).toInt,
-      pageSizeBytes)
+      pageSizeBytes,
+      INITIAL_PAGE_SIZE_BYTES)
 
     // Create a mapping of buildKeys -> rows
     val keyGenerator = UnsafeProjection.create(key)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index 23c41f4ed77b..f81203ed17f9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -24,6 +24,7 @@ import scala.collection.mutable.ArrayBuffer
 import scala.util.Random
 
 import org.apache.spark.SparkConf
+import org.apache.spark.SparkEnv
 import org.apache.spark.SparkException
 import org.apache.spark.internal.config.{BUFFER_PAGESIZE, 
MEMORY_OFFHEAP_ENABLED, MEMORY_OFFHEAP_SIZE}
 import org.apache.spark.internal.config.Kryo._
@@ -141,6 +142,72 @@ abstract class HashedRelationSuite extends 
SharedSparkSession {
     assert(java.util.Arrays.equals(os2.toByteArray, os.toByteArray))
   }
 
+  test("SPARK-61082: many UnsafeHashedRelations fit in one task when each 
holds little data") {
+    // UnsafeHashedRelation takes its page size from SparkEnv, not from this 
memory manager. With a
+    // page larger than 16 MiB, 16 relations rounded up to a full page each 
would need more than
+    // the 256 MiB available.
+    assert(SparkEnv.get.memoryManager.pageSizeBytes > ByteUnit.MiB.toBytes(16))
+    val smallUmm = newMemoryManager(ByteUnit.MiB.toBytes(64), 
ByteUnit.MiB.toBytes(256))
+    val smallMm = new TaskMemoryManager(smallUmm, 0)
+    val buildKey = Seq(BoundReference(0, StringType, false))
+    val toUnsafeRow = UnsafeProjection.create(Array[DataType](StringType, 
IntegerType))
+    val toUnsafeKey = UnsafeProjection.create(Array[DataType](StringType))
+    val relations = ArrayBuffer.empty[HashedRelation]
+    try {
+      (0 until 16).foreach { i =>
+        val rows = (0 until 10).map { k =>
+          toUnsafeRow(InternalRow(UTF8String.fromString(s"key$k"), k + 
i)).copy()
+        }
+        val relation = HashedRelation(rows.iterator, buildKey, 
taskMemoryManager = smallMm)
+        assert(relation.isInstanceOf[UnsafeHashedRelation])
+        relations += relation
+      }
+      relations.zipWithIndex.foreach { case (relation, i) =>
+        assert(relation.estimatedSize < 2 * 
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+        (0 until 10).foreach { k =>
+          val key = toUnsafeKey(InternalRow(UTF8String.fromString(s"key$k")))
+          assert(relation.getValue(key).getInt(1) === k + i)
+        }
+      }
+    } catch {
+      // SparkOutOfMemoryError is fatal to ScalaTest and would abort the whole 
suite.
+      case e: SparkOutOfMemoryError =>
+        fail("Could not build 16 small UnsafeHashedRelations in one task", e)
+    } finally {
+      relations.foreach(_.close())
+    }
+    assert(smallUmm.executionMemoryUsed === 0)
+  }
+
+  test("SPARK-61082: UnsafeHashedRelation does not round its page up to the 
page size") {
+    // The single string column is also the key, so a row is its own key.
+    val buildKey = Seq(BoundReference(0, StringType, false))
+    val toUnsafeRow = UnsafeProjection.create(Array[DataType](StringType))
+    val row = toUnsafeRow(InternalRow(UTF8String.fromString("a"))).copy()
+    val relation = HashedRelation(Iterator(row), buildKey, taskMemoryManager = 
mm)
+    val serializers = Seq(
+      new JavaSerializer(new SparkConf()).newInstance(),
+      new KryoSerializer(new SparkConf().set(KRYO_REFERENCE_TRACKING, 
false)).newInstance())
+    // The map grows its pages up to the SparkEnv page size, not up to the 
initial page size.
+    val pageSize = SparkEnv.get.memoryManager.pageSizeBytes
+    try {
+      assert(relation.estimatedSize < 2 * 
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+      assert(relation.asInstanceOf[UnsafeHashedRelation].mapPageSizeBytes === 
pageSize)
+      serializers.foreach { ser =>
+        val deserialized = 
ser.deserialize[UnsafeHashedRelation](ser.serialize(relation))
+        try {
+          assert(deserialized.estimatedSize < 2 * 
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+          assert(deserialized.mapPageSizeBytes === pageSize)
+          assert(deserialized.getValue(row) === row)
+        } finally {
+          deserialized.close()
+        }
+      }
+    } finally {
+      relation.close()
+    }
+  }
+
   test("LongToUnsafeRowMap") {
     val unsafeProj = UnsafeProjection.create(
       Seq(BoundReference(0, LongType, false), BoundReference(1, IntegerType, 
true)))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to