This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3dbf1dc18976 [SPARK-61082][SQL] Fix UnsafeHashedRelation build failure
when many small relations are alive in one task
3dbf1dc18976 is described below
commit 3dbf1dc189766a74b053d60cd340c2633074999a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 7 12:07:52 2026 -0800
[SPARK-61082][SQL] Fix UnsafeHashedRelation build failure when many small
relations are alive in one task
### What changes were proposed in this pull request?
This PR lets `BytesToBytesMap` start with a small data page and grow its
pages by doubling, and uses it in `UnsafeHashedRelation`, the hash relation for
join keys other than a single `LONG` (for example a `STRING` key or multiple
key columns).
- `BytesToBytesMap`: a new constructor takes `initialPageSizeBytes` after
`pageSizeBytes`. The first data page has that size (at most `pageSizeBytes`),
and each later page doubles until the pages add up to `pageSizeBytes`. The last
of them is cut to what is left, and all pages after them have `pageSizeBytes`.
A record larger than the next page still gets a page of its own size, and
`reset()` starts again from the initial page size. The existing constructors
pass `pageSizeBytes` as the [...]
- `MemoryConsumer`: a new `allocatePage(size, required)` requests `size`
bytes and, like `allocatePage(required)`, accepts a smaller page of at least
`required` bytes. `allocatePage(required)` now calls it with `max(pageSize,
required)`. `BytesToBytesMap` needs both values so that a partial memory grant
still works when the requested page size grows.
- `UnsafeHashedRelation`: both building (`apply`) and deserializing
(`read`) use a 1 MiB initial page,
`UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES`, the same size as
`LongToUnsafeRowMap.INITIAL_PAGE_SIZE_BYTES` from SPARK-60998.
For a large relation with a 64 MiB page size, the pages are 1, 2, 4, 8, 16,
32 and 1 MiB, then 64 MiB each. Because the growing pages end exactly at one
page size, they do not move the sizes at which a relation needs another full
page. The relation needs 7 more pages than before, which is small next to the
limit of 8192 pages per task.
The growth is opt-in because changing every `BytesToBytesMap` would also
change when hash aggregation falls back to sort-based aggregation. A smaller
fixed page size in `UnsafeHashedRelation` alone is not enough either: each
relation would still reserve a whole page, and a large relation would need many
more pages.
### Why are the changes needed?
`BytesToBytesMap` allocates its data pages through
`MemoryConsumer.allocatePage`, which rounds every request up to `max(pageSize,
required)`. `UnsafeHashedRelation` passes
`SparkEnv.get.memoryManager.pageSizeBytes`, which is `spark.buffer.pageSize` or
a value of up to 64 MiB derived from the executor memory and cores. So every
relation reserves a full page, even when it holds only a few rows.
- A shuffled hash join keeps its relation until the task ends, and the
relation cannot spill. A task with a chain of shuffled hash joins on such keys
runs out of execution memory long before the relations hold that much data:
```
[UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 2048 bytes of memory, got 0.
Memory used by task 0 grouped by consumer:
org.apache.spark.unsafe.map.BytesToBytesMap5aa9a000: 64.0 MiB
org.apache.spark.unsafe.map.BytesToBytesMap6bc9c3f9: 64.0 MiB
org.apache.spark.unsafe.map.BytesToBytesMap5240e37f: 64.0 MiB
org.apache.spark.unsafe.map.BytesToBytesMap31cf8e3c: 64.0 MiB
```
This is the output of the new test `SPARK-61082: many
UnsafeHashedRelations fit in one task when each holds little data` without this
fix: 4 relations of 10 rows each take the whole 256 MiB, and the 5th cannot
allocate its 2 KiB pointer array.
- A broadcast hash join builds the relation on the driver and deserializes
it on each executor with an on-heap memory manager that is not bounded by
execution memory. So every broadcast relation allocates a full page as a
`long[]` on the heap, 64 MiB even for a single row.
SPARK-60998 fixed the same problem for `LongToUnsafeRowMap`, which is used
for a single `LONG` key. For `UnsafeHashedRelation`, this is not a regression:
it has rounded its pages up to the page size since Spark 2.0, when it started
to build a `BytesToBytesMap` directly (SPARK-14052).
### Does this PR introduce _any_ user-facing change?
Yes. Queries that keep many `UnsafeHashedRelation`s in one task, such as a
chain of shuffled hash joins on non-`LONG` keys, no longer fail with
`UNABLE_TO_ACQUIRE_MEMORY` when the relations are small. Each relation reserves
memory close to what it stores, and a small broadcast relation uses much less
driver and executor heap. For small relations, the `data size of build side`
metric of `ShuffledHashJoin` and the `data size` metric of `BroadcastExchange`
are smaller accordingly. Hash a [...]
### How was this patch tested?
New tests:
- `HashedRelationSuite`, which runs in both `HashedRelationOnHeapSuite` and
`HashedRelationOffHeapSuite`:
- `SPARK-61082: many UnsafeHashedRelations fit in one task when each
holds little data`: 16 relations of 10 rows each on a `STRING` key, in 256 MiB
of execution memory. The test first checks that the page size from `SparkEnv`
is larger than 16 MiB, so that it covers the bug.
- `SPARK-61082: UnsafeHashedRelation does not round its page up to the
page size`: a relation with one row and its copies deserialized by the Java and
the Kryo serializer each reserve less than 2 MiB, instead of about 64 MiB, and
their maps grow their pages up to the `SparkEnv` page size.
- `AbstractBytesToBytesMapSuite`, which runs in both
`BytesToBytesMapOnHeapSuite` and `BytesToBytesMapOffHeapSuite`:
- `dataPagesGrowFromInitialPageSize`: with a 1 KiB initial page and a 6
KiB page size, the pages are 1, 2 and 3 KiB, then 6 KiB. After `reset()`, the
first page is 1 KiB again, and a record larger than the next page size (2 KiB)
gets a page of exactly its size. A map whose initial page size is larger than
its page size starts with a page of the page size.
- `initialCapacityBoundsChecking` also checks that the initial page size
must be positive.
Both `HashedRelationSuite` tests fail without the fix in both memory modes.
The existing suites for `BytesToBytesMap`, `TaskMemoryManager`, hash
aggregation and joins also pass.
```
build/sbt "core/testOnly
org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite
org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite
org.apache.spark.memory.TaskMemoryManagerSuite"
build/sbt "sql/testOnly
org.apache.spark.sql.execution.joins.HashedRelationOnHeapSuite
org.apache.spark.sql.execution.joins.HashedRelationOffHeapSuite
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMapSuite
org.apache.spark.sql.execution.UnsafeKVExternalSorterSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60196 from dongjoon-hyun/SPARK-61082.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 874fd21c4dfaac3ed24e2dae07fd14455a9a827a)
Signed-off-by: Dongjoon Hyun <[email protected]>
skip-checks: true
---
.../org/apache/spark/memory/MemoryConsumer.java | 13 +++-
.../apache/spark/unsafe/map/BytesToBytesMap.java | 83 ++++++++++++++++++++--
.../unsafe/map/AbstractBytesToBytesMapSuite.java | 55 ++++++++++++++
.../spark/sql/execution/joins/HashedRelation.scala | 21 +++++-
.../sql/execution/joins/HashedRelationSuite.scala | 67 +++++++++++++++++
5 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
index 3ec3c5cdf084..696d24a7b652 100644
--- a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
+++ b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
@@ -128,8 +128,17 @@ public abstract class MemoryConsumer {
* @throws SparkOutOfMemoryError
*/
protected MemoryBlock allocatePage(long required) {
- MemoryBlock page =
- taskMemoryManager.allocatePageWithMinimum(Math.max(pageSize, required),
required, this);
+ return allocatePage(Math.max(pageSize, required), required);
+ }
+
+ /**
+ * Allocate a memory block of `size` bytes. A smaller block may be returned,
but it has at least
+ * `required` bytes, which must not exceed `size`.
+ *
+ * @throws SparkOutOfMemoryError
+ */
+ protected MemoryBlock allocatePage(long size, long required) {
+ MemoryBlock page = taskMemoryManager.allocatePageWithMinimum(size,
required, this);
if (page == null || page.size() < required) {
throwOom(page, required);
}
diff --git
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index b9ef89292c2a..850ff34219c5 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -170,11 +170,30 @@ public final class BytesToBytesMap extends MemoryConsumer
{
@Nullable private final KeyOperationsFactory keyOperationsFactory;
/**
- * The size of the data pages that hold key and value data. Map entries
cannot span multiple
- * pages, so this limits the maximum entry size.
+ * The maximum size of the regular data pages that hold key and value data.
A map with a smaller
+ * initial page size grows its pages up to this size. Map entries cannot
span multiple pages, so
+ * a record larger than a page gets a page of its own size.
*/
private final long pageSizeBytes;
+ /**
+ * The size of the first data page, at most {@link #pageSizeBytes}.
+ */
+ private final long initialPageSizeBytes;
+
+ /**
+ * The size of the next data page while the map grows. It starts at
+ * {@link #initialPageSizeBytes} and doubles with every new page, up to
{@link #pageSizeBytes}.
+ */
+ private long nextPageSizeBytes;
+
+ /**
+ * The total size requested for the data pages so far. While it is below
+ * {@link #pageSizeBytes}, a new page is at most what is left to reach it,
so the growing pages
+ * add up to exactly one page size before the pages of {@link
#pageSizeBytes} begin.
+ */
+ private long requestedDataPagesBytes;
+
/**
* Number of keys defined in the map.
*/
@@ -240,6 +259,26 @@ public final class BytesToBytesMap extends MemoryConsumer {
double loadFactor,
long pageSizeBytes,
@Nullable KeyOperationsFactory keyOperationsFactory) {
+ this(
+ taskMemoryManager,
+ blockManager,
+ serializerManager,
+ initialCapacity,
+ loadFactor,
+ pageSizeBytes,
+ pageSizeBytes,
+ keyOperationsFactory);
+ }
+
+ private BytesToBytesMap(
+ TaskMemoryManager taskMemoryManager,
+ BlockManager blockManager,
+ SerializerManager serializerManager,
+ int initialCapacity,
+ double loadFactor,
+ long pageSizeBytes,
+ long initialPageSizeBytes,
+ @Nullable KeyOperationsFactory keyOperationsFactory) {
super(taskMemoryManager, pageSizeBytes,
taskMemoryManager.getTungstenMemoryMode());
this.taskMemoryManager = taskMemoryManager;
this.blockManager = blockManager;
@@ -259,10 +298,38 @@ public final class BytesToBytesMap extends MemoryConsumer
{
throw new IllegalArgumentException("Page size " + pageSizeBytes + "
cannot exceed " +
TaskMemoryManager.MAXIMUM_PAGE_SIZE_BYTES);
}
+ if (initialPageSizeBytes <= 0) {
+ throw new IllegalArgumentException("Initial page size must be greater
than 0");
+ }
+ this.initialPageSizeBytes = Math.min(initialPageSizeBytes, pageSizeBytes);
+ this.nextPageSizeBytes = this.initialPageSizeBytes;
this.initialCapacity = initialCapacity;
allocate(initialCapacity);
}
+ /**
+ * Creates a map whose first data page has `initialPageSizeBytes` (at most
`pageSizeBytes`), and
+ * whose later pages double in size until they add up to `pageSizeBytes`.
The last of them is cut
+ * to what is left, and all pages after them have `pageSizeBytes`. A map
holding little data then
+ * does not reserve a full page. A record larger than the next page gets a
page of its size.
+ */
+ public BytesToBytesMap(
+ TaskMemoryManager taskMemoryManager,
+ int initialCapacity,
+ long pageSizeBytes,
+ long initialPageSizeBytes) {
+ this(
+ taskMemoryManager,
+ SparkEnv.get() != null ? SparkEnv.get().blockManager() : null,
+ SparkEnv.get() != null ? SparkEnv.get().serializerManager() : null,
+ initialCapacity,
+ // In order to re-use the longArray for sorting, the load factor cannot
be larger than 0.5.
+ 0.5,
+ pageSizeBytes,
+ initialPageSizeBytes,
+ null);
+ }
+
public BytesToBytesMap(
TaskMemoryManager taskMemoryManager,
int initialCapacity,
@@ -1037,18 +1104,24 @@ public final class BytesToBytesMap extends
MemoryConsumer {
* @return whether there is enough space to allocate the new page.
*/
private boolean acquireNewPage(long required) {
+ final long remainingGrowthBytes = pageSizeBytes - requestedDataPagesBytes;
+ final long size = Math.max(
+ remainingGrowthBytes > 0 ? Math.min(nextPageSizeBytes,
remainingGrowthBytes) : pageSizeBytes,
+ required);
final MemoryBlock page;
try {
- page = allocatePage(required);
+ page = allocatePage(size, required);
} catch (SparkOutOfMemoryError e) {
return false;
}
// Retaining exact-fit minimum-retry pages would consume one page-table
slot per map entry.
- if (required < pageSizeBytes && page.size() == required &&
+ if (required < size && page.size() == required &&
isPageAllocationFromMinimumRetry(page)) {
freePage(page);
return false;
}
+ requestedDataPagesBytes += size;
+ nextPageSizeBytes = Math.min(pageSizeBytes, nextPageSizeBytes * 2);
currentPage = page;
dataPages.add(currentPage);
UnsafeAlignedOffset.putSize(currentPage.getBaseObject(),
currentPage.getBaseOffset(), 0);
@@ -1194,6 +1267,8 @@ public final class BytesToBytesMap extends MemoryConsumer
{
canGrowArray = true;
currentPage = null;
pageCursor = 0;
+ nextPageSizeBytes = initialPageSizeBytes;
+ requestedDataPagesBytes = 0;
if (longArray != null) {
freeArray(longArray);
}
diff --git
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
index 0fee9b91f4eb..6612e4cb6755 100644
---
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
+++
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
@@ -51,6 +51,7 @@ import org.apache.spark.serializer.SerializerInstance;
import org.apache.spark.serializer.SerializerManager;
import org.apache.spark.storage.*;
import org.apache.spark.unsafe.Platform;
+import org.apache.spark.unsafe.UnsafeAlignedOffset;
import org.apache.spark.unsafe.array.ByteArrayMethods;
import org.apache.spark.util.Utils;
import org.apache.spark.internal.config.package$;
@@ -772,6 +773,8 @@ public abstract class AbstractBytesToBytesMapSuite {
assertThrows(IllegalArgumentException.class,
() -> new BytesToBytesMap(taskMemoryManager, 1,
TaskMemoryManager.MAXIMUM_PAGE_SIZE_BYTES + 1));
+ assertThrows(IllegalArgumentException.class,
+ () -> new BytesToBytesMap(taskMemoryManager, 1, PAGE_SIZE_BYTES, 0));
}
@Test
@@ -817,6 +820,58 @@ public abstract class AbstractBytesToBytesMapSuite {
}
}
+ @Test
+ public void dataPagesGrowFromInitialPageSize() {
+ final BytesToBytesMap map = new BytesToBytesMap(taskMemoryManager, 1,
6144, 1024);
+ try {
+ // The pages double from 1 KiB, and the last growing page is cut to the
3 KiB left, so the
+ // growing pages add up to exactly one page size before the pages of 6
KiB begin.
+ assertEquals(1024, appendUntilNewDataPage(map, 1));
+ assertEquals(2048, appendUntilNewDataPage(map, 1));
+ assertEquals(3072, appendUntilNewDataPage(map, 1));
+ assertEquals(6144, appendUntilNewDataPage(map, 1));
+ assertEquals(6144, appendUntilNewDataPage(map, 1));
+
+ // reset() starts again from the initial page size.
+ map.reset();
+ assertEquals(1024, appendUntilNewDataPage(map, 1));
+ // A record larger than the next page size, 2 KiB, gets a page of its
own size: the record
+ // count of the page, the two lengths, the key, the value and the
pointer to the next value.
+ final int largeValueWords = 300;
+ assertEquals(3L * UnsafeAlignedOffset.getUaoSize() + 8 + largeValueWords
* 8L + 8,
+ appendUntilNewDataPage(map, largeValueWords));
+ } finally {
+ map.free();
+ }
+
+ // An initial page size larger than the page size is capped at the page
size.
+ final BytesToBytesMap cappedMap = new BytesToBytesMap(taskMemoryManager,
1, 1024, 4096);
+ try {
+ assertEquals(1024, appendUntilNewDataPage(cappedMap, 1));
+ } finally {
+ cappedMap.free();
+ }
+ }
+
+ /**
+ * Appends records with a value of `valueWords` longs until the map
allocates a new data page, and
+ * returns the size of that page. All records share one key, so the pointer
array does not grow
+ * and every increase of the memory consumption is a new data page.
+ */
+ private static long appendUntilNewDataPage(BytesToBytesMap map, int
valueWords) {
+ final long[] key = new long[]{0};
+ final long[] value = new long[valueWords];
+ final long previousMemory = map.getTotalMemoryConsumption();
+ while (true) {
+ assertTrue(map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8).append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET,
valueWords * 8));
+ final long memory = map.getTotalMemoryConsumption();
+ if (memory > previousMemory) {
+ return memory - previousMemory;
+ }
+ }
+ }
+
@Test
public void avoidDeadlock() throws InterruptedException {
memoryManager.limit(PAGE_SIZE_BYTES);
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
index 396ef237a05d..ae5e3cda25c8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
@@ -222,6 +222,9 @@ private[joins] class UnsafeHashedRelation(
override def estimatedSize: Long = binaryMap.getTotalMemoryConsumption
+ // Visible for testing: the page size that the map grows its data pages up
to.
+ private[joins] def mapPageSizeBytes: Long = binaryMap.getPageSizeBytes
+
// re-used in
get()/getValue()/getWithKeyIndex()/getValueWithKeyIndex()/valuesWithKeyIndex()
var resultRow = new UnsafeRow(numFields)
@@ -417,7 +420,8 @@ private[joins] class UnsafeHashedRelation(
binaryMap = new BytesToBytesMap(
taskMemoryManager,
(nKeys * 1.5 + 1).toInt, // reduce hash collision
- pageSizeBytes)
+ pageSizeBytes,
+ UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
var i = 0
var keyBuffer = new Array[Byte](1024)
@@ -452,6 +456,18 @@ private[joins] class UnsafeHashedRelation(
private[joins] object UnsafeHashedRelation {
+ /**
+ * The size of the first data page of the map. Later pages double in size
until they add up to
+ * the page size, and then all pages have the page size.
+ *
+ * Otherwise `BytesToBytesMap` rounds its first page up to the page size
(`spark.buffer.pageSize`
+ * or a default of up to 64 MiB), so even a few rows reserve a full page. A
task that keeps many
+ * relations alive at once, e.g. a stage with a chain of shuffled hash
joins, then runs out of
+ * execution memory long before the relations hold that much data. Each
broadcast relation would
+ * also allocate a full page on the heap of the driver and of every executor.
+ */
+ val INITIAL_PAGE_SIZE_BYTES: Long = 1L << 20
+
def apply(
input: Iterator[InternalRow],
key: Seq[Expression],
@@ -469,7 +485,8 @@ private[joins] object UnsafeHashedRelation {
taskMemoryManager,
// Only 70% of the slots can be used before growing, more capacity help
to reduce collision
(sizeEstimate * 1.5 + 1).toInt,
- pageSizeBytes)
+ pageSizeBytes,
+ INITIAL_PAGE_SIZE_BYTES)
// Create a mapping of buildKeys -> rows
val keyGenerator = UnsafeProjection.create(key)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
index 23c41f4ed77b..f81203ed17f9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala
@@ -24,6 +24,7 @@ import scala.collection.mutable.ArrayBuffer
import scala.util.Random
import org.apache.spark.SparkConf
+import org.apache.spark.SparkEnv
import org.apache.spark.SparkException
import org.apache.spark.internal.config.{BUFFER_PAGESIZE,
MEMORY_OFFHEAP_ENABLED, MEMORY_OFFHEAP_SIZE}
import org.apache.spark.internal.config.Kryo._
@@ -141,6 +142,72 @@ abstract class HashedRelationSuite extends
SharedSparkSession {
assert(java.util.Arrays.equals(os2.toByteArray, os.toByteArray))
}
+ test("SPARK-61082: many UnsafeHashedRelations fit in one task when each
holds little data") {
+ // UnsafeHashedRelation takes its page size from SparkEnv, not from this
memory manager. With a
+ // page larger than 16 MiB, 16 relations rounded up to a full page each
would need more than
+ // the 256 MiB available.
+ assert(SparkEnv.get.memoryManager.pageSizeBytes > ByteUnit.MiB.toBytes(16))
+ val smallUmm = newMemoryManager(ByteUnit.MiB.toBytes(64),
ByteUnit.MiB.toBytes(256))
+ val smallMm = new TaskMemoryManager(smallUmm, 0)
+ val buildKey = Seq(BoundReference(0, StringType, false))
+ val toUnsafeRow = UnsafeProjection.create(Array[DataType](StringType,
IntegerType))
+ val toUnsafeKey = UnsafeProjection.create(Array[DataType](StringType))
+ val relations = ArrayBuffer.empty[HashedRelation]
+ try {
+ (0 until 16).foreach { i =>
+ val rows = (0 until 10).map { k =>
+ toUnsafeRow(InternalRow(UTF8String.fromString(s"key$k"), k +
i)).copy()
+ }
+ val relation = HashedRelation(rows.iterator, buildKey,
taskMemoryManager = smallMm)
+ assert(relation.isInstanceOf[UnsafeHashedRelation])
+ relations += relation
+ }
+ relations.zipWithIndex.foreach { case (relation, i) =>
+ assert(relation.estimatedSize < 2 *
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+ (0 until 10).foreach { k =>
+ val key = toUnsafeKey(InternalRow(UTF8String.fromString(s"key$k")))
+ assert(relation.getValue(key).getInt(1) === k + i)
+ }
+ }
+ } catch {
+ // SparkOutOfMemoryError is fatal to ScalaTest and would abort the whole
suite.
+ case e: SparkOutOfMemoryError =>
+ fail("Could not build 16 small UnsafeHashedRelations in one task", e)
+ } finally {
+ relations.foreach(_.close())
+ }
+ assert(smallUmm.executionMemoryUsed === 0)
+ }
+
+ test("SPARK-61082: UnsafeHashedRelation does not round its page up to the
page size") {
+ // The single string column is also the key, so a row is its own key.
+ val buildKey = Seq(BoundReference(0, StringType, false))
+ val toUnsafeRow = UnsafeProjection.create(Array[DataType](StringType))
+ val row = toUnsafeRow(InternalRow(UTF8String.fromString("a"))).copy()
+ val relation = HashedRelation(Iterator(row), buildKey, taskMemoryManager =
mm)
+ val serializers = Seq(
+ new JavaSerializer(new SparkConf()).newInstance(),
+ new KryoSerializer(new SparkConf().set(KRYO_REFERENCE_TRACKING,
false)).newInstance())
+ // The map grows its pages up to the SparkEnv page size, not up to the
initial page size.
+ val pageSize = SparkEnv.get.memoryManager.pageSizeBytes
+ try {
+ assert(relation.estimatedSize < 2 *
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+ assert(relation.asInstanceOf[UnsafeHashedRelation].mapPageSizeBytes ===
pageSize)
+ serializers.foreach { ser =>
+ val deserialized =
ser.deserialize[UnsafeHashedRelation](ser.serialize(relation))
+ try {
+ assert(deserialized.estimatedSize < 2 *
UnsafeHashedRelation.INITIAL_PAGE_SIZE_BYTES)
+ assert(deserialized.mapPageSizeBytes === pageSize)
+ assert(deserialized.getValue(row) === row)
+ } finally {
+ deserialized.close()
+ }
+ }
+ } finally {
+ relation.close()
+ }
+ }
+
test("LongToUnsafeRowMap") {
val unsafeProj = UnsafeProjection.create(
Seq(BoundReference(0, LongType, false), BoundReference(1, IntegerType,
true)))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]