Github user juliuszsompolski commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r141843193
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorterSuite.java
---
@@ -139,4 +139,44 @@ public int compare(
}
assertEquals(dataToSort.length, iterLength);
}
+
+ @Test
+ public void freeAfterOOM() {
+ final TestMemoryManager testMemoryManager = new TestMemoryManager(new
SparkConf().set("spark.memory.offHeap.enabled", "false"));
+ final TaskMemoryManager memoryManager = new TaskMemoryManager(
+ testMemoryManager, 0);
+ final TestMemoryConsumer consumer = new
TestMemoryConsumer(memoryManager);
+ final MemoryBlock dataPage = memoryManager.allocatePage(2048,
consumer);
+ final Object baseObject = dataPage.getBaseObject();
+ // Write the records into the data page:
+ long position = dataPage.getBaseOffset();
+
+ final HashPartitioner hashPartitioner = new HashPartitioner(4);
+ // Use integer comparison for comparing prefixes (which are partition
ids, in this case)
+ final PrefixComparator prefixComparator = PrefixComparators.LONG;
+ final RecordComparator recordComparator = new RecordComparator() {
+ @Override
+ public int compare(
+ Object leftBaseObject,
+ long leftBaseOffset,
+ Object rightBaseObject,
+ long rightBaseOffset) {
+ return 0;
+ }
+ };
+ UnsafeInMemorySorter sorter = new UnsafeInMemorySorter(consumer,
memoryManager,
+ recordComparator, prefixComparator, 100, shouldUseRadixSort());
+
+ testMemoryManager.markExecutionAsOutOfMemoryOnce();
+ try {
+ sorter.reset();
+ } catch( OutOfMemoryError oom ) {
+ //as expected
+ }
+ // this currently fails on NPE at
org.apache.spark.memory.MemoryConsumer.freeArray(MemoryConsumer.java:108)
--- End diff --
nit: tense: "this currently fails" -> "[SPARK-21907] this failed ..."
At the point when anyone reads it, it will hopefully not fail :-)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
