JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core]
Prevent freed page in BytesToBytesMap free again
URL: https://github.com/apache/spark/pull/25953#discussion_r329322322
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
##########
@@ -382,4 +382,48 @@ class UnsafeFixedWidthAggregationMapSuite
}
}
}
+
+ testWithMemoryLeakDetection("SPARK-29244: freed page should not be freed
again") {
+ memoryManager.limit(8000)
+
+ val pageSize = 4096
+ val map = new UnsafeFixedWidthAggregationMap(
+ emptyAggregationBuffer,
+ aggBufferSchema,
+ groupKeySchema,
+ TaskContext.get(), // We want to run completion listener.
+ 256, // initial capacity
+ pageSize
+ )
+
+ val rand = new Random(42)
+ val str = rand.nextString(1024)
+ val buf = map.getAggregationBuffer(InternalRow(UTF8String.fromString(str)))
+ buf.setInt(0, str.length)
+
+ var stopThread = false
+ val thread = new Thread {
+ override def run {
+ val testConsumer = new TestMemoryConsumer(taskMemoryManager)
+ while (!stopThread) {
+ try {
+ testConsumer.allocateArray(4000)
+ } catch {
+ case _: SparkOutOfMemoryError =>
+ }
+ }
+ }
+ }
+ thread.start()
+
+ try {
+ map.destructAndCreateExternalSorter()
+ } catch {
+ // In some chance, the `BytesToBytesMap` inside
`UnsafeFixedWidthAggregationMap` cannot
+ // allocate enough memory when calling `reset`. It will throw OOM
exception.
+ case s: SparkOutOfMemoryError =>
+ TaskContext.get().markTaskCompleted(Some(s))
+ }
+ stopThread = true
Review comment:
Should this be in a `finally` block to try to ensure that we always clean up
the background thread at the end of the test (and to clarify the intent of this
line)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
