Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r191077231
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite extends
SparkFunSuite with LocalSparkContext {
sc.stop()
}
- test("external aggregation updates peak execution memory") {
+ test("SPARK-22713 spill during iteration leaks internal map") {
+ val size = 1000
+ val conf = createSparkConf(loadDefaults = true)
+ sc = new SparkContext("local-cluster[1,1,1024]", "test", conf)
+ val map = createExternalMap[Int]
--- End diff --
@cloud-fan , I've tried the `System.gc` approach with and without
`eventually`, and for no avail.
seems like scalatest's asserts use some black magic macro based voodoo to
generate a local method/val (when I got about 10 classes deep into my quest, I
gave up)
basically what I'm seeing in visualvm is the expected ref via the
`WeakReference` and another surprising one via `UnaryMacroBool.left`.
it seems this sneaky ref was generated by the following assertion:
`assert(map.currentMap == null)`, but it could have easily been generated
elsewhere.
@cloud-fan , @gatorsmile , can you please confirm if and how can we import
the scala code? otherwise, can you think of an alternative approach for testing
this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
