Repository: spark Updated Branches: refs/heads/master 8ae004b46 -> d32337b1e
[SPARK-22721] BytesToBytesMap peak memory usage not accurate after reset() ## What changes were proposed in this pull request? BytesToBytesMap doesn't update peak memory usage before shrinking back to initial capacity in reset(), so after a disk spill one never knows what was the size of hash table was before spilling. ## How was this patch tested? Checked manually. Author: Juliusz Sompolski <ju...@databricks.com> Closes #19915 from juliuszsompolski/SPARK-22721. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d32337b1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d32337b1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d32337b1 Branch: refs/heads/master Commit: d32337b1effd5359e1ce7e46893767c908d4b16a Parents: 8ae004b Author: Juliusz Sompolski <ju...@databricks.com> Authored: Thu Dec 7 13:05:59 2017 +0100 Committer: Herman van Hovell <hvanhov...@databricks.com> Committed: Thu Dec 7 13:05:59 2017 +0100 ---------------------------------------------------------------------- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d32337b1/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java index 4fadfe3..7fdcf22 100644 --- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java +++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java @@ -879,6 +879,7 @@ public final class BytesToBytesMap extends MemoryConsumer { * Reset this map to initialized state. */ public void reset() { + updatePeakMemoryUsed(); numKeys = 0; numValues = 0; freeArray(longArray); --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
