Repository: spark Updated Branches: refs/heads/master f88a67bf0 -> f28b1a4c4
[SPARK-22721] BytesToBytesMap peak memory not updated. ## What changes were proposed in this pull request? Follow-up to earlier commit. The peak memory of BytesToBytesMap is not updated in more places - spill() and destructiveIterator(). ## How was this patch tested? Manually. Author: Juliusz Sompolski <ju...@databricks.com> Closes #19923 from juliuszsompolski/SPARK-22721cd. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f28b1a4c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f28b1a4c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f28b1a4c Branch: refs/heads/master Commit: f28b1a4c41d41e069b882bac06a68b2c45eb156c Parents: f88a67b Author: Juliusz Sompolski <ju...@databricks.com> Authored: Fri Dec 8 12:19:45 2017 +0100 Committer: Herman van Hovell <hvanhov...@databricks.com> Committed: Fri Dec 8 12:19:45 2017 +0100 ---------------------------------------------------------------------- .../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/f28b1a4c/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java index 7fdcf22..5f00455 100644 --- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java +++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java @@ -347,6 +347,8 @@ public final class BytesToBytesMap extends MemoryConsumer { return 0L; } + updatePeakMemoryUsed(); + // TODO: use existing ShuffleWriteMetrics ShuffleWriteMetrics writeMetrics = new ShuffleWriteMetrics(); @@ -424,6 +426,7 @@ public final class BytesToBytesMap extends MemoryConsumer { * `lookup()`, the behavior of the returned iterator is undefined. */ public MapIterator destructiveIterator() { + updatePeakMemoryUsed(); return new MapIterator(numValues, loc, true); } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
