Repository: spark Updated Branches: refs/heads/master 3980ebdf1 -> 0af9ea74a
[SPARK-5845][Shuffle] Time to cleanup spilled shuffle files not included in shuffle write time I've added a timer in the right place to fix this inaccuracy. Author: Ilya Ganelin <ilya.gane...@capitalone.com> Closes #4965 from ilganeli/SPARK-5845 and squashes the following commits: bfabf88 [Ilya Ganelin] Changed to using a foreach vs. getorelse 3e059b0 [Ilya Ganelin] Switched to using getorelse b946d08 [Ilya Ganelin] Fixed error with option 9434b50 [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-5845 db8647e [Ilya Ganelin] Added update for shuffleWriteTime around spilled file cleanup in ExternalSorter Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0af9ea74 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0af9ea74 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0af9ea74 Branch: refs/heads/master Commit: 0af9ea74a07ecdc08c43fa63cb9c9f0c57e3029b Parents: 3980ebd Author: Ilya Ganelin <ilya.gane...@capitalone.com> Authored: Fri Mar 13 13:21:04 2015 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Fri Mar 13 13:21:04 2015 +0000 ---------------------------------------------------------------------- .../scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/0af9ea74/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala index 27496c5..fa2e617 100644 --- a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala +++ b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala @@ -88,7 +88,10 @@ private[spark] class SortShuffleWriter[K, V, C]( } finally { // Clean up our sorter, which may have its own intermediate files if (sorter != null) { + val startTime = System.nanoTime() sorter.stop() + context.taskMetrics.shuffleWriteMetrics.foreach( + _.incShuffleWriteTime(System.nanoTime - startTime)) sorter = null } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
