yangpengyu created SPARK-27417:
----------------------------------
Summary: CLONE - ExternalSorter and ExternalAppendOnlyMap should
free shuffle memory in their stop() methods
Key: SPARK-27417
URL: https://issues.apache.org/jira/browse/SPARK-27417
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
Reporter: yangpengyu
Assignee: Josh Rosen
Fix For: 1.6.0
I discovered multiple leaks of shuffle memory while working on my memory
manager consolidation patch, which added the ability to do strict memory leak
detection for the bookkeeping that used to be performed by the
ShuffleMemoryManager. This uncovered a handful of places where tasks can
acquire execution/shuffle memory but never release it, starving themselves of
memory.
Problems that I found:
* {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
memory.
* BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
{{CompletionIterator}}.
* {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing its
resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
