Josh Rosen created SPARK-11293:
----------------------------------
Summary: Spillable collections leak shuffle memory
Key: SPARK-11293
URL: https://issues.apache.org/jira/browse/SPARK-11293
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.5.1, 1.4.1, 1.3.1
Reporter: Josh Rosen
Assignee: Josh Rosen
Priority: Critical
I discovered multiple leaks of shuffle memory while working on my memory
manager consolidation patch, which added the ability to do strict memory leak
detection for the bookkeeping that used to be performed by the
ShuffleMemoryManager. This uncovered a handful of places where tasks can
acquire execution/shuffle memory but never release it, starving themselves of
memory.
Problems that I found:
* {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
memory.
* BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
{{CompletionIterator}}.
* {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing its
resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
