GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/9260
[SPARK-11293] Fix shuffle memory leaks in Spillable collections and
UnsafeShuffleWriter
This patch fixes multiple memory leaks in `Spillable` collections, as well
as a leak in `UnsafeShuffleWriter`. There were a small handful of places where
tasks would acquire memory from the `ShuffleMemoryManager` but would not
release it by the time the task had ended. The `UnsafeShuffleWriter` case was
harmless, since the leak could only occur at the very end of a task, but the
other two cases are somewhat serious:
- `ExternalSorter.stop()` did not release the sorter's memory. In addition,
`BlockStoreShuffleReader` never called `stop()` once the sorter's iterator was
fully-consumed. Put together, these bugs meant that a shuffle which performed a
reduce-side could starve downstream piplelined transformations of shuffle
memory.
- `ExternalAppendOnlyMap` exposes no equivalent of `stop()` and its
iterators do not automatically free its in-memory data upon completion. This
could cause aggregation operations to starve other operations of shuffle memory.
This patch adds a regression test and fixes all three leaks. I'd like to
backport this patch to Spark 1.5.x and possibly to other maintenance releases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-11293
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9260.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9260
----
commit 90eba9fe7c603c150c5cb5a6a178d881a2cb73bd
Author: Josh Rosen <[email protected]>
Date: 2015-10-24T00:42:26Z
Enable shuffle memory leak detection in tests.
commit 255248bc7eefdf9838cb63939dc7aaae786a0fd1
Author: Josh Rosen <[email protected]>
Date: 2015-10-24T00:45:11Z
Fix sorter-related leaks:
- Fix leak in ExternalSorter.stop().
- Use CompletionIterator in BlockStoreShuffleReader.
- Fix leak in UnsafeShuffleWriter (this one wouldn't affect users, since
the leak could only occur precisely before the task finished, but it broke the
new tests).
commit f77544542c6734e650435ca08b463a7253df4fb0
Author: Josh Rosen <[email protected]>
Date: 2015-10-24T01:33:27Z
Fix leaks in ExternalAppendOnlyMap.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
