GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/1707
SPARK-2711. Create a ShuffleMemoryManager to track memory for all spilling
collections
This tracks memory properly if there are multiple spilling collections in
the same task (which was a problem before), and also implements an algorithm
that lets each thread grow up to 1 / 2N of the memory pool (where N is the
number of threads) before spilling, which avoids an inefficiency with small
spills we had before (some threads would spill many times at 0-1 MB because the
pool was allocated elsewhere).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark spark-2711
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1707.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1707
----
commit e40de34ce90aba130a268c7c2665ede1322414f4
Author: Matei Zaharia <[email protected]>
Date: 2014-08-01T02:49:40Z
Create central manager to track memory for all spilling collections
This tracks memory properly if there are multiple spilling collections
in the same task (which was a problem before), and also implements an
algorithm that lets each thread grow up to 1 / 2N of the memory pool
(where N is the number of threads) before spilling, which avoids an
inefficiency with small spills we had before (some threads would spill
many times at 0-1 MB because the pool was allocated elsewhere).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
