GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/5403
[SPARK-3376] Add in-memory shuffle option.
This commit adds a new ShuffleManager that stores all shuffle data
in-memory using the block manager.
With this change, in-memory shuffle data is cleaned up in the
same way as disk shuffle data: using the metadata cleaner. It
would probably be better to clean up in-memory shuffle data more
aggressively, in order to avoid running out of memory.
One idea proposed by @pwendell was to publish this as a Spark
package rather than adding it directly to Spark. The concern is that
this option may be confusing to naive users, because it can result in
Spark's memory filling up in unexpected ways. If it is decided that
this should actually be added to Spark, I can update the documentation
to show the in-memory shuffle as one of the possible shuffle managers.
cc @shivaram
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kayousterhout/spark-1 SPARK-3376
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5403.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5403
----
commit 293fd69965be4fd5003c19e9cf85cf0c84bf7d54
Author: Kay Ousterhout <[email protected]>
Date: 2014-10-15T00:45:55Z
[SPARK-3376] Add in-memory shuffle option.
With this change, in-memory shuffle data is cleaned up in the
same way as disk shuffle data: using the metadata cleaner. It
would probably be better to clean up in-memory shuffle data more
aggressively, in order to avoid running out of memory.
commit 57fb068ea28f9754dddd1961712ad04d9759b8a6
Author: Kay Ousterhout <[email protected]>
Date: 2015-04-07T21:14:30Z
Added missing newlines
commit 00bdcd6f1cd6035108ed4c27ac5242ed70a3a286
Author: Kay Ousterhout <[email protected]>
Date: 2015-04-07T21:17:10Z
Fixed header
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
