GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/1678
SPARK-2532: Minimal shuffle consolidation fixes
All changes from this PR are by @mridulm and are drawn from his work in
#1609. This patch is intended to fix all major issues related to shuffle file
consolidation that @mridulm found, while minimizing changes to the code, with
the hope that it may be more easily merged into 1.1.
This patch is **not** intended as a replacement for #1609, which provides
many additional benefits, including fixes to ExternalAppendOnlyMap,
improvements to DiskBlockObjectWriter's API, and several new unit tests.
If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable
option.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark consol
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1678.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1678
----
commit f298ac92f7df30f53636fca874c8bc6ed764af19
Author: Aaron Davidson <[email protected]>
Date: 2014-07-28T19:02:24Z
SPARK-2532: Minimal shuffle consolidation fixes
All changes from this PR are by @mridulm and are drawn from his work in
#1609.
This patch is intended to fix all major issues related to shuffle file
consolidation
that @mridulm found, while minimizing changes to the code, with the hope
that it may
be more easily merged into 1.1.
This patch is **not** intended as a replacement for #1609, which provides
many
additional benefits, including fixes to ExternalAppendOnlyMap, improvements
to
DiskBlockObjectWriter's API, and several new unit tests.
If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable
option.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
