GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/3147
[SPARK-4281] Package Yarn shuffle service into its own jar
This is another addendum to #3082, which added the Yarn shuffle service to
run inside the NM. This PR makes the feature much more usable by packaging
enough dependencies into the jar to run the service inside an NM. After these
changes, the user can run `./make-distribution.sh` and find a
`spark-network-yarn*.jar` in their `lib` directory. The equivalent change is
done in SBT by making the `network-yarn` module an assembly project.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark yarn-shuffle-build
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3147
----
commit c6530280859ef8ec8419c89fca94dd9e9d866506
Author: Andrew Or <[email protected]>
Date: 2014-11-07T05:04:11Z
Package network-yarn and its dependencies
This allows make-distribution to create a small uber jar for the
network-yarn module, such that all uses of the Yarn shuffle service
can just drop this jar onto the NM classpath and start the shuffle
service after configuring the NM to include it.
commit abcefd168baa2b5d73ca6c637ff0168338efc191
Author: Andrew Or <[email protected]>
Date: 2014-11-07T05:07:18Z
Do the same for SBT
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
