GitHub user xiaochang-wu opened a pull request:
https://github.com/apache/spark/pull/16411
[SPARK-17984][YARN][Mesos][Deploy][WIP] add executor launch prefix support
## What changes were proposed in this pull request?
Complete Standalone, Yarn, Mesos support from #15579 and adapt to latest
master branch.
## How was this patch tested?
Standalone and Yarn mode tested on spark cluster with 1 master + 2 slaves,
Mesos mode is not tested due to lack of resources.
## How to use
1) Global Environment Variable
export SPARK_EXECUTOR_LAUNCH_PREFIX="/tmp/spark-numa-example.sh"
2) Config Files
- Standalone mode: add
SPARK_EXECUTOR_LAUNCH_PREFIX="/tmp/spark-numa-example.sh" in conf/spark-env.sh
- Yarn client mode: add
spark.yarn.appMasterEnv.SPARK_EXECUTOR_LAUNCH_PREFIX="/tmp/spark-numa-example.sh"
in conf/spark-defaults.conf
Attach the example script for adding executor launch prefix to enable
NUMA-aware binding for executors. Same apply to adding other launch prefix such
as strace, vtune etc..
[spark-numa-example.zip](https://github.com/apache/spark/files/673415/spark-numa-example.zip)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xiaochang-wu/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16411.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16411
----
commit 62c8552c778637ef4b074b9f27e9a5de0669c259
Author: xiaochang-wu <[email protected]>
Date: 2016-12-22T07:31:40Z
add executor launch prefix support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
