GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/1214
Remove use of spark.worker.instances
spark.worker.instances as added as part of this commit:
https://github.com/apache/spark/commit/1617816090e7b20124a512a43860a21232ebf511
My understanding is that SPARK_WORKER_INSTANCES is supported for backwards
compatibility,
but spark.worker.instances is never used (SparkSubmit.scala sets
spark.executor.instances) so should
not have been added.
@sryza @pwendell @tgravescs LMK if I'm understanding this correctly
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kayousterhout/spark-1 yarn_config
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1214.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1214
----
commit 3d7c49168c7b451b4226d7accb10ed4725281baf
Author: Kay Ousterhout <[email protected]>
Date: 2014-06-25T18:44:19Z
Remove use of spark.worker.instances
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
