As of applying SPARK-15781 the documentation of SPARK_WORKER_INSTANCES have
been removed. This was due to a warning in spark-submit which suggested:
WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '4').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the spark
config.
The problem is that there is no replacement method to launch spark standalone
with multiple workers per node. The options -num-executors and
SPARK_EXECUTOR_INSTANCES configure the job rather than the resource manager
behavior.
If I look at the spark standalone scripts, the only way to set multiple workers
per node is the use of SPARK_WORKER_INSTANCES. The fixed in SPARK-15781 fixed
the documentation without solving the problem.
A possible simple solution would be to add a SPARK_STANDALONE_WORKERS variable
and add it to the start-slave.sh script and update the documentation
accordingly.
Am I missing something here? Should I open a new JIRA issue?
Thanks,
Assaf
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Using-SPARK-WORKER-INSTANCES-and-SPARK-15781-tp19571.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
