[
https://issues.apache.org/jira/browse/SPARK-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224690#comment-15224690
]
Michael Gummelt commented on SPARK-13258:
-----------------------------------------
[~jayv] Does your PR fix this problem?
> --conf properties not honored in Mesos cluster mode
> ---------------------------------------------------
>
> Key: SPARK-13258
> URL: https://issues.apache.org/jira/browse/SPARK-13258
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 1.6.0
> Reporter: Michael Gummelt
>
> Spark properties set on {{spark-submit}} via the deprecated
> {{SPARK_JAVA_OPTS}} are passed along to the driver, but those set via the
> preferred {{--conf}} are not.
> For example, this results in the URI being fetched in the executor:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md
> -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0"
> ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077
> --class org.apache.spark.examples.SparkPi
> http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> This does not:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0"
> ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077
> --conf
> spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md
> --class org.apache.spark.examples.SparkPi
> http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369
> In the above line of code, you can see that SPARK_JAVA_OPTS is passed along
> to the driver, so those properties take effect.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373
> Whereas in this line of code, you see that {{--conf}} variables are set on
> {{SPARK_EXECUTOR_OPTS}}, which AFAICT has absolutely no effect because this
> env var is being set on the driver, not the executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
