[
https://issues.apache.org/jira/browse/SPARK-23988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23988:
------------------------------------
Assignee: (was: Apache Spark)
> [Mesos] Improve handling of appResource in mesos dispatcher when using Docker
> -----------------------------------------------------------------------------
>
> Key: SPARK-23988
> URL: https://issues.apache.org/jira/browse/SPARK-23988
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 2.2.1, 2.3.0
> Reporter: paul mackles
> Priority: Minor
>
> Our organization makes heavy use of Docker containers when running Spark on
> Mesos. The images we use for our containers include Spark along with all of
> the application dependencies. We find this to be a great way to manage our
> artifacts.
> When specifying the primary application jar (i.e. appResource), the mesos
> dispatcher insists on adding it to the list of URIs for Mesos to fetch as
> part of launching the driver's container. This leads to confusing behavior
> where paths such as:
> * file:///application.jar
> * local:/application.jar
> * /application.jar
> wind up being fetched from the host where the driver is running. Obviously,
> this doesn't work since all of the above examples are referencing the path of
> the jar on the container image itself.
> Here is an example that I used for testing:
> {code:java}
> spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master mesos://spark-dispatcher \
> --deploy-mode cluster \
> --conf spark.cores.max=4 \
> --conf spark.mesos.executor.docker.image=spark:2.2.1 \
> local:/usr/local/spark/examples/jars/spark-examples_2.11-2.2.1.jar 10{code}
> The "spark:2.2.1" image contains an installation of spark under
> "/usr/local/spark". Notice how we reference the appResource using the
> "local:/" scheme.
> If you try the above with the current version of the mesos dispatcher, it
> will try to fetch the path
> "/usr/local/spark/examples/jars/spark-examples_2.11-2.2.1.jar" from the host
> filesystem where the driver's container is running. On our systems, this
> fails since we don't have spark installed on the hosts.
> For the PR, all I did was modify the mesos dispatcher to not add the
> "appResource to the list of URIs for Mesos to fetch if it uses the "local:/"
> scheme.
> For now, I didn't change the behavior of absolute paths or the "file:/"
> scheme because I wanted to leave some form for the old behavior in place for
> backwards compatibility. Anyone have any opinions on whether these schemes
> should change as well?
> The PR also includes support for using "spark-internal" with Mesos in cluster
> mode which is something we need for another use-case. I can separate them if
> that makes more sense.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
