Apache Spark reassigned SPARK-23988:

    Assignee:     (was: Apache Spark)

> [Mesos] Improve handling of appResource in mesos dispatcher when using Docker
> -----------------------------------------------------------------------------
>                 Key: SPARK-23988
>                 URL: https://issues.apache.org/jira/browse/SPARK-23988
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 2.2.1, 2.3.0
>            Reporter: paul mackles
>            Priority: Minor
> Our organization makes heavy use of Docker containers when running Spark on 
> Mesos. The images we use for our containers include Spark along with all of 
> the application dependencies. We find this to be a great way to manage our 
> artifacts.
> When specifying the primary application jar (i.e. appResource), the mesos 
> dispatcher insists on adding it to the list of URIs for Mesos to fetch as 
> part of launching the driver's container. This leads to confusing behavior 
> where paths such as:
>  * file:///application.jar
>  * local:/application.jar
>  * /application.jar
> wind up being fetched from the host where the driver is running. Obviously, 
> this doesn't work since all of the above examples are referencing the path of 
> the jar on the container image itself.
> Here is an example that I used for testing:
> {code:java}
> spark-submit \
>   --class org.apache.spark.examples.SparkPi \
>   --master mesos://spark-dispatcher \
>   --deploy-mode cluster \
>   --conf spark.cores.max=4 \
>   --conf spark.mesos.executor.docker.image=spark:2.2.1 \
>   local:/usr/local/spark/examples/jars/spark-examples_2.11-2.2.1.jar 10{code}
> The "spark:2.2.1" image contains an installation of spark under 
> "/usr/local/spark". Notice how we reference the appResource using the 
> "local:/" scheme.
> If you try the above with the current version of the mesos dispatcher, it 
> will try to fetch the path 
> "/usr/local/spark/examples/jars/spark-examples_2.11-2.2.1.jar" from the host 
> filesystem where the driver's container is running. On our systems, this 
> fails since we don't have spark installed on the hosts. 
> For the PR, all I did was modify the mesos dispatcher to not add the 
> "appResource to the list of URIs for Mesos to fetch if it uses the "local:/" 
> scheme.
> For now, I didn't change the behavior of absolute paths or the "file:/" 
> scheme because I wanted to leave some form for the old behavior in place for 
> backwards compatibility. Anyone have any opinions on whether these schemes 
> should change as well?
> The PR also includes support for using "spark-internal" with Mesos in cluster 
> mode which is something we need for another use-case. I can separate them if 
> that makes more sense.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to