@Marcelo is correct. Mesos does not have something similar. Only Yarn does due to the distributed cache thing. I have described most of the above in the the jira also there are some other options.
Best, Stavros On Fri, Oct 5, 2018 at 8:28 PM, Marcelo Vanzin <van...@cloudera.com.invalid> wrote: > On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse <rve...@dotnetrdf.org> wrote: > > Ideally this would all just be handled automatically for users in the > way that all other resource managers do > > I think you're giving other resource managers too much credit. In > cluster mode, only YARN really distributes local dependencies, because > YARN has that feature (its distributed cache) and Spark just uses it. > > Standalone doesn't do it (see SPARK-4160) and I don't remember seeing > anything similar on the Mesos side. > > There are things that could be done; e.g. if you have HDFS you could > do a restricted version of what YARN does (upload files to HDFS, and > change the "spark.jars" and "spark.files" URLs to point to HDFS > instead). Or you could turn the submission client into a file server > that the cluster-mode driver downloads files from - although that > requires connectivity from the driver back to the client. > > Neither is great, but better than not having that feature. > > Just to be clear: in client mode things work right? (Although I'm not > really familiar with how client mode works in k8s - never tried it.) > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
