Hi Pradeep, You'll need to specify a s3:// url instead of specifying a relative path like you did, you can also also use http or hdfs if you want.
You also need to make sure s3 credentials are available in your hadoop configuration that's also embedded in your docker image, you should be able to find help information about that easily (We're still working around these user experience problems around configuration, for now it's easier if the docker image has all the right configuration). Tim On Tue, Mar 15, 2016 at 11:18 AM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Hello TIm, > > I was able to start the spark tasks also as docker containers. > > I have one question: > > Currently, when i am submit a sample job like follows: > > $ bin/spark-submit --deploy-mode cluster --master > mesos://spark-dispatcher.service.consul:7077 --class > org.apache.spark.examples.SparkPi lib/spark-examples*.jar 10 > > It tries to copy the spark-examples*.jar on host from absolute path to the > sandbox. Is there any way i can make it to pull from S3 instead of looking > on host for the jar ? So that my build pipeline will push the jar to S3 and > running spark-submit as a deployment job will pull that jar from S3. > > Thanks. > > > > On Tue, Mar 15, 2016 at 5:17 PM, Pradeep Chhetri < > pradeep.chhetr...@gmail.com> wrote: > >> TIm, sorry I am wrong above. >> >> The above config is from spark dispatcher container and those >> configuration is not being propagated to the driver. >> >> I will try the workaround you told and let you know how it goes. >> >> Thanks >> >> >> >> On Tue, Mar 15, 2016 at 4:42 PM, Pradeep Chhetri < >> pradeep.chhetr...@gmail.com> wrote: >> >>> Hello Tim, >>> >>> Here is my conf/spark-defaults.conf which is inside the docker image: >>> >>> $ cat conf/spark-defaults.conf >>> >>> spark.mesos.coarse: false >>> spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12 >>> spark.mesos.mesosExecutor.cores: 0.25 >>> spark.mesos.executor.home: /opt/spark >>> spark.mesos.uris: file:///etc/docker.tar.gz >>> >>> I am already setting it inside the docker image. >>> >>> Am I missing something ? >>> >>> Regards, >>> >>> On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <t...@mesosphere.io> wrote: >>> >>>> Hi Pradeep, >>>> >>>> Yes we still have a pending PR that will start propagating these >>>> settings down to the executors, right now it's only applied on the driver. >>>> As a work around you can download or set spark.mesos.executor.docker.image >>>> in the spark-default.conf file in the docker image you use to launch the >>>> driver and it should automatically get this setting when the driver is >>>> launched. >>>> >>>> Tim >>>> >>>> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri < >>>> pradeep.chhetr...@gmail.com> wrote: >>>> >>>>> Hello Timothy, >>>>> >>>>> I am setting spark.mesos.executor.docker.image. In my case, the >>>>> driver is actually started as a docker container (SparkPi in screenshot) >>>>> but the tasks which are spawned by driver are not starting as containers >>>>> but plain java processes. Is this expected ? >>>>> >>>>> Thanks >>>>> >>>>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <t...@mesosphere.io> >>>>> wrote: >>>>> >>>>>> You can launch the driver and executor in docker containers as well >>>>>> by setting spark.mesos.executor.docker.image to the image you want to use >>>>>> to launch them. >>>>>> >>>>>> Tim >>>>>> >>>>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski < >>>>>> ra...@gruchalski.com> wrote: >>>>>> >>>>>> Pradeep, >>>>>> >>>>>> You can mount a spark directory as a volume. This means you have to >>>>>> have spark deployed on every agent. >>>>>> >>>>>> Another thing you can do, place spark in hdfs, assuming that you have >>>>>> hdfs available but that too will download a copy to the sandbox. >>>>>> >>>>>> I'd prefer the former. >>>>>> >>>>>> Sent from Outlook Mobile <https://aka.ms/qtex0l> >>>>>> >>>>>> _____________________________ >>>>>> From: Pradeep Chhetri <pradeep.chhetr...@gmail.com> >>>>>> Sent: Tuesday, March 15, 2016 4:41 pm >>>>>> Subject: Apache Spark Over Mesos >>>>>> To: <user@mesos.apache.org> >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> I am able to run Apache Spark over Mesos. Its quite simple to run >>>>>> Spark Dispatcher over marathon and ask it to run Spark Executor (I guess >>>>>> also can be called as Spark Driver) as docker container. >>>>>> >>>>>> I have a query regarding this: >>>>>> >>>>>> All spark tasks are spawned directly by first downloading the spark >>>>>> artifacts. I was thinking if there is some way I can start them too as >>>>>> docker containers. This will save the time for downloading the spark >>>>>> artifacts. I am running spark in fine-grained mode. >>>>>> >>>>>> I have attached a screenshot of a sample job >>>>>> >>>>>> <Screen Shot 2016-03-15 at 15.15.06.png> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> Pradeep Chhetri >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pradeep Chhetri >>>>> >>>> >>>> >>> >>> >>> -- >>> Pradeep Chhetri >>> >> >> >> >> -- >> Pradeep Chhetri >> > > > > -- > Pradeep Chhetri >
