No you don't need to install spark on each slave, we have been running this setup in Mesosphere without any problem at this point, I think most likely configuration problem and perhaps a chance something is missing in the code to handle some cases.
What version of spark are you guys running? And also did you set the working dir in your image to be spark home? Tim > On Mar 10, 2016, at 3:11 AM, Ashish Soni <asoni.le...@gmail.com> wrote: > > You need to install spark on each mesos slave and then while starting > container make a workdir to your spark home so that it can find the spark > class. > > Ashish > >> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps >> <g.eynard.bonte...@gmail.com> wrote: >> >> For an answer to my question see this: >> http://stackoverflow.com/a/35660466?noredirect=1. >> >> But for your problem did you define the Spark.mesos.docker. home or >> something like that property? >> >> >> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana >> <eran.chinth...@gmail.com> a écrit : >>> Hi >>> >>> I'm also having this issue and can not get the tasks to work inside mesos. >>> >>> In my case, the spark-submit command is the following. >>> >>> $SPARK_HOME/bin/spark-submit \ >>> --class com.mycompany.SparkStarter \ >>> --master mesos://mesos-dispatcher:7077 \ >>> --name SparkStarterJob \ >>> --driver-memory 1G \ >>> --executor-memory 4G \ >>> --deploy-mode cluster \ >>> --total-executor-cores 1 \ >>> --conf >>> spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \ >>> http://abc.com/spark-starter.jar >>> >>> And the error I'm getting is the following. >>> >>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1 >>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave >>> 20160223-000314-3439362570-5050-631-S0 >>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found >>> >>> (Looked into Spark JIRA and I found that >>> https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed since >>> https://issues.apache.org/jira/browse/SPARK-12345 is marked as resolved) >>> >>> Really appreciate if I can get some help here. >>> >>> Thanks, >>> Eran Chinthaka Withana >>> >>>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bonte...@gmail.com >>>> <g.eynard.bonte...@gmail.com> wrote: >>>> Hi everybody, >>>> >>>> I am testing the use of Docker for executing Spark algorithms on MESOS. I >>>> managed to execute Spark in client mode with executors inside Docker, but I >>>> wanted to go further and have also my Driver running into a Docker >>>> Container. Here I ran into a behavior that I'm not sure is normal, let me >>>> try to explain. >>>> >>>> I submit my spark application through MesosClusterDispatcher using a >>>> command >>>> like: >>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf >>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2 >>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar >>>> 10 >>>> >>>> My driver is running fine, inside its docker container, but the executors >>>> fail: >>>> "sh: /some/spark/home/bin/spark-class: No such file or directory" >>>> >>>> Looking on MESOS slaves log, I think that the executors do not run inside >>>> docker: "docker.cpp:775] No container info found, skipping launch". As my >>>> Mesos slaves do not have spark installed, it fails. >>>> >>>> *It seems that the spark conf that I gave in the first spark-submit is not >>>> transmitted to the Driver submitted conf*, when launched in the docker >>>> container. The only workaround I found is to modify my Docker image in >>>> order >>>> to define inside its spark conf the spark.mesos.executor.docker.image >>>> property. This way, my executors get the conf well and are launched inside >>>> docker on Mesos. This seems a little complicated to me, and I feel the >>>> configuration passed to the early spark-submit should be transmitted to the >>>> Driver submit... >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org
