Really excited to try out the new Docker executor support on 1.4.1, I'm making progress but feel like I'm missing something.
(versions: spark-1.4.1-hadoop2.6 - not using hadoop yet mac os x yosemite java 8 spark-shell mesos 0.22.1 : 2 slaves, 1 master + zk , all on centos 6.x docker 1.8.x ) I wanted to use a generic docker image with e.g. Java 8 in it, and then deploy the 1.4.1 distro into it. The docs seem to indicate that's supported. Mesos and Docker pull the requested docker images ( 'docker pull java' essentially) and extract the spark.executor.uri distro correctly. It seems to fall over at cd'ing of all places: "cd: can't cd to spark-1*" this causes the task to fail and eventually have spark blacklist the slave. Is this just because 'spark-1*' matches both the tarball and the directory? full mesos stderr looks like: -------------------------------------------- I0823 19:13:25.608206 3069 fetcher.cpp:214] Fetching URI 'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz' I0823 19:13:25.608582 3069 fetcher.cpp:125] Fetching URI 'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz' with os::net I0823 19:13:25.608620 3069 fetcher.cpp:135] Downloading 'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz' to '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429/spark-1.4.1-bin-hadoop2.6.tgz' I0823 19:14:37.765060 3069 fetcher.cpp:78] Extracted resource '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429/spark-1.4.1-bin-hadoop2.6.tgz' into '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429' /bin/sh: 1: cd: can't cd to spark-1* /bin/sh: 1: ./bin/spark-class: not found I0823 19:14:38.365190 3138 exec.cpp:132] Version: 0.22.1 I0823 19:14:38.369495 3156 exec.cpp:206] Executor registered on slave 20150823-110659-1862270986-5050-3230-S1 ----------------------------------------------------- and the spark-shell says: 15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is now TASK_RUNNING 15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is now TASK_FAILED 15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Blacklisting Mesos slave value: "20150823-110659-1862270986-5050-3230-S0" due to too many failures; is Spark installed on it? --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org