Andrew, thank you so much! That worked! I had to manually set the
spark.home configuration in the SparkConf object using
.set("spark.home","/cluster/path/to/spark/"), and then I was able to submit
from my laptop to the cluster!Aris On Thu, Jul 10, 2014 at 11:41 AM, Andrew Or <and...@databricks.com> wrote: > Setting SPARK_HOME is not super effective, because it is overridden very > quickly by bin/spark-submit here > <https://github.com/apache/spark/blob/88006a62377d2b7c9886ba49ceef158737bc1b97/bin/spark-submit#L20>. > Instead you should set the config "spark.home". Here's why: > > Each of your executors inherits its spark home from the application > description, and this is created by your SparkContext on your local > machine. By default, as you noticed, this uses your local spark home that > is not applicable to your remote executors. There are two ways of > controlling the spark home set in your application description: through the > "spark.home" config or "SPARK_HOME" environment variable, with the former > taking priority over the latter > <https://github.com/apache/spark/blob/88006a62377d2b7c9886ba49ceef158737bc1b97/core/src/main/scala/org/apache/spark/SparkContext.scala#L1017>. > However, since spark-submit overwrites whatever value you set SPARK_HOME > to, there is really only one way: by setting "spark.home". Note that this > config is only used to launch the executors, meaning the driver has already > started by the time this config is consumed, so this does not create any > ordering issue. > > Does that make sense? > > Andrew > > > 2014-07-10 10:17 GMT-07:00 Aris Vlasakakis <a...@vlasakakis.com>: > > Thank you very much Yana for replying! >> >> So right now the set up is a single-node machine which is my "cluster", >> and YES you are right my submitting laptop has a different path to the >> spark-1.0.0 installation than the "cluster" machine. >> >> I tried to set SPARK_HOME on my submittor laptop using the actual path of >> the CLUSTER's directories...hoping this would help. It didn't change >> anything. Let me show you the command from the submission side on my laptop >> - keep in mind I set SPARK_HOME as the path for the cluster. >> >> SPARK_HOME=/home/data/Documents/spark-1.0.0/ >> ~/Documents/spark-1.0.0/bin/spark-submit --verbose --class >> org.apache.spark.examples.SparkPi --master spark://10.20.10.152:7077 >> ~/Documents/spark-1.0.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >> >> I get the same problems as originally. I am actually confused here - if I >> set the SPARK_HOME on the same line as the spark-submit command, I don't >> get how the submittor script did not get all confused just running >> bin/spark-submit...but what do I know. >> >> Am I setting SPARK_HOME correctly? >> >> >> On Wed, Jul 9, 2014 at 7:09 PM, Yana Kadiyska <yana.kadiy...@gmail.com> >> wrote: >> >>> class java.io.IOException: Cannot run program >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" >>> (in directory "."): error=2, No such file or directory >>> >>> By any chance, are your SPARK_HOME directories different on the >>> machine where you're submitting from and the cluster? I'm on an older >>> drop so not sure about the finer points of spark-submit but do >>> remember a very similar issue when trying to run a Spark driver on a >>> windows machine against a Spark Master on Ubuntu cluster (the >>> SPARK_HOME directories were obviously different) >>> >>> On Wed, Jul 9, 2014 at 7:18 PM, Aris Vlasakakis <a...@vlasakakis.com> >>> wrote: >>> > Hello everybody, >>> > >>> > I am trying to figure out how to submit a Spark application from one >>> > separate physical machine to a Spark stand alone cluster. I have an >>> > application that I wrote in Python that works if I am on the 1-Node >>> Spark >>> > server itself, and from that spark installation I run bin/spark-submit >>> with >>> > 1) MASTER=local[*] or if 2) MASTER=spark://localhost:7077. >>> > >>> > However, I want to be on a separate machine that submits a job to >>> Spark. Am >>> > I doing something wrong here? I think something is wrong because I am >>> > working from two different spark "installations" -- as in, on the big >>> server >>> > I have one spark installation and I am running sbin/start-all.sh to >>> run the >>> > standalone server (and that works), and then on a separate laptop I >>> have a >>> > different installation of spark-1.0.0, but I am using the laptop's >>> > bin/spark-submit script to submit to the remote Spark server (using >>> > MASTER=spark://<remote-spark-master>:7077 >>> > >>> > This "submit-to-remote cluster" does not work, even for the Scala >>> examples >>> > like SparkPi. >>> > >>> > Concrete Example: I want to do submit the example SparkPi to the >>> cluster, >>> > from my laptop. >>> > >>> > Server is 10.20.10.152, running master and slave, I can look at the >>> Master >>> > web UI at http://10.20.10.152:8080. Great. >>> > >>> > From laptop (10.20.10.154), I try the following, using bin/run-example >>> from >>> > a locally built version of spark 1.0.0 (so that I have the script >>> > spark-submit!): >>> > >>> > bin/spark-submit --verbose --class org.apache.spark.examples.SparkPi >>> > --master spark://10.20.10.152:7077 >>> > examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > >>> > >>> > This fails, with the errors at the bottom of this email. >>> > >>> > Am I doing something wrong? How can I submit to a remote cluster? I >>> get the >>> > same problem with bin/spark-submit. >>> > >>> > >>> > bin/spark-submit --verbose --class org.apache.spark.examples.SparkPi >>> > --master spark://10.20.10.152:7077 >>> > examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > Using properties file: null >>> > Using properties file: null >>> > Parsed arguments: >>> > master spark://10.20.10.152:7077 >>> > deployMode null >>> > executorMemory null >>> > executorCores null >>> > totalExecutorCores null >>> > propertiesFile null >>> > driverMemory null >>> > driverCores null >>> > driverExtraClassPath null >>> > driverExtraLibraryPath null >>> > driverExtraJavaOptions null >>> > supervise false >>> > queue null >>> > numExecutors null >>> > files null >>> > pyFiles null >>> > archives null >>> > mainClass org.apache.spark.examples.SparkPi >>> > primaryResource >>> > >>> file:/Users/aris.vlasakakis/Documents/spark-1.0.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > name org.apache.spark.examples.SparkPi >>> > childArgs [] >>> > jars null >>> > verbose true >>> > >>> > Default properties from null: >>> > >>> > >>> > >>> > Using properties file: null >>> > Main class: >>> > org.apache.spark.examples.SparkPi >>> > Arguments: >>> > >>> > System properties: >>> > SPARK_SUBMIT -> true >>> > spark.app.name -> org.apache.spark.examples.SparkPi >>> > spark.jars -> >>> > >>> file:/Users/aris.vlasakakis/Documents/spark-1.0.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > spark.master -> spark://10.20.10.152:7077 >>> > Classpath elements: >>> > >>> file:/Users/aris.vlasakakis/Documents/spark-1.0.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > >>> > >>> > 14/07/09 16:16:08 INFO SecurityManager: Using Spark's default log4j >>> profile: >>> > org/apache/spark/log4j-defaults.properties >>> > 14/07/09 16:16:08 INFO SecurityManager: Changing view acls to: >>> > aris.vlasakakis >>> > 14/07/09 16:16:08 INFO SecurityManager: SecurityManager: authentication >>> > disabled; ui acls disabled; users with view permissions: >>> > Set(aris.vlasakakis) >>> > 14/07/09 16:16:08 INFO Slf4jLogger: Slf4jLogger started >>> > 14/07/09 16:16:08 INFO Remoting: Starting remoting >>> > 14/07/09 16:16:08 INFO Remoting: Remoting started; listening on >>> addresses >>> > :[akka.tcp://spark@10.20.10.154:50478] >>> > 14/07/09 16:16:08 INFO Remoting: Remoting now listens on addresses: >>> > [akka.tcp://spark@10.20.10.154:50478] >>> > 14/07/09 16:16:08 INFO SparkEnv: Registering MapOutputTracker >>> > 14/07/09 16:16:08 INFO SparkEnv: Registering BlockManagerMaster >>> > 14/07/09 16:16:08 INFO DiskBlockManager: Created local directory at >>> > >>> /var/folders/ch/yfyhs7px5h90505g4n21n8d5k3svt3/T/spark-local-20140709161608-0531 >>> > 14/07/09 16:16:08 INFO MemoryStore: MemoryStore started with capacity >>> 5.8 >>> > GB. >>> > 14/07/09 16:16:08 INFO ConnectionManager: Bound socket to port 50479 >>> with id >>> > = ConnectionManagerId(10.20.10.154,50479) >>> > 14/07/09 16:16:08 INFO BlockManagerMaster: Trying to register >>> BlockManager >>> > 14/07/09 16:16:08 INFO BlockManagerInfo: Registering block manager >>> > 10.20.10.154:50479 with 5.8 GB RAM >>> > 14/07/09 16:16:08 INFO BlockManagerMaster: Registered BlockManager >>> > 14/07/09 16:16:08 INFO HttpServer: Starting HTTP Server >>> > 14/07/09 16:16:09 INFO HttpBroadcast: Broadcast server started at >>> > http://10.20.10.154:50480 >>> > 14/07/09 16:16:09 INFO HttpFileServer: HTTP File server directory is >>> > >>> /var/folders/ch/yfyhs7px5h90505g4n21n8d5k3svt3/T/spark-edd787f4-f606-473c-965b-9f3b131cfb43 >>> > 14/07/09 16:16:09 INFO HttpServer: Starting HTTP Server >>> > 14/07/09 16:16:09 INFO SparkUI: Started SparkUI at >>> http://10.20.10.154:4040 >>> > 2014-07-09 16:16:09.439 java[37388:d17] Unable to load realm mapping >>> info >>> > from SCDynamicStore >>> > 14/07/09 16:16:09 INFO SparkContext: Added JAR >>> > >>> file:/Users/aris.vlasakakis/Documents/spark-1.0.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop1.0.4.jar >>> > at http://10.20.10.154:50481/jars/spark-examples-1.0.0-hadoop1.0.4.jar >>> with >>> > timestamp 1404947769853 >>> > 14/07/09 16:16:09 INFO AppClient$ClientActor: Connecting to master >>> > spark://10.20.10.152:7077... >>> > 14/07/09 16:16:09 INFO SparkContext: Starting job: reduce at >>> > SparkPi.scala:35 >>> > 14/07/09 16:16:09 INFO DAGScheduler: Got job 0 (reduce at >>> SparkPi.scala:35) >>> > with 2 output partitions (allowLocal=false) >>> > 14/07/09 16:16:09 INFO DAGScheduler: Final stage: Stage 0(reduce at >>> > SparkPi.scala:35) >>> > 14/07/09 16:16:10 INFO DAGScheduler: Parents of final stage: List() >>> > 14/07/09 16:16:10 INFO DAGScheduler: Missing parents: List() >>> > 14/07/09 16:16:10 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] >>> at map >>> > at SparkPi.scala:31), which has no missing parents >>> > 14/07/09 16:16:10 INFO DAGScheduler: Submitting 2 missing tasks from >>> Stage 0 >>> > (MappedRDD[1] at map at SparkPi.scala:31) >>> > 14/07/09 16:16:10 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 >>> tasks >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Connected to Spark >>> > cluster with app ID app-20140709161541-0001 >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/0 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/0 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/0 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/0 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/0 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/1 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/1 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/1 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/1 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/1 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/2 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/2 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/2 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/2 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/2 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/3 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/3 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/3 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/3 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/3 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/4 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/4 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/4 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/4 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/4 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/5 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/5 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/5 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/5 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/5 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/6 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/6 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/6 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/6 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/6 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/7 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/7 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/7 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/7 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/7 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/8 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/8 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/8 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/8 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/8 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor added: >>> > app-20140709161541-0001/9 on worker-20140709160420-10.20.10.152-57674 >>> > (10.20.10.152:57674) with 4 cores >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Granted executor ID >>> > app-20140709161541-0001/9 on hostPort 10.20.10.152:57674 with 4 >>> cores, 4.0 >>> > GB RAM >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/9 is now RUNNING >>> > 14/07/09 16:16:10 INFO AppClient$ClientActor: Executor updated: >>> > app-20140709161541-0001/9 is now FAILED (class java.io.IOException: >>> Cannot >>> > run program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory) >>> > 14/07/09 16:16:10 INFO SparkDeploySchedulerBackend: Executor >>> > app-20140709161541-0001/9 removed: class java.io.IOException: Cannot >>> run >>> > program >>> > >>> "/Users/aris.vlasakakis/Documents/spark-1.0.0/bin/compute-classpath.sh" (in >>> > directory "."): error=2, No such file or directory >>> > 14/07/09 16:16:10 ERROR SparkDeploySchedulerBackend: Application has >>> been >>> > killed. Reason: Master removed our application: FAILED >>> > 14/07/09 16:16:10 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose >>> tasks >>> > have all completed, from pool >>> > 14/07/09 16:16:10 INFO DAGScheduler: Failed to run reduce at >>> > SparkPi.scala:35 >>> > Exception in thread "main" 14/07/09 16:16:10 INFO TaskSchedulerImpl: >>> > Cancelling stage 0 >>> > org.apache.spark.SparkException: Job aborted due to stage failure: >>> Master >>> > removed our application: FAILED >>> > at >>> > org.apache.spark.scheduler.DAGScheduler.org >>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015) >>> > at >>> > >>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>> > at >>> > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633) >>> > at scala.Option.foreach(Option.scala:236) >>> > at >>> > >>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633) >>> > at >>> > >>> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207) >>> > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) >>> > at akka.actor.ActorCell.invoke(ActorCell.scala:456) >>> > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) >>> > at akka.dispatch.Mailbox.run(Mailbox.scala:219) >>> > at >>> > >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) >>> > at >>> > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> > at >>> > >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> > at >>> > >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> > at >>> > >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> > -- >>> > Άρης Βλασακάκης >>> > Aris Vlasakakis >>> >> >> >> >> -- >> Άρης Βλασακάκης >> Aris Vlasakakis >> > > -- Άρης Βλασακάκης Aris Vlasakakis
