Hi, I'm trying to run spark-shell and point it to Spark standalone cluster. I have Spark locally on a different directory than on cluster. Locally, I have it in "/home/grega/mab/analyzer/target/spark" and on the cluster I have it in "/opt/spark".
When I run the spark-shell script with: SPARK_HOME="/opt/spark" SPARK_PRINT_LAUNCH_COMMAND=1 SPARK_HADOOP_VERSION=1.1.0 SPARK_MEM="5g" MASTER="spark://..." spark-shell it fails with 13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT 13/11/25 10:46:24 INFO server.AbstractConnector: Started [email protected]:43729 13/11/25 10:46:24 INFO broadcast.HttpBroadcast: Broadcast server started at http://10.170.8.11:43729 13/11/25 10:46:24 INFO spark.SparkEnv: Registering MapOutputTracker 13/11/25 10:46:24 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-aa291810-da5e-4d09-8cef-d025975813d7 13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT 13/11/25 10:46:24 INFO server.AbstractConnector: Started [email protected]:44042 13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage/rdd,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/stage,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/pool,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/environment,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/executors,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null} 13/11/25 10:46:24 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/,null} 13/11/25 10:46:24 INFO server.AbstractConnector: Started [email protected]:4040 13/11/25 10:46:24 INFO ui.SparkUI: Started Spark Web UI at http://ip-10-170-8-11.ec2.internal:4040 13/11/25 10:46:24 INFO client.Client$ClientActor: Connecting to master spark://ec2-107-20-48-235.compute-1.amazonaws.com:7077 Spark context available as sc. 13/11/25 10:46:24 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20131125104624-0028 13/11/25 10:46:24 INFO client.Client$ClientActor: Executor added: app-20131125104624-0028/0 on worker-20131125094226-ip-10-235-17-154.ec2.internal-49711 (ip-10-235-17-154.ec2.internal:49711) with 8 cores 13/11/25 10:46:24 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20131125104624-0028/0 on hostPort ip-10-235-17-154.ec2.internal:49711 with 8 cores, 5.0 GB RAM 13/11/25 10:46:25 INFO client.Client$ClientActor: Executor updated: app-20131125104624-0028/0 is now RUNNING *13/11/25 10:46:25 INFO client.Client$ClientActor: Executor updated: app-20131125104624-0028/0 is now FAILED (class java.io.IOException: Cannot run program "/home/grega/mab/analyzer/target/spark/bin/compute-classpath.sh" (in directory "."): java.io.IOException: error=2, No such file or directory)* 13/11/25 10:46:25 INFO cluster.SparkDeploySchedulerBackend: Executor app-20131125104624-0028/0 removed: class java.io.IOException: Cannot run program "/home/grega/mab/analyzer/target/spark/bin/compute-classpath.sh" (in directory "."): java.io.IOException: error=2, No such file or directory .... 13/11/25 10:46:25 ERROR client.Client$ClientActor: Master removed our application: FAILED; stopping client 13/11/25 10:46:25 ERROR cluster.SparkDeploySchedulerBackend: Disconnected from Spark cluster! 13/11/25 10:46:25 ERROR cluster.ClusterScheduler: Exiting due to error from cluster scheduler: Disconnected from Spark cluster It seems to me that it tries to run compute-classpath.sh on a worker node from wrong Spark path, even though I am passing SPARK_HOME through environment variable on the driver node. Any help appreciated. Thanks, Grega -- [image: Inline image 1] *Grega Kešpret* Analytics engineer Celtra — Rich Media Mobile Advertising celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
<<celtra_logo.png>>
