Hi Robin and Sabarish I figure out what the problem
To submit my java app so that it runs in cluster mode (ie. I can close my laptop and go home) I need to do the following 1. make sure my jar file is available on all the slaves. Spark-submit will cause my driver to run on a slave, It will not automatically copy my jar file to slaves. I found placing the jar in hdfs the easiest way to handle this 2. I needed to pass the command argument deploy-mode cluster 3. Use specify the path the jar file as a url hdfs:// Here are two tricks to figure out the correct URL for master 1. If you know the name of your cluster you can find the public DNS name for your master. By default use port 7077 cd spark-1.5.1-bin-hadoop2.6/ec2 $ spark-ec2 get-master --region=us-west-1 streamingDC Searching for existing cluster streamingDC in region us-west-1... Found 1 master, 3 slaves. ec2-54-251-207-123.us-west-1.compute.amazonaws.com $ 2. If you know the public DNS name of the master go to http://mastername..compute.amazonaws.com/8080 . The Title should be the correct url (ie. Port 7077) On master /root/ephemeral-hdfs/bin/hadoop fs -mkdir /home/ec2-user/sparkExamples /root/ephemeral-hdfs/bin/hadoop fs -put sparkPi-1.0-SNAPSHOT.jar /home/ec2-user/sparkExamples /root/ephemeral-hdfs/bin/hadoop fs -ls /home/ec2-user/sparkExamples $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 --deploy-mode cluster hdfs:///home/ec2-user/sparkExamples/sparkPi-1.0-SNAPSHOT.jar 100 Running Spark using the REST application submission protocol. 15/10/29 16:39:08 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077. 15/10/29 16:39:09 WARN rest.RestSubmissionClient: Unable to connect to server spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077. Warning: Master endpoint spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 was not a REST server. Falling back to legacy submission gateway instead. [ec2-user@ip-172-31-29-60 ~]$ I really appreciate everyone¹s help Andy
