Hi Raghav, I'm assuming you're using standalone mode. When using the Spark EC2 scripts you need to make sure that every machine has the most updated jars. Once you have built on one of the nodes, you must *rsync* the Spark directory to the rest of the nodes (see /root/spark-ec2/copy-dir).
That said, I usually build it locally on my laptop and *scp* the assembly jar to the cluster instead of building it there. The EC2 machines often take much longer to build for some reason. Also it's cumbersome to set up proper IDE there. -Andrew 2015-06-19 19:11 GMT-07:00 Raghav Shankar <raghav0110...@gmail.com>: > Thanks Andrew! Is this all I have to do when using the spark ec2 script to > setup a spark cluster? It seems to be getting an assembly jar that is not > from my project(perhaps from a maven repo). Is there a way to make the > ec2 script use the assembly jar that I created? > > Thanks, > Raghav > > > On Friday, June 19, 2015, Andrew Or <and...@databricks.com> wrote: > >> Hi Raghav, >> >> If you want to make changes to Spark and run your application with it, >> you may follow these steps. >> >> 1. git clone g...@github.com:apache/spark >> 2. cd spark; build/mvn clean package -DskipTests [...] >> 3. make local changes >> 4. build/mvn package -DskipTests [...] (no need to clean again here) >> 5. bin/spark-submit --master spark://[...] --class your.main.class >> your.jar >> >> No need to pass in extra --driver-java-options or >> --driver-extra-classpath as others have suggested. When using spark-submit, >> the main jar comes from assembly/target/scala_2.10, which is prepared >> through "mvn package". You just have to make sure that you re-package the >> assembly jar after each modification. >> >> -Andrew >> >> 2015-06-18 16:35 GMT-07:00 maxdml <max...@cs.duke.edu>: >> >>> You can specify the jars of your application to be included with >>> spark-submit >>> with the /--jars/ switch. >>> >>> Otherwise, are you sure that your newly compiled spark jar assembly is in >>> assembly/target/scala-2.10/? >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-Spark-Applications-using-Spark-Submit-tp23352p23400.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >>
