Thanks much for your help Tobias. I am trying what you have suggested but am receiving error messages which I think are generally related to incorrect spark configuration. Will try and fix those and let you know how it goes.
Cheers, -Rob On Wed, Jul 30, 2014 at 6:01 PM, Tobias Pfeiffer [via Apache Spark User List] <[email protected]> wrote: > Hi, > > if your Spark driver runs in Eclipse, you should be able to get what you > want. That is, create a main() function that creates a SparkContext with > the correct master URL of your cluster and execute it with your standard > Eclipse method. For example, I am using Mesos and the following works fine: > > val conf = new SparkConf() > .setMaster("zk://HOSTNAME:2181/mesos") > .setAppName("My Application") > .setJars("/path/to/my-application-assembly-1.0.jar" :: Nil) > .set("spark.executor.uri", > "/path/to/spark-1.0.0-bin-hadoop2.tgz") > .set("spark.mesos.coarse", "true") > val sc = new SparkContext(conf) > > However, if you change something in the cluster node processing code, you > will have to redeploy the my-application-assembly-1.0.jar file. > > Tobias > > > On Thu, Jul 31, 2014 at 9:20 AM, nunarob <[hidden email] > <http://user/SendEmail.jtp?type=node&node=11007&i=0>> wrote: > >> Hi all, >> >> I'm a new Spark user and I'm interested in deploying java-based Spark >> applications directly from Eclipse. Specifically, I'd like to write a >> Spark/java application, have it directly submitted to a spark cluster, and >> then see the output of the spark execution directly in the eclipse >> console. >> >> Following the quick start guide, I'm able to package up a spark >> application >> using maven from within eclipse and then able to manually submit the jar >> using the spark-submit script. Is there anyway to submit the packaged jar >> to >> the spark cluster directly from within eclipse, and to see the output in >> the >> eclipse console? >> >> Apologies if this is documented somewhere - still learning about the >> entire >> spark ecosystem. >> >> Thanks! >> >> -Rob >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11007.html > To unsubscribe from Deploying spark applications from within Eclipse?, click > here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11001&code=cm9iQG51bmFoZWFsdGguY29tfDExMDAxfC00ODcwNDM5MTI=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11190.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
