I don't see why using SparkSubmit.scala as your entry point would be any different, because all that does is invoke the main class of Client.scala (e.g. for Yarn) after setting up all the class paths and configuration options. (Though I haven't tried this myself)
2014-07-09 9:40 GMT-07:00 Ron Gonzalez <zlgonza...@yahoo.com>: > I am able to use Client.scala or LauncherExecutor.scala as my programmatic > entry point for Yarn. > > Thanks, > Ron > > Sent from my iPad > > On Jul 9, 2014, at 7:14 AM, Jerry Lam <chiling...@gmail.com> wrote: > > +1 as well for being able to submit jobs programmatically without using > shell script. > > we also experience issues of submitting jobs programmatically without > using spark-submit. In fact, even in the Hadoop World, I rarely used > "hadoop jar" to submit jobs in shell. > > > > On Wed, Jul 9, 2014 at 9:47 AM, Robert James <srobertja...@gmail.com> > wrote: > >> +1 to be able to do anything via SparkConf/SparkContext. Our app >> worked fine in Spark 0.9, but, after several days of wrestling with >> uber jars and spark-submit, and so far failing to get Spark 1.0 >> working, we'd like to go back to doing it ourself with SparkConf. >> >> As the previous poster said, a few scripts should be able to give us >> the classpath and any other params we need, and be a lot more >> transparent and debuggable. >> >> On 7/9/14, Surendranauth Hiraman <suren.hira...@velos.io> wrote: >> > Are there any gaps beyond convenience and code/config separation in >> using >> > spark-submit versus SparkConf/SparkContext if you are willing to set >> your >> > own config? >> > >> > If there are any gaps, +1 on having parity within SparkConf/SparkContext >> > where possible. In my use case, we launch our jobs programmatically. In >> > theory, we could shell out to spark-submit but it's not the best option >> for >> > us. >> > >> > So far, we are only using Standalone Cluster mode, so I'm not >> knowledgeable >> > on the complexities of other modes, though. >> > >> > -Suren >> > >> > >> > >> > On Wed, Jul 9, 2014 at 8:20 AM, Koert Kuipers <ko...@tresata.com> >> wrote: >> > >> >> not sure I understand why unifying how you submit app for different >> >> platforms and dynamic configuration cannot be part of SparkConf and >> >> SparkContext? >> >> >> >> for classpath a simple script similar to "hadoop classpath" that shows >> >> what needs to be added should be sufficient. >> >> >> >> on spark standalone I can launch a program just fine with just >> SparkConf >> >> and SparkContext. not on yarn, so the spark-launch script must be >> doing a >> >> few things extra there I am missing... which makes things more >> difficult >> >> because I am not sure its realistic to expect every application that >> >> needs >> >> to run something on spark to be launched using spark-submit. >> >> On Jul 9, 2014 3:45 AM, "Patrick Wendell" <pwend...@gmail.com> wrote: >> >> >> >>> It fulfills a few different functions. The main one is giving users a >> >>> way to inject Spark as a runtime dependency separately from their >> >>> program and make sure they get exactly the right version of Spark. So >> >>> a user can bundle an application and then use spark-submit to send it >> >>> to different types of clusters (or using different versions of Spark). >> >>> >> >>> It also unifies the way you bundle and submit an app for Yarn, Mesos, >> >>> etc... this was something that became very fragmented over time before >> >>> this was added. >> >>> >> >>> Another feature is allowing users to set configuration values >> >>> dynamically rather than compile them inside of their program. That's >> >>> the one you mention here. You can choose to use this feature or not. >> >>> If you know your configs are not going to change, then you don't need >> >>> to set them with spark-submit. >> >>> >> >>> >> >>> On Wed, Jul 9, 2014 at 10:22 AM, Robert James <srobertja...@gmail.com >> > >> >>> wrote: >> >>> > What is the purpose of spark-submit? Does it do anything outside of >> >>> > the standard val conf = new SparkConf ... val sc = new SparkContext >> >>> > ... ? >> >>> >> >> >> > >> > >> > -- >> > >> > SUREN HIRAMAN, VP TECHNOLOGY >> > Velos >> > Accelerating Machine Learning >> > >> > 440 NINTH AVENUE, 11TH FLOOR >> > NEW YORK, NY 10001 >> > O: (917) 525-2466 ext. 105 >> > F: 646.349.4063 >> > E: suren.hiraman@v <suren.hira...@sociocast.com>elos.io >> > W: www.velos.io >> > >> > >
