Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn client mode (1.6.0)
This is likely the cause as I replaced cdh-manager deployment (1.3.0 spark) with a compiled 1.6.0. On 1/13/16, 10:06 AM, "Marcelo Vanzin" wrote: >SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it >seems you have a mixbag of 1.3 / 1.6 in your path / classpath and >things are failing because of that. > >On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao wrote: >> My job runs fine in yarn cluster mode but I have reason to use client >>mode >> instead. But I'm hitting this error when submitting: >>> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest >>> --master yarn --deploy-mode client --executor-memory 90G >>>--num-executors 3 >>> --executor-cores 14 Martini-assembly-0.1.jar yarn-client >> >> Error: Could not find or load main class >> org.apache.spark.deploy.SparkSubmitDriverBootstrapper >> >> >> If I replace deploy-mode to cluster the job is submitted successfully. >>Is >> there a dependency missing from my project? Right now only one I >>included is >> spark-streaming 1.6.0. > > > >-- >Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn client mode (1.6.0)
Lin - if you add "--verbose" to your original *spark-submit* command, it will let you know the location in which Spark is running. As Marcelo pointed out, this will likely indicate version 1.3, which may help you confirm if this is your problem. On Wed, Jan 13, 2016 at 12:06 PM, Marcelo Vanzin wrote: > SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it > seems you have a mixbag of 1.3 / 1.6 in your path / classpath and > things are failing because of that. > > On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao wrote: > > My job runs fine in yarn cluster mode but I have reason to use client > mode > > instead. But I'm hitting this error when submitting: > >> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest > >> --master yarn --deploy-mode client --executor-memory 90G > --num-executors 3 > >> --executor-cores 14 Martini-assembly-0.1.jar yarn-client > > > > Error: Could not find or load main class > > org.apache.spark.deploy.SparkSubmitDriverBootstrapper > > > > > > If I replace deploy-mode to cluster the job is submitted successfully. > Is > > there a dependency missing from my project? Right now only one I > included is > > spark-streaming 1.6.0. > > > > -- > Marcelo > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn client mode (1.6.0)
SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it seems you have a mixbag of 1.3 / 1.6 in your path / classpath and things are failing because of that. On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao wrote: > My job runs fine in yarn cluster mode but I have reason to use client mode > instead. But I'm hitting this error when submitting: >> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest >> --master yarn --deploy-mode client --executor-memory 90G --num-executors 3 >> --executor-cores 14 Martini-assembly-0.1.jar yarn-client > > Error: Could not find or load main class > org.apache.spark.deploy.SparkSubmitDriverBootstrapper > > > If I replace deploy-mode to cluster the job is submitted successfully. Is > there a dependency missing from my project? Right now only one I included is > spark-streaming 1.6.0. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn client mode (1.6.0)
I also didn't find SparkSubmitDriverBootstrapper, which version of spark are you using ? On Wed, Jan 13, 2016 at 9:36 AM, Ted Yu wrote: > Can you show the complete stack trace for the error ? > > I searched 1.6.0 code base but didn't find the > class SparkSubmitDriverBootstrapper > > Thanks > > On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao wrote: > >> My job runs fine in yarn cluster mode but I have reason to use client >> mode instead. But I'm hitting this error when submitting: >> > spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest >> --master yarn --deploy-mode client --executor-memory 90G --num-executors 3 >> --executor-cores 14 Martini-assembly-0.1.jar yarn-client >> >> Error: Could not find or load main class >> org.apache.spark.deploy.SparkSubmitDriverBootstrapper >> >> If I replace deploy-mode to cluster the job is submitted successfully. >> Is there a dependency missing from my project? Right now only one I >> included is spark-streaming 1.6.0. >> > > -- Best Regards Jeff Zhang
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn client mode (1.6.0)
Can you show the complete stack trace for the error ? I searched 1.6.0 code base but didn't find the class SparkSubmitDriverBootstrapper Thanks On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao wrote: > My job runs fine in yarn cluster mode but I have reason to use client mode > instead. But I'm hitting this error when submitting: > > spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest > --master yarn --deploy-mode client --executor-memory 90G --num-executors 3 > --executor-cores 14 Martini-assembly-0.1.jar yarn-client > > Error: Could not find or load main class > org.apache.spark.deploy.SparkSubmitDriverBootstrapper > > If I replace deploy-mode to cluster the job is submitted successfully. Is > there a dependency missing from my project? Right now only one I included > is spark-streaming 1.6.0. >
