Re: spark on yarn cluster can't launch

Hi, Do you know what YARN scheduler you're using and what version of YARN? It seems like this would be caused by YarnClient.getQueueInfo returning null, though, from browsing the YARN code, I'm not sure how this could happen. -Sandy On Fri, Aug 15, 2014 at 11:23 AM, Andrew Or

Re: spark on yarn cluster can't launch

On closer look, it seems like this can occur if the queue doesn't exist. Filed https://issues.apache.org/jira/browse/SPARK-3082. -Sandy On Sat, Aug 16, 2014 at 12:49 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi, Do you know what YARN scheduler you're using and what version of YARN? It

spark on yarn cluster can't launch

The code does not run as follows ../bin/spark-submit --class org.apache.spark.examples.SparkPi \ --master yarn \ --deploy-mode cluster \ --verbose \ --num-executors 3 \ --driver-memory 4g \ --executor-memory 2g \ --executor-cores 1 \ ../lib/spark-examples*.jar \ 100 Exception in thread

Re: spark on yarn cluster can't launch

Hi 齐忠, Thanks for reporting this. You're correct that the default deploy mode is client. However, this seems to be a bug in the YARN integration code; we should not throw null pointer exception in any case. What version of Spark are you using? Andrew 2014-08-15 0:23 GMT-07:00 centerqi hu