On closer look, it seems like this can occur if the queue doesn't exist. Filed https://issues.apache.org/jira/browse/SPARK-3082.
-Sandy On Sat, Aug 16, 2014 at 12:49 AM, Sandy Ryza <[email protected]> wrote: > Hi, > > Do you know what YARN scheduler you're using and what version of YARN? It > seems like this would be caused by YarnClient.getQueueInfo returning null, > though, from browsing the YARN code, I'm not sure how this could happen. > > -Sandy > > > On Fri, Aug 15, 2014 at 11:23 AM, Andrew Or <[email protected]> wrote: > >> Hi 齐忠, >> >> Thanks for reporting this. You're correct that the default deploy mode is >> "client". However, this seems to be a bug in the YARN integration code; we >> should not throw null pointer exception in any case. What version of Spark >> are you using? >> >> Andrew >> >> >> 2014-08-15 0:23 GMT-07:00 centerqi hu <[email protected]>: >> >> The code does not run as follows >>> >>> ../bin/spark-submit --class org.apache.spark.examples.SparkPi \ >>> >>> --master yarn \ >>> >>> --deploy-mode cluster \ >>> >>> --verbose \ >>> >>> --num-executors 3 \ >>> >>> --driver-memory 4g \ >>> >>> --executor-memory 2g \ >>> >>> --executor-cores 1 \ >>> >>> ../lib/spark-examples*.jar \ >>> >>> 100 >>> >>> Exception in thread "main" java.lang.NullPointerException >>> >>> at >>> org.apache.spark.deploy.yarn.Client$anonfun$logClusterResourceDetails$2.apply(Client.scala:109) >>> >>> at >>> org.apache.spark.deploy.yarn.Client$anonfun$logClusterResourceDetails$2.apply(Client.scala:108) >>> >>> at org.apache.spark.Logging$class.logInfo(Logging.scala:58) >>> >>> >>> However, when I removed "--deploy-mode cluster \" >>> >>> Exception disappear. >>> >>> I think with the "deploy-mode cluster" is running in yarn cluster mode, >>> if not, the default will be run in yarn client mode. >>> >>> But why did yarn cluster get Exception? >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> -- >>> [email protected]|齐忠 >>> >> >> >
