Another update that doesn't make much sense: The SparkPi example does work on yarn-cluster mode with dynamicAllocation.
That is, the following command works (as well as with yarn-client mode): spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi spark-examples.jar 100 But the following one does not work (nor does it work for yarn-client mode): spark-submit --deploy-mode cluster --class org.apache.spark.examples.JavaWordCount spark-examples.jar /tmp/word-count-input.txt So this JavaWordCount example hangs on requesting executors, while SparkPi and spark-shell do work. ~ Jonathan On Wed, Sep 23, 2015 at 6:22 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote: > Thanks for the quick response! > > spark-shell is indeed using yarn-client. I forgot to mention that I also > have "spark.master yarn-client" in my spark-defaults.conf file too. > > The working spark-shell and my non-working example application both > display spark.scheduler.mode=FIFO on the Spark UI. Is that what you are > asking about? I haven't actually messed around with different scheduler > modes yet. > > One more thing I should mention is that the YARN ResourceManager tells me > the following on my 5-node cluster, with one node being the master and not > running a NodeManager: > Memory Used: 1.50 GB (this is the running ApplicationMaster that's waiting > and waiting for the executors to start up) > Memory Total: 45 GB (11.25 from each of the 4 slave nodes) > VCores Used: 1 > VCores Total: 32 > Active Nodes: 4 > > ~ Jonathan > > On Wed, Sep 23, 2015 at 6:10 PM, Andrew Duffy <andrewedu...@gmail.com> > wrote: > >> What pool is the spark shell being put into? (You can see this through >> the YARN UI under scheduler) >> >> Are you certain you're starting spark-shell up on YARN? By default it >> uses a local spark executor, so if it "just works" then it's because it's >> not using dynamic allocation. >> >> >> On Wed, Sep 23, 2015 at 18:04 Jonathan Kelly <jonathaka...@gmail.com> >> wrote: >> >>> I'm running into a problem with YARN dynamicAllocation on Spark 1.5.0 >>> after using it successfully on an identically configured cluster with Spark >>> 1.4.1. >>> >>> I'm getting the dreaded warning "YarnClusterScheduler: Initial job has >>> not accepted any resources; check your cluster UI to ensure that workers >>> are registered and have sufficient resources", though there's nothing else >>> running on my cluster, and the nodes should have plenty of resources to run >>> my application. >>> >>> Here are the applicable properties in spark-defaults.conf: >>> spark.dynamicAllocation.enabled true >>> spark.dynamicAllocation.minExecutors 1 >>> spark.shuffle.service.enabled true >>> >>> When trying out my example application (just the JavaWordCount example >>> that comes with Spark), I had not actually set spark.executor.memory or any >>> CPU core-related properties, but setting the spark.executor.memory to a low >>> value like 64m doesn't help either. >>> >>> I've tried a 5-node cluster and 1-node cluster of m3.xlarges, so each >>> node has 15.0GB and 4 cores. >>> >>> I've also tried both yarn-cluster and yarn-client mode and get the same >>> behavior for both, except that for yarn-client mode the application never >>> even shows up in the YARN ResourceManager. However, spark-shell seems to >>> work just fine (when I run commands, it starts up executors dynamically >>> just fine), which makes no sense to me. >>> >>> What settings/logs should I look at to debug this, and what more >>> information can I provide? Your help would be very much appreciated! >>> >>> Thanks, >>> Jonathan >>> >> >
