OK, I figured out this. The maximum number of containers YARN can create per node is based on the total available RAM and the maximum allocation per container ( yarn.scheduler.maximum-allocation-mb ). The default is 8192; setting to a lower value allowed me to create more containers per node.
On Mon, Jun 22, 2015 at 10:42 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > 1) Can you try with yarn-cluster > 2) Does your queue have enough capacity > > On Mon, Jun 22, 2015 at 11:10 AM, Saiph Kappa <saiph.ka...@gmail.com> > wrote: > >> Hi, >> >> I am running a simple spark streaming application on hadoop 2.7.0/YARN >> (master: yarn-client) cluster with 2 different machines (12GB RAM with 8 >> CPU cores each). >> >> I am launching my application like this: >> >> ~/myapp$ ~/my-spark/bin/spark-submit --class App --master yarn-client >> --driver-memory 4g --executor-memory 2g --executor-cores 1 --num-executors >> 6 target/scala-2.10/my-app_2.10-0.1-SNAPSHOT.jar 1 mymachine3 9999 1000 8 >> 10 4 stdev 3 >> >> Despite I required 6 executors for my application, it seems that I am >> unable to get more than 4 executors (2 per machine). If I request any >> number of executors below 5 it works fine, but otherwise it seems that it >> is not able to allocate more than 4. Why does this happen? >> >> Thanks. >> > > > > -- > Deepak > >
