Hi, I am running a simple spark streaming application on hadoop 2.7.0/YARN (master: yarn-client) cluster with 2 different machines (12GB RAM with 8 CPU cores each).
I am launching my application like this: ~/myapp$ ~/my-spark/bin/spark-submit --class App --master yarn-client --driver-memory 4g --executor-memory 2g --executor-cores 1 --num-executors 6 target/scala-2.10/my-app_2.10-0.1-SNAPSHOT.jar 1 mymachine3 9999 1000 8 10 4 stdev 3 Despite I required 6 executors for my application, it seems that I am unable to get more than 4 executors (2 per machine). If I request any number of executors below 5 it works fine, but otherwise it seems that it is not able to allocate more than 4. Why does this happen? Thanks.
