I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator.
Thanks Jerry On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <sunxiaoy...@gmail.com> wrote: > Hi, > > I am using Spark 1.6.1 and Yarn 2.7.4. > I want to submit a Spark application to a Yarn cluster. However, I found > that the number of vcores assigned to a container/executor is always 1, > even if I set spark.executor.cores=2. I also found the number of tasks an > executor runs concurrently is 2. So, it seems that Spark knows that an > executor/container has two CPU cores but the request is not correctly sent > to Yarn resource scheduler. I am using the org.apache.hadoop.yarn. > server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn. > > I am wondering that is it possible to assign multiple vcores to a > container when a Spark job is submitted to a Yarn cluster in yarn-cluster > mode. > > Thanks! > Best, > Xiaoye >
