Hi Patrick, It's hard to diagnose CDH since I don't know what patch-sets they have for the CapacityScheduler - afaik they only support FairScheduler, but that might have changed.
On Oct 15, 2011, at 4:45 PM, patrick sang wrote: > 4. from webUI, scheduling information of orange queue. > > It said "Used capacity: 12 (100.0% of Capacity)" > while next line said "Maximum capacity: 16 slots" > So what's going on with other 4 slots ? why they are not get used. > > Is capacity-scheduler supposed to start using extra slots until it hit the > Max capacity ? > (from the variable of > mapred.capacity-scheduler.queue.<queue-name>.maximum-capacity) > (there are no other jobs at all in the cluster) > > I am really thankful for reading up to this point. > Truly hope someone can shed some light on this. > However, if you were using Apache Hadoop 0.20.203 or 0.20.204 (or upcoming 0.20.205 with security + append) you would still see this behaviour because you are hitting 'user limits' where the CS will not allow a single user to take more than the queue 'configured' capacity (12 slots here). You will need more than one user in the 'orange' queue to go over the queue's capacity. This is to prevent a single user from hogging the system's resources. If you really want one user to acquire more resources in 'orange' queue, you need to tweak mapred.capacity-scheduler.queue.orange.user-limit-factor. More details here: http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html Arun
