I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
what is the best value? Tx, Munna On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[email protected]> wrote: > The log shows the both queues are properly picked up by the RM. > If the problem is that your submitted application is not able to run, you > may try increasing yarn.scheduler.capacity.maximum-am-resource-percent, > this controls the max number of concurrently running AMs in the cluster. > > Jian > > > On Wed, Nov 27, 2013 at 9:42 AM, Munna <[email protected]> wrote: > >> Hi Flocks, >> >> >> >> Since, last two days I am about to configure Capacity Scheduler. Here, >> how I have struggling L…. >> >> >> >> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4 >> Queue’s as per the Capacity Scheduler Documentation and those queues shown >> in RM UI. >> >> >> >> After configuration I tried to run Jobs, Cap Scheduler not identified >> that queue’s. where I have check queues list with “mapred queue –list”, >> which showing all configured Q’s. >> >> >> >> I wrote a mail’s to groups for solution, Mr.Olivier has been given some >> idea about that, based on his views I dig more. >> >> >> >> After I went to all the RM log, Cap Scheduler initiating only default >> “default”, I have tested with *default queue* it works for me. And I >> have created one more queue called “dev”, in this Queue User unable to run >> the jobs and its unable to identifying users Queue. >> >> >> >> I have attached Cap Scheduler configuration file for your information. >> Some O/P for ur information. >> >> >> >> *[user@host ~]$ mapred queue -list* >> >> *13/11/27 09:26:38 INFO service.AbstractService: >> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.* >> >> *13/11/27 09:26:38 INFO service.AbstractService: >> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.* >> >> *======================* >> >> *Queue Name : dev* >> >> *Queue State : running* >> >> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5, >> CurrentCapacity: 0.0* >> >> *======================* >> >> *Queue Name : default* >> >> *Queue State : running* >> >> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: >> 0.0* >> >> >> >> *RM log Scheduler loading info:* >> >> 2013-11-27 08:54:58,521 INFO >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, >> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS: >> ADMINISTER_QUEUE: >> >> 2013-11-27 08:54:58,521 INFO >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >> Initialized parent-queue root name=root, fullname=root >> >> 2013-11-27 08:54:58,534 INFO >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >> *Initializing >> default* >> >> capacity = 0.7 [= (float) configuredCapacity / 100 ] >> >> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ] >> >> maxCapacity = 1.0 [= configuredMaxCapacity ] >> >> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, >> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >> >> userLimit = 100 [= configuredUserLimit ] >> >> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >> >> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or >> (int)(configuredMaximumSystemApplications * absoluteCapacity)] >> >> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit / >> 100.0f) * userLimitFactor) ] >> >> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) >> ] >> >> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / >> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >> >> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >> (userLimit / 100.0f) * userLimitFactor),1) ] >> >> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >> absoluteCapacity)] >> >> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory] >> >> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] >> >> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >> minimumAllocationMemory) / maximumAllocationMemory ] >> >> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] >> >> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >> minimumAllocationMemory) / maximumAllocationMemory ] >> >> numContainers = 0 [= currentNumContainers ] >> >> state = RUNNING [= configuredState ] >> >> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= >> configuredAcls ] >> >> >> >> 2013-11-27 08:54:58,534 INFO >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: >> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7, >> usedResources=<memory:0, vCores:0>usedCapacity=0.0, >> absoluteUsedCapacity=0.0, numApps=0, numContainers=0 >> >> 2013-11-27 08:54:58,543 INFO >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >> *Initializing >> dev* >> >> capacity = 0.3 [= (float) configuredCapacity / 100 ] >> >> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] >> >> maxCapacity = 0.5 [= configuredMaxCapacity ] >> >> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined, >> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >> >> userLimit = 100 [= configuredUserLimit ] >> >> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >> >> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or >> (int)(configuredMaximumSystemApplications * absoluteCapacity)] >> >> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / >> 100.0f) * userLimitFactor) ] >> >> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) >> ] >> >> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / >> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >> >> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >> (userLimit / 100.0f) * userLimitFactor),1) ] >> >> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >> absoluteCapacity)] >> >> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory] >> >> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] >> >> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >> minimumAllocationMemory) / maximumAllocationMemory ] >> >> numContainers = 0 [= currentNumContainers ] >> >> state = RUNNING [= configuredState ] >> >> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= >> configuredAcls ] >> >> >> >> Can you guys please confirm, did I miss anything on configurations part >> or is there any bug persist on 2.0.0? >> >> >> >> Thanks >> >> Munna >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. -- *Regards* *Munna*
