Hi,
I think there is no solution on above issue, so i'll move to fair scheduler. Thanks to all... Munna On Thu, Nov 28, 2013 at 9:11 AM, Munna <[email protected]> wrote: > I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1* > > > what is the best value? > > Tx, > Munna > > > On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[email protected]> wrote: > >> The log shows the both queues are properly picked up by the RM. >> If the problem is that your submitted application is not able to run, you >> may try increasing yarn.scheduler.capacity.maximum-am-resource-percent, >> this controls the max number of concurrently running AMs in the cluster. >> >> Jian >> >> >> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[email protected]> wrote: >> >>> Hi Flocks, >>> >>> >>> >>> Since, last two days I am about to configure Capacity Scheduler. Here, >>> how I have struggling L…. >>> >>> >>> >>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4 >>> Queue’s as per the Capacity Scheduler Documentation and those queues shown >>> in RM UI. >>> >>> >>> >>> After configuration I tried to run Jobs, Cap Scheduler not identified >>> that queue’s. where I have check queues list with “mapred queue –list”, >>> which showing all configured Q’s. >>> >>> >>> >>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some >>> idea about that, based on his views I dig more. >>> >>> >>> >>> After I went to all the RM log, Cap Scheduler initiating only default >>> “default”, I have tested with *default queue* it works for me. And I >>> have created one more queue called “dev”, in this Queue User unable to run >>> the jobs and its unable to identifying users Queue. >>> >>> >>> >>> I have attached Cap Scheduler configuration file for your information. >>> Some O/P for ur information. >>> >>> >>> >>> *[user@host ~]$ mapred queue -list* >>> >>> *13/11/27 09:26:38 INFO service.AbstractService: >>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.* >>> >>> *13/11/27 09:26:38 INFO service.AbstractService: >>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.* >>> >>> *======================* >>> >>> *Queue Name : dev* >>> >>> *Queue State : running* >>> >>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5, >>> CurrentCapacity: 0.0* >>> >>> *======================* >>> >>> *Queue Name : default* >>> >>> *Queue State : running* >>> >>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, >>> CurrentCapacity: 0.0* >>> >>> >>> >>> *RM log Scheduler loading info:* >>> >>> 2013-11-27 08:54:58,521 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, >>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS: >>> ADMINISTER_QUEUE: >>> >>> 2013-11-27 08:54:58,521 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >>> Initialized parent-queue root name=root, fullname=root >>> >>> 2013-11-27 08:54:58,534 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >>> *Initializing >>> default* >>> >>> capacity = 0.7 [= (float) configuredCapacity / 100 ] >>> >>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ] >>> >>> maxCapacity = 1.0 [= configuredMaxCapacity ] >>> >>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, >>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >>> >>> userLimit = 100 [= configuredUserLimit ] >>> >>> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >>> >>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or >>> (int)(configuredMaximumSystemApplications * absoluteCapacity)] >>> >>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit / >>> 100.0f) * userLimitFactor) ] >>> >>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) >>> ] >>> >>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / >>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >>> >>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >>> (userLimit / 100.0f) * userLimitFactor),1) ] >>> >>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >>> absoluteCapacity)] >>> >>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / >>> clusterResourceMemory] >>> >>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent >>> ] >>> >>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>> minimumAllocationMemory) / maximumAllocationMemory ] >>> >>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent >>> ] >>> >>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>> minimumAllocationMemory) / maximumAllocationMemory ] >>> >>> numContainers = 0 [= currentNumContainers ] >>> >>> state = RUNNING [= configuredState ] >>> >>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= >>> configuredAcls ] >>> >>> >>> >>> 2013-11-27 08:54:58,534 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: >>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7, >>> usedResources=<memory:0, vCores:0>usedCapacity=0.0, >>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0 >>> >>> 2013-11-27 08:54:58,543 INFO >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >>> *Initializing >>> dev* >>> >>> capacity = 0.3 [= (float) configuredCapacity / 100 ] >>> >>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] >>> >>> maxCapacity = 0.5 [= configuredMaxCapacity ] >>> >>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined, >>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >>> >>> userLimit = 100 [= configuredUserLimit ] >>> >>> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >>> >>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or >>> (int)(configuredMaximumSystemApplications * absoluteCapacity)] >>> >>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / >>> 100.0f) * userLimitFactor) ] >>> >>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) >>> ] >>> >>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / >>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >>> >>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >>> (userLimit / 100.0f) * userLimitFactor),1) ] >>> >>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >>> absoluteCapacity)] >>> >>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / >>> clusterResourceMemory] >>> >>> maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent >>> ] >>> >>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>> minimumAllocationMemory) / maximumAllocationMemory ] >>> >>> numContainers = 0 [= currentNumContainers ] >>> >>> state = RUNNING [= configuredState ] >>> >>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= >>> configuredAcls ] >>> >>> >>> >>> Can you guys please confirm, did I miss anything on configurations part >>> or is there any bug persist on 2.0.0? >>> >>> >>> >>> Thanks >>> >>> Munna >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > > > > -- > *Regards* > > *Munna* > -- *Regards* *Munna*
