Hi All, Thank you for taking your time to help me.
Now I can able to run the jobs within configured Queues. Regards, Munna On Fri, Nov 29, 2013 at 2:22 AM, Jian He <[email protected]> wrote: > Did the application get accepted ? If the application gets accepted but > just not able to run, as I said, try increasing > yarn.scheduler.capacity.maximum-am-resource-percent > to a much larger value, maybe 0.8, just for troubleshoot. How large it > should be depends on your requirements. > > yarn.scheduler.capacity.maximum-am-resource-percent controls the max > number of concurrently running AMs. > By that I mean the max memory allowed for allocating AM (defined by this > property) divided by per AM memory usage , equals to the max number of > concurrently running AMs. > > I have seen both of your queues only allow 1 active application. > maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / > minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) > ] > > Jian > > > On Thu, Nov 28, 2013 at 12:23 PM, Adam Kawa <[email protected]> wrote: > >> I see that you have different settings for ACL: >> >> <name>yarn.scheduler.capacity.root.*default* >> .acl_submit_applications</name><value>*yarn,mapred* >> </value></property><property> >> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= >> configuredAcls ] >> >> <name>yarn.scheduler.capacity.root.*dev* >> .acl_submit_applications</name><value>*smunnavar,test*</value></property> >> >> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls >> ] >> >> Do you submit your jobs to dev queue using good user accounts? Did you >> include a right config xml in a previous post? (xml says *smunnavar,test*, >> but RM loads user,test). >> >> As a first troubleshooting steps, could you disable ACL for dev queue to >> check if you can submit a job there? >> >> <property> >> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name> >> <value>*</value> >> </property> >> >> >> >> 2013/11/28 Munna <[email protected]> >> >>> Hi, >>> >>> >>> I think there is no solution on above issue, so i'll move to fair >>> scheduler. >>> >>> Thanks to all... >>> >>> Munna >>> >>> >>> On Thu, Nov 28, 2013 at 9:11 AM, Munna <[email protected]> wrote: >>> >>>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1* >>>> >>>> >>>> what is the best value? >>>> >>>> Tx, >>>> Munna >>>> >>>> >>>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[email protected]> wrote: >>>> >>>>> The log shows the both queues are properly picked up by the RM. >>>>> If the problem is that your submitted application is not able to run, >>>>> you may try increasing >>>>> yarn.scheduler.capacity.maximum-am-resource-percent, >>>>> this controls the max number of concurrently running AMs in the cluster. >>>>> >>>>> Jian >>>>> >>>>> >>>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[email protected]> wrote: >>>>> >>>>>> Hi Flocks, >>>>>> >>>>>> >>>>>> >>>>>> Since, last two days I am about to configure Capacity Scheduler. >>>>>> Here, how I have struggling L…. >>>>>> >>>>>> >>>>>> >>>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have >>>>>> created 4 Queue’s as per the Capacity Scheduler Documentation and those >>>>>> queues shown in RM UI. >>>>>> >>>>>> >>>>>> >>>>>> After configuration I tried to run Jobs, Cap Scheduler not identified >>>>>> that queue’s. where I have check queues list with “mapred queue –list”, >>>>>> which showing all configured Q’s. >>>>>> >>>>>> >>>>>> >>>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given >>>>>> some idea about that, based on his views I dig more. >>>>>> >>>>>> >>>>>> >>>>>> After I went to all the RM log, Cap Scheduler initiating only default >>>>>> “default”, I have tested with *default queue* it works for me. And I >>>>>> have created one more queue called “dev”, in this Queue User unable to >>>>>> run >>>>>> the jobs and its unable to identifying users Queue. >>>>>> >>>>>> >>>>>> >>>>>> I have attached Cap Scheduler configuration file for your >>>>>> information. Some O/P for ur information. >>>>>> >>>>>> >>>>>> >>>>>> *[user@host ~]$ mapred queue -list* >>>>>> >>>>>> *13/11/27 09:26:38 INFO service.AbstractService: >>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.* >>>>>> >>>>>> *13/11/27 09:26:38 INFO service.AbstractService: >>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.* >>>>>> >>>>>> *======================* >>>>>> >>>>>> *Queue Name : dev* >>>>>> >>>>>> *Queue State : running* >>>>>> >>>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5, >>>>>> CurrentCapacity: 0.0* >>>>>> >>>>>> *======================* >>>>>> >>>>>> *Queue Name : default* >>>>>> >>>>>> *Queue State : running* >>>>>> >>>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, >>>>>> CurrentCapacity: 0.0* >>>>>> >>>>>> >>>>>> >>>>>> *RM log Scheduler loading info:* >>>>>> >>>>>> 2013-11-27 08:54:58,521 INFO >>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >>>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, >>>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS: >>>>>> ADMINISTER_QUEUE: >>>>>> >>>>>> 2013-11-27 08:54:58,521 INFO >>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: >>>>>> Initialized parent-queue root name=root, fullname=root >>>>>> >>>>>> 2013-11-27 08:54:58,534 INFO >>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >>>>>> *Initializing >>>>>> default* >>>>>> >>>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ] >>>>>> >>>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ] >>>>>> >>>>>> maxCapacity = 1.0 [= configuredMaxCapacity ] >>>>>> >>>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, >>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >>>>>> >>>>>> userLimit = 100 [= configuredUserLimit ] >>>>>> >>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >>>>>> >>>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue >>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)] >>>>>> >>>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit / >>>>>> 100.0f) * userLimitFactor) ] >>>>>> >>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * >>>>>> absoluteMaxCapacity),1) >>>>>> ] >>>>>> >>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory >>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >>>>>> >>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >>>>>> (userLimit / 100.0f) * userLimitFactor),1) ] >>>>>> >>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >>>>>> absoluteCapacity)] >>>>>> >>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / >>>>>> clusterResourceMemory] >>>>>> >>>>>> maxAMResourcePerQueuePercent = 0.1 [= >>>>>> configuredMaximumAMResourcePercent ] >>>>>> >>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>>>>> minimumAllocationMemory) / maximumAllocationMemory ] >>>>>> >>>>>> maxAMResourcePerQueuePercent = 0.1 [= >>>>>> configuredMaximumAMResourcePercent ] >>>>>> >>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>>>>> minimumAllocationMemory) / maximumAllocationMemory ] >>>>>> >>>>>> numContainers = 0 [= currentNumContainers ] >>>>>> >>>>>> state = RUNNING [= configuredState ] >>>>>> >>>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= >>>>>> configuredAcls ] >>>>>> >>>>>> >>>>>> >>>>>> 2013-11-27 08:54:58,534 INFO >>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: >>>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7, >>>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0, >>>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0 >>>>>> >>>>>> 2013-11-27 08:54:58,543 INFO >>>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: >>>>>> *Initializing >>>>>> dev* >>>>>> >>>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ] >>>>>> >>>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] >>>>>> >>>>>> maxCapacity = 0.5 [= configuredMaxCapacity ] >>>>>> >>>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined, >>>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] >>>>>> >>>>>> userLimit = 100 [= configuredUserLimit ] >>>>>> >>>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ] >>>>>> >>>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue >>>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)] >>>>>> >>>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / >>>>>> 100.0f) * userLimitFactor) ] >>>>>> >>>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / >>>>>> minimumAllocation) * maxAMResourcePerQueuePercent * >>>>>> absoluteMaxCapacity),1) >>>>>> ] >>>>>> >>>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory >>>>>> / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] >>>>>> >>>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * >>>>>> (userLimit / 100.0f) * userLimitFactor),1) ] >>>>>> >>>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * >>>>>> absoluteCapacity)] >>>>>> >>>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory / >>>>>> clusterResourceMemory] >>>>>> >>>>>> maxAMResourcePerQueuePercent = 0.1 [= >>>>>> configuredMaximumAMResourcePercent ] >>>>>> >>>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - >>>>>> minimumAllocationMemory) / maximumAllocationMemory ] >>>>>> >>>>>> numContainers = 0 [= currentNumContainers ] >>>>>> >>>>>> state = RUNNING [= configuredState ] >>>>>> >>>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= >>>>>> configuredAcls ] >>>>>> >>>>>> >>>>>> >>>>>> Can you guys please confirm, did I miss anything on configurations >>>>>> part or is there any bug persist on 2.0.0? >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Munna >>>>>> >>>>> >>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>> If the reader of this message is not the intended recipient, you are >>>>> hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>> you have received this communication in error, please contact the sender >>>>> immediately and delete it from your system. Thank You. >>>> >>>> >>>> >>>> >>>> -- >>>> *Regards* >>>> >>>> *Munna* >>>> >>> >>> >>> >>> -- >>> *Regards* >>> >>> *Munna* >>> >> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- *Regards* *Munna*
