I see that you have different settings for ACL:
<name>yarn.scheduler.capacity.root.*default*
.acl_submit_applications</name><value>*yarn,mapred*
</value></property><property>
acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls
]
<name>yarn.scheduler.capacity.root.*dev*
.acl_submit_applications</name><value>*smunnavar,test*</value></property>
acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ]
Do you submit your jobs to dev queue using good user accounts? Did you
include a right config xml in a previous post? (xml says *smunnavar,test*,
but RM loads user,test).
As a first troubleshooting steps, could you disable ACL for dev queue to
check if you can submit a job there?
<property>
<name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
<value>*</value>
</property>
2013/11/28 Munna <[email protected]>
> Hi,
>
>
> I think there is no solution on above issue, so i'll move to fair
> scheduler.
>
> Thanks to all...
>
> Munna
>
>
> On Thu, Nov 28, 2013 at 9:11 AM, Munna <[email protected]> wrote:
>
>> I have set: *yarn.scheduler.capacity.maximum-am-resource-percent=0.1*
>>
>>
>> what is the best value?
>>
>> Tx,
>> Munna
>>
>>
>> On Thu, Nov 28, 2013 at 12:34 AM, Jian He <[email protected]> wrote:
>>
>>> The log shows the both queues are properly picked up by the RM.
>>> If the problem is that your submitted application is not able to run,
>>> you may try increasing yarn.scheduler.capacity.maximum-am-resource-percent,
>>> this controls the max number of concurrently running AMs in the cluster.
>>>
>>> Jian
>>>
>>>
>>> On Wed, Nov 27, 2013 at 9:42 AM, Munna <[email protected]> wrote:
>>>
>>>> Hi Flocks,
>>>>
>>>>
>>>>
>>>> Since, last two days I am about to configure Capacity Scheduler. Here,
>>>> how I have struggling L….
>>>>
>>>>
>>>>
>>>> I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created
>>>> 4 Queue’s as per the Capacity Scheduler Documentation and those queues
>>>> shown in RM UI.
>>>>
>>>>
>>>>
>>>> After configuration I tried to run Jobs, Cap Scheduler not identified
>>>> that queue’s. where I have check queues list with “mapred queue –list”,
>>>> which showing all configured Q’s.
>>>>
>>>>
>>>>
>>>> I wrote a mail’s to groups for solution, Mr.Olivier has been given some
>>>> idea about that, based on his views I dig more.
>>>>
>>>>
>>>>
>>>> After I went to all the RM log, Cap Scheduler initiating only default
>>>> “default”, I have tested with *default queue* it works for me. And I
>>>> have created one more queue called “dev”, in this Queue User unable to run
>>>> the jobs and its unable to identifying users Queue.
>>>>
>>>>
>>>>
>>>> I have attached Cap Scheduler configuration file for your information.
>>>> Some O/P for ur information.
>>>>
>>>>
>>>>
>>>> *[user@host ~]$ mapred queue -list*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*
>>>>
>>>> *13/11/27 09:26:38 INFO service.AbstractService:
>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : dev*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
>>>> CurrentCapacity: 0.0*
>>>>
>>>> *======================*
>>>>
>>>> *Queue Name : default*
>>>>
>>>> *Queue State : running*
>>>>
>>>> *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0,
>>>> CurrentCapacity: 0.0*
>>>>
>>>>
>>>>
>>>> *RM log Scheduler loading info:*
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
>>>> asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
>>>> ADMINISTER_QUEUE:
>>>>
>>>> 2013-11-27 08:54:58,521 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
>>>> Initialized parent-queue root name=root, fullname=root
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
>>>> *Initializing
>>>> default*
>>>>
>>>> capacity = 0.7 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 1.0 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> 2013-11-27 08:54:58,534 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>>>> Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
>>>> usedResources=<memory:0, vCores:0>usedCapacity=0.0,
>>>> absoluteUsedCapacity=0.0, numApps=0, numContainers=0
>>>>
>>>> 2013-11-27 08:54:58,543 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
>>>> *Initializing
>>>> dev*
>>>>
>>>> capacity = 0.3 [= (float) configuredCapacity / 100 ]
>>>>
>>>> asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
>>>>
>>>> maxCapacity = 0.5 [= configuredMaxCapacity ]
>>>>
>>>> absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
>>>> (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
>>>>
>>>> userLimit = 100 [= configuredUserLimit ]
>>>>
>>>> userLimitFactor = 1.0 [= configuredUserLimitFactor ]
>>>>
>>>> maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue
>>>> or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
>>>>
>>>> maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
>>>> 100.0f) * userLimitFactor) ]
>>>>
>>>> maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
>>>> ]
>>>>
>>>> maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
>>>> minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
>>>>
>>>> maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
>>>> (userLimit / 100.0f) * userLimitFactor),1) ]
>>>>
>>>> usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
>>>> absoluteCapacity)]
>>>>
>>>> absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
>>>> clusterResourceMemory]
>>>>
>>>> maxAMResourcePerQueuePercent = 0.1 [=
>>>> configuredMaximumAMResourcePercent ]
>>>>
>>>> minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
>>>> minimumAllocationMemory) / maximumAllocationMemory ]
>>>>
>>>> numContainers = 0 [= currentNumContainers ]
>>>>
>>>> state = RUNNING [= configuredState ]
>>>>
>>>> acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [=
>>>> configuredAcls ]
>>>>
>>>>
>>>>
>>>> Can you guys please confirm, did I miss anything on configurations part
>>>> or is there any bug persist on 2.0.0?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Munna
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> *Regards*
>>
>> *Munna*
>>
>
>
>
> --
> *Regards*
>
> *Munna*
>
