When launching a job, you need to specify the queue.
"...
mapred to unknown queue
... "
Something along the line of (this is for hadoop1)
-Dmapred.queue.name=<queue name>
Thanks
Olivier
*mapred queue -list* result:
[user@host ~]$ mapred queue -list
13/11/26 09:37:48 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/26 09:37:48 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
======================
Queue Name : exploration
Queue State : running
Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
======================
Queue Name : a
Queue State : running
Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
======================
Queue Name : b
Queue State : running
Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
======================
Queue Name : c
Queue State : running
Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0,
CurrentCapacity: 0.0
======================
Queue Name : production
Queue State : running
Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0
and I saw in RM logs, mapred user try to find default queue, logs are below:
2013-11-26 05:22:59,804 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
with id 1 submitted by user mapred
2013-11-26 05:22:59,808 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
IP=10.10.10.1 OPERATION=Submit Application Request
TARGET=ClientRMService RESULT=SUCCESS APPID=application_1385472086605_0001
2013-11-26 05:22:59,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1385472086605_0001 State change from NEW to SUBMITTED
2013-11-26 05:22:59,829 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering appattempt_1385472086605_0001_000001
2013-11-26 05:22:59,830 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED
2013-11-26 05:22:59,832 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED
2013-11-26 05:22:59,836 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1385472086605_0001 State change from SUBMITTED to FAILED
2013-11-26 05:22:59,837 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred
OPERATION=Application Finished - Failed TARGET=RMAppManager
RESULT=FAILURE DESCRIPTION=App failed with state: FAILED
PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by
user mapred to unknown queue: default APPID=application_1385472086605_0001
2013-11-26 05:22:59,841 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
Removing info for app: application_1385472086605_0001
2013-11-26 05:22:59,842 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
appId=application_1385472086605_0001,name=select count(*) from
crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836
On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault
<[email protected]>wrote:
> Could you confirm that the queue are running by:
> mapred queue -list
>
> Otherwise what error do you get when submitting a job ?
>
> Thanks,
> Olivier
>
>
> On 26 November 2013 15:33, Munna <[email protected]> wrote:
>
>> Below configured queue's are showing, then why i cannt able to run the
>> job.
>>
>> [root@host ~]# hadoop queue -showacls
>> DEPRECATED: Use of this script to execute mapred command is deprecated.
>> Instead use the mapred command for it.
>>
>> 13/11/26 05:25:43 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>> 13/11/26 05:25:44 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>> Queue acls for user : root
>>
>> Queue Operations
>> =====================
>> root ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> exploration ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> a ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> b ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> c ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>> production ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
>>
>>
>>
>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault <
>> [email protected]> wrote:
>>
>>> Sorry my mistake, it's :
>>> mapred queue -showacls
>>>
>>> Or you can also use hadoop queue -showacls as suggested by Jitendra
>>>
>>> Thanks,
>>> Olviier
>>>
>>>
>>> On 26 November 2013 14:17, Munna <[email protected]> wrote:
>>>
>>>> Yes! I have executed same command:
>>>>
>>>> [user@host ~]$* yarn queue -showacls*
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue
>>>> Caused by: java.lang.ClassNotFoundException: queue
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>> Could not find the main class: queue. Program will exit.
>>>>
>>>>
>>>>
>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault <
>>>> [email protected]> wrote:
>>>>
>>>>> it should be :
>>>>> yarn queue -showacls
>>>>>
>>>>> Olivier
>>>>>
>>>>>
>>>>> On 26 November 2013 13:28, Munna <[email protected]> wrote:
>>>>>
>>>>>> Hi Olivier,
>>>>>>
>>>>>> Thank you for your reply.
>>>>>>
>>>>>> As you said, i ran those commands and i am getting following error
>>>>>> message on both the commands.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in thread
>>>>>> "main" java.lang.NoClassDefFoundError: queueCaused by:
>>>>>> java.lang.ClassNotFoundException: queue at
>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) at
>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) at
>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306) at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at
>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the
>>>>>> main class: queue. Program will exit.*
>>>>>>
>>>>>> Regards,
>>>>>> Munna
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Could you maybe send us the output of :
>>>>>>> - yarn queue -showacls
>>>>>>> - yarn queue -list
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>> On 26 November 2013 05:53, Munna <[email protected]> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I working with Capacity Scheduler on YARN and I have configured
>>>>>>>> different queues. I can able to see all the queues on RM ui. But, when
>>>>>>>> i
>>>>>>>> start to run MR jobs with configured user names(yarn,mapred), i am
>>>>>>>> unable
>>>>>>>> to run the Jobs and job are suspended. Again i set default as FIFO
>>>>>>>> working
>>>>>>>> fine.
>>>>>>>>
>>>>>>>> Can you please help me out sort this issue and configured
>>>>>>>> Configurations are given below.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *<property> <name>yarn.scheduler.capacity.root.queues</name>
>>>>>>>> <value>production,exploration</value> </property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name>
>>>>>>>> <value>a,b,c</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name>
>>>>>>>> <value>100</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name>
>>>>>>>> <value>70</value></property> <property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name>
>>>>>>>> <value>30</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name>
>>>>>>>> <value>30</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name>
>>>>>>>> <value>30</value></property> <property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name>
>>>>>>>> <value>40</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.
>>>>>>>> production.acl_submit_applications</name>
>>>>>>>> <value>yarn,mapred</value></property><property> <value>userb</value>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name>
>>>>>>>> </property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name>
>>>>>>>> <value>userb</value></property><property>
>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name>
>>>>>>>> <value>userc</value></property></configuration>*
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Regards*
>>>>>>>>
>>>>>>>> *Munna*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
*Regards*
*Munna*
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
