I am using Hadoop 2.0.0 and YARN 2.0.0,... is it require to set any parameter on mapred-site.xml or yarn-site.xml?
On Wed, Nov 27, 2013 at 1:03 AM, Olivier Renault <[email protected]>wrote: > When launching a job, you need to specify the queue. > "... > mapred to unknown queue > ... " > > Something along the line of (this is for hadoop1) > > -Dmapred.queue.name=<queue name> > > Thanks > Olivier > *mapred queue -list* result: > > [user@host ~]$ mapred queue -list > 13/11/26 09:37:48 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. > 13/11/26 09:37:48 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. > ====================== > Queue Name : exploration > Queue State : running > Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, > CurrentCapacity: 0.0 > ====================== > Queue Name : a > Queue State : running > Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, > CurrentCapacity: 0.0 > ====================== > Queue Name : b > Queue State : running > Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, > CurrentCapacity: 0.0 > ====================== > Queue Name : c > Queue State : running > Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, > CurrentCapacity: 0.0 > ====================== > Queue Name : production > Queue State : running > Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: > 0.0 > > and I saw in RM logs, mapred user try to find default queue, logs are > below: > > 2013-11-26 05:22:59,804 INFO > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application > with id 1 submitted by user mapred > 2013-11-26 05:22:59,808 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred > IP=10.10.10.1 OPERATION=Submit Application Request > TARGET=ClientRMService RESULT=SUCCESS APPID=application_1385472086605_0001 > 2013-11-26 05:22:59,828 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: > application_1385472086605_0001 State change from NEW to SUBMITTED > 2013-11-26 05:22:59,829 INFO > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Registering appattempt_1385472086605_0001_000001 > 2013-11-26 05:22:59,830 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED > 2013-11-26 05:22:59,832 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED > 2013-11-26 05:22:59,836 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: > application_1385472086605_0001 State change from SUBMITTED to FAILED > 2013-11-26 05:22:59,837 WARN > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred > OPERATION=Application Finished - Failed TARGET=RMAppManager > RESULT=FAILURE DESCRIPTION=App failed with state: FAILED > PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by > user mapred to unknown queue: default APPID=application_1385472086605_0001 > 2013-11-26 05:22:59,841 INFO > org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: > Removing info for app: application_1385472086605_0001 > 2013-11-26 05:22:59,842 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: > appId=application_1385472086605_0001,name=select count(*) from > crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836 > > > > On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault < > [email protected]> wrote: > >> Could you confirm that the queue are running by: >> mapred queue -list >> >> Otherwise what error do you get when submitting a job ? >> >> Thanks, >> Olivier >> >> >> On 26 November 2013 15:33, Munna <[email protected]> wrote: >> >>> Below configured queue's are showing, then why i cannt able to run the >>> job. >>> >>> [root@host ~]# hadoop queue -showacls >>> DEPRECATED: Use of this script to execute mapred command is deprecated. >>> Instead use the mapred command for it. >>> >>> 13/11/26 05:25:43 INFO service.AbstractService: >>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >>> 13/11/26 05:25:44 INFO service.AbstractService: >>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >>> Queue acls for user : root >>> >>> Queue Operations >>> ===================== >>> root ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> exploration ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> a ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> b ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> c ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> production ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>> >>> >>> >>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault < >>> [email protected]> wrote: >>> >>>> Sorry my mistake, it's : >>>> mapred queue -showacls >>>> >>>> Or you can also use hadoop queue -showacls as suggested by Jitendra >>>> >>>> Thanks, >>>> Olviier >>>> >>>> >>>> On 26 November 2013 14:17, Munna <[email protected]> wrote: >>>> >>>>> Yes! I have executed same command: >>>>> >>>>> [user@host ~]$* yarn queue -showacls* >>>>> >>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue >>>>> Caused by: java.lang.ClassNotFoundException: queue >>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>>> at >>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>>> Could not find the main class: queue. Program will exit. >>>>> >>>>> >>>>> >>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault < >>>>> [email protected]> wrote: >>>>> >>>>>> it should be : >>>>>> yarn queue -showacls >>>>>> >>>>>> Olivier >>>>>> >>>>>> >>>>>> On 26 November 2013 13:28, Munna <[email protected]> wrote: >>>>>> >>>>>>> Hi Olivier, >>>>>>> >>>>>>> Thank you for your reply. >>>>>>> >>>>>>> As you said, i ran those commands and i am getting following error >>>>>>> message on both the commands. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in >>>>>>> thread "main" java.lang.NoClassDefFoundError: queueCaused by: >>>>>>> java.lang.ClassNotFoundException: queue at >>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) at >>>>>>> java.security.AccessController.doPrivileged(Native Method) at >>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) at >>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306) at >>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at >>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the >>>>>>> main class: queue. Program will exit.* >>>>>>> >>>>>>> Regards, >>>>>>> Munna >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Could you maybe send us the output of : >>>>>>>> - yarn queue -showacls >>>>>>>> - yarn queue -list >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Olivier >>>>>>>> >>>>>>>> >>>>>>>> On 26 November 2013 05:53, Munna <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I working with Capacity Scheduler on YARN and I have configured >>>>>>>>> different queues. I can able to see all the queues on RM ui. But, >>>>>>>>> when i >>>>>>>>> start to run MR jobs with configured user names(yarn,mapred), i am >>>>>>>>> unable >>>>>>>>> to run the Jobs and job are suspended. Again i set default as FIFO >>>>>>>>> working >>>>>>>>> fine. >>>>>>>>> >>>>>>>>> Can you please help me out sort this issue and configured >>>>>>>>> Configurations are given below. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *<property> <name>yarn.scheduler.capacity.root.queues</name> >>>>>>>>> <value>production,exploration</value> </property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name> >>>>>>>>> <value>a,b,c</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name> >>>>>>>>> <value>100</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name> >>>>>>>>> <value>70</value></property> <property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name> >>>>>>>>> <value>30</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name> >>>>>>>>> <value>30</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name> >>>>>>>>> <value>30</value></property> <property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name> >>>>>>>>> <value>40</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root. >>>>>>>>> production.acl_submit_applications</name> >>>>>>>>> <value>yarn,mapred</value></property><property> >>>>>>>>> <value>userb</value> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name> >>>>>>>>> </property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name> >>>>>>>>> <value>userb</value></property><property> >>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name> >>>>>>>>> <value>userc</value></property></configuration>* >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Regards* >>>>>>>>> >>>>>>>>> *Munna* >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > > > -- > *Regards* > > *Munna* > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- *Regards* *Munna*
