Hi, Can i able to get the solution on Capacity Scheduler ?
On Wed, Nov 27, 2013 at 2:24 AM, Munna <[email protected]> wrote: > yes... acl's are not enforced. > > > On Wed, Nov 27, 2013 at 2:16 AM, Olivier Renault <[email protected] > > wrote: > >> Do you mean that acl are not being enforced? >> >> Olivier >> On 26 Nov 2013 20:18, "Munna" <[email protected]> wrote: >> >>> Hi Olivier, >>> >>> have u hold any solution, why user are not identifying their queues? >>> >>> Thanks >>> Munna >>> >>> >>> On Tue, Nov 26, 2013 at 11:17 PM, Munna <[email protected]> wrote: >>> >>>> *mapred queue -list* result: >>>> >>>> [user@host ~]$ mapred queue -list >>>> 13/11/26 09:37:48 INFO service.AbstractService: >>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >>>> 13/11/26 09:37:48 INFO service.AbstractService: >>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >>>> ====================== >>>> Queue Name : exploration >>>> Queue State : running >>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>> CurrentCapacity: 0.0 >>>> ====================== >>>> Queue Name : a >>>> Queue State : running >>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>> CurrentCapacity: 0.0 >>>> ====================== >>>> Queue Name : b >>>> Queue State : running >>>> Scheduling Info : Capacity: 30.000002, MaximumCapacity: 1.0, >>>> CurrentCapacity: 0.0 >>>> ====================== >>>> Queue Name : c >>>> Queue State : running >>>> Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, >>>> CurrentCapacity: 0.0 >>>> ====================== >>>> Queue Name : production >>>> Queue State : running >>>> Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, >>>> CurrentCapacity: 0.0 >>>> >>>> and I saw in RM logs, mapred user try to find default queue, logs are >>>> below: >>>> >>>> 2013-11-26 05:22:59,804 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application >>>> with id 1 submitted by user mapred >>>> 2013-11-26 05:22:59,808 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred >>>> IP=10.10.10.1 OPERATION=Submit Application Request >>>> TARGET=ClientRMService RESULT=SUCCESS >>>> APPID=application_1385472086605_0001 >>>> 2013-11-26 05:22:59,828 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: >>>> application_1385472086605_0001 State change from NEW to SUBMITTED >>>> 2013-11-26 05:22:59,829 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: >>>> Registering appattempt_1385472086605_0001_000001 >>>> 2013-11-26 05:22:59,830 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: >>>> appattempt_1385472086605_0001_000001 State change from NEW to SUBMITTED >>>> 2013-11-26 05:22:59,832 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: >>>> appattempt_1385472086605_0001_000001 State change from SUBMITTED to FAILED >>>> 2013-11-26 05:22:59,836 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: >>>> application_1385472086605_0001 State change from SUBMITTED to FAILED >>>> 2013-11-26 05:22:59,837 WARN >>>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapred >>>> OPERATION=Application Finished - Failed TARGET=RMAppManager >>>> RESULT=FAILURE DESCRIPTION=App failed with state: FAILED >>>> PERMISSIONS=Application appattempt_1385472086605_0001_000001 submitted by >>>> user mapred to unknown queue: default APPID=application_1385472086605_0001 >>>> 2013-11-26 05:22:59,841 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: >>>> Removing info for app: application_1385472086605_0001 >>>> 2013-11-26 05:22:59,842 INFO >>>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: >>>> appId=application_1385472086605_0001,name=select count(*) from >>>> crc_exchange_rate(Stage-1),user=mapred,queue=default,state=FAILED,trackingUrl=hostname:8088/proxy/application_1385472086605_0001/,appMasterHost=N/A,startTime=1385472179799,finishTime=1385472179836 >>>> >>>> >>>> >>>> On Tue, Nov 26, 2013 at 10:36 PM, Olivier Renault < >>>> [email protected]> wrote: >>>> >>>>> Could you confirm that the queue are running by: >>>>> mapred queue -list >>>>> >>>>> Otherwise what error do you get when submitting a job ? >>>>> >>>>> Thanks, >>>>> Olivier >>>>> >>>>> >>>>> On 26 November 2013 15:33, Munna <[email protected]> wrote: >>>>> >>>>>> Below configured queue's are showing, then why i cannt able to run >>>>>> the job. >>>>>> >>>>>> [root@host ~]# hadoop queue -showacls >>>>>> DEPRECATED: Use of this script to execute mapred command is >>>>>> deprecated. >>>>>> Instead use the mapred command for it. >>>>>> >>>>>> 13/11/26 05:25:43 INFO service.AbstractService: >>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. >>>>>> 13/11/26 05:25:44 INFO service.AbstractService: >>>>>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. >>>>>> Queue acls for user : root >>>>>> >>>>>> Queue Operations >>>>>> ===================== >>>>>> root ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> exploration ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> a ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> b ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> c ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> production ADMINISTER_QUEUE,SUBMIT_APPLICATIONS >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Nov 26, 2013 at 8:05 PM, Olivier Renault < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Sorry my mistake, it's : >>>>>>> mapred queue -showacls >>>>>>> >>>>>>> Or you can also use hadoop queue -showacls as suggested by Jitendra >>>>>>> >>>>>>> Thanks, >>>>>>> Olviier >>>>>>> >>>>>>> >>>>>>> On 26 November 2013 14:17, Munna <[email protected]> wrote: >>>>>>> >>>>>>>> Yes! I have executed same command: >>>>>>>> >>>>>>>> [user@host ~]$* yarn queue -showacls* >>>>>>>> >>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: queue >>>>>>>> Caused by: java.lang.ClassNotFoundException: queue >>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>> Method) >>>>>>>> at >>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>>>>>> at >>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>>>>>> Could not find the main class: queue. Program will exit. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Nov 26, 2013 at 7:25 PM, Olivier Renault < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> it should be : >>>>>>>>> yarn queue -showacls >>>>>>>>> >>>>>>>>> Olivier >>>>>>>>> >>>>>>>>> >>>>>>>>> On 26 November 2013 13:28, Munna <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Olivier, >>>>>>>>>> >>>>>>>>>> Thank you for your reply. >>>>>>>>>> >>>>>>>>>> As you said, i ran those commands and i am getting following >>>>>>>>>> error message on both the commands. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *[root@host ~]# sudo -u yarn yarn queue -showacls Exception in >>>>>>>>>> thread "main" java.lang.NoClassDefFoundError: queueCaused by: >>>>>>>>>> java.lang.ClassNotFoundException: queue at >>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) at >>>>>>>>>> java.security.AccessController.doPrivileged(Native Method) at >>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) at >>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:306) at >>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>>>>>> at >>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find >>>>>>>>>> the >>>>>>>>>> main class: queue. Program will exit.* >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Munna >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Nov 26, 2013 at 4:10 PM, Olivier Renault < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Could you maybe send us the output of : >>>>>>>>>>> - yarn queue -showacls >>>>>>>>>>> - yarn queue -list >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Olivier >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 26 November 2013 05:53, Munna <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I working with Capacity Scheduler on YARN and I have configured >>>>>>>>>>>> different queues. I can able to see all the queues on RM ui. But, >>>>>>>>>>>> when i >>>>>>>>>>>> start to run MR jobs with configured user names(yarn,mapred), i am >>>>>>>>>>>> unable >>>>>>>>>>>> to run the Jobs and job are suspended. Again i set default as FIFO >>>>>>>>>>>> working >>>>>>>>>>>> fine. >>>>>>>>>>>> >>>>>>>>>>>> Can you please help me out sort this issue and configured >>>>>>>>>>>> Configurations are given below. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *<property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.queues</name> >>>>>>>>>>>> <value>production,exploration</value> </property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.queues</name> >>>>>>>>>>>> <value>a,b,c</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.capacity</name> >>>>>>>>>>>> <value>100</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.production.capacity</name> >>>>>>>>>>>> <value>70</value></property> <property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.capacity</name> >>>>>>>>>>>> <value>30</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.capacity</name> >>>>>>>>>>>> <value>30</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.capacity</name> >>>>>>>>>>>> <value>30</value></property> <property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.capacity</name> >>>>>>>>>>>> <value>40</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root. >>>>>>>>>>>> production.acl_submit_applications</name> >>>>>>>>>>>> <value>yarn,mapred</value></property><property> >>>>>>>>>>>> <value>userb</value> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.a.acl_submit_applications</name> >>>>>>>>>>>> </property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.b.acl_submit_applications</name> >>>>>>>>>>>> <value>userb</value></property><property> >>>>>>>>>>>> <name>yarn.scheduler.capacity.root.exploration.c.acl_submit_applications</name> >>>>>>>>>>>> <value>userc</value></property></configuration>* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Regards* >>>>>>>>>>>> >>>>>>>>>>>> *Munna* >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>> If the reader of this message is not the intended recipient, you are >>>>> hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>> you have received this communication in error, please contact the sender >>>>> immediately and delete it from your system. Thank You. >>>>> >>>> >>>> >>>> >>>> -- >>>> *Regards* >>>> >>>> *Munna* >>>> >>> >>> >>> >>> -- >>> *Regards* >>> >>> *Munna* >>> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > > > -- > *Regards* > > *Munna* > -- *Regards* *Munna*
