slider.zookeeper.quorum is deprecated and should not be used. hadoop.registry.zk.quorum is used instead and is typically defined in yarn-site.xml. So is hadoop.registry.zk.root.
It is not encouraged to specify slider.yarn.queue at the cluster config level. Ideally it is best to specify the queue during the application submission. So you can use --queue option with slider create cmd. You can also set on the command line using -D slider.yarn.queue=<> during the create call. If indeed all slider apps should go to one and only one queue, then this prop can be specified in any one of the existing site xml files under /etc/hadoop/conf. -Gour On 7/28/16, 4:43 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >Following slider specific properties are at present added in >/data/slider/conf/slider-client.xml. If you think they should be picked up >from HADOOP_CONF_DIR (/etc/hadoop/conf) file, which file in >HADOOP_CONF_DIR >should these be added ? > > - slider.zookeeper.quorum > - hadoop.registry.zk.quorum > - hadoop.registry.zk.root > - slider.yarn.queue > > >On Thu, Jul 28, 2016 at 4:37 PM, Gour Saha <gs...@hortonworks.com> wrote: > >> That is strange, since it is indeed not required to contain anything in >> slider-client.xml (except <configuration></configuration>) if >> HADOOP_CONF_DIR has everything that Slider needs. This probably gives an >> indication that there might be some issue with cluster configuration >>based >> on files solely under HADOOP_CONF_DIR to begin with. >> >> Suggest you to upload all the config files to the jira to help debug >>this >> further. >> >> -Gour >> >> On 7/28/16, 4:27 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >> >> >Thanks Gour for prompt reply >> > >> >BTW - Creating a empty slider-client.xml (with just >> ><configuration></configuration>) does not works. The AM starts but >>fails >> >to >> >create any components and shows errors like >> > >> >2016-07-28 23:18:46,018 >> >[AmExecutor-006-SendThread(localhost.localdomain:2181)] WARN >> > zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, >> >closing socket connection and attempting reconnect >> >java.net.ConnectException: Connection refused >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> > at >> >sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) >> > at >> >>>org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO >>>.j >> >ava:361) >> > at >> >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) >> > >> >Also, command "slider destroy <app>" fails with zookeeper errors ... >> > >> >I had to keep a minimal slider-client.xml. It does not have any RM info >> >etc. but does contain slider ZK related properties like >> >"slider.zookeeper.quorum", "hadoop.registry.zk.quorum", >> >"hadoop.registry.zk.root". I haven't yet distilled the absolute minimal >> >set >> >of properties required, but this should suffice for now. All RM / HDFS >> >properties will be read from HADOOP_CONF_DIR files. >> > >> >Let me know if this could cause any issues. >> > >> >On Thu, Jul 28, 2016 at 3:36 PM, Gour Saha <gs...@hortonworks.com> >>wrote: >> > >> >> No need to copy any files. Pointing HADOOP_CONF_DIR to >>/etc/hadoop/conf >> >>is >> >> good. >> >> >> >> -Gour >> >> >> >> On 7/28/16, 3:24 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >> >> >> >> >Follow up question regarding Gour's comment in earlier thread - >> >> > >> >> >Slider is installed on one of the hadoop nodes. SLIDER_HOME/conf >> >>directory >> >> >(say /data/slider/conf) is different than HADOOP_CONF_DIR >> >> >(/etc/hadoop/conf). Is it required/recommended that files in >> >> >HADOOP_CONF_DIR be copied to SLIDER_HOME/conf and slider-env.sh >>script >> >> >sets >> >> >HADOOP_CONF_DIR to /data/slider/conf ? >> >> > >> >> >Or can the slider-env.sh set HADOOP_CONF_DIR to /etc/hadoop/conf , >> >>without >> >> >copying the files ? >> >> > >> >> >Using slider .80 for now, but would like to know recommendation for >> >>this >> >> >and future versions as well. >> >> > >> >> >Thanks in advance, >> >> > >> >> >Manoj >> >> > >> >> >On Tue, Jul 26, 2016 at 3:27 PM, Manoj Samel >><manojsamelt...@gmail.com >> > >> >> >wrote: >> >> > >> >> >> Filed https://issues.apache.org/jira/browse/SLIDER-1158 with logs >> and >> >> my >> >> >> analysis of logs. >> >> >> >> >> >> On Tue, Jul 26, 2016 at 10:36 AM, Gour Saha >><gs...@hortonworks.com> >> >> >>wrote: >> >> >> >> >> >>> Please file a JIRA and upload the logs to it. >> >> >>> >> >> >>> On 7/26/16, 10:21 AM, "Manoj Samel" <manojsamelt...@gmail.com> >> >>wrote: >> >> >>> >> >> >>> >Hi Gour, >> >> >>> > >> >> >>> >Can you please reach me using your own email-id? I will then >>send >> >> >>>logs to >> >> >>> >you, along with my analysis - I don't want to send logs on >>public >> >>list >> >> >>> > >> >> >>> >Thanks, >> >> >>> > >> >> >>> >On Mon, Jul 25, 2016 at 5:39 PM, Gour Saha >><gs...@hortonworks.com> >> >> >>> wrote: >> >> >>> > >> >> >>> >> Ok, so this node is not a gateway. It is part of the cluster, >> >>which >> >> >>> >>means >> >> >>> >> you don¹t need slider-client.xml at all. Just have >> >>HADOOP_CONF_DIR >> >> >>> >> pointing to /etc/hadoop/conf in slider-env.sh and that should >>be >> >>it. >> >> >>> >> >> >> >>> >> So the above simplifies your config setup. It will not solve >> >>either >> >> >>>of >> >> >>> >>the >> >> >>> >> 2 problems you are facing. >> >> >>> >> >> >> >>> >> Now coming to the 2 issues you are facing, you have to provide >> >> >>> >>additional >> >> >>> >> logs for us to understand better. Let¹s start with - >> >> >>> >> 1. RM logs (specifically between the time when rm1->rm2 >>failover >> >>is >> >> >>> >> simulated) >> >> >>> >> 2. Slider App logs >> >> >>> >> >> >> >>> >> -Gour >> >> >>> >> >> >> >>> >> On 7/25/16, 5:16 PM, "Manoj Samel" <manojsamelt...@gmail.com> >> >> wrote: >> >> >>> >> >> >> >>> >> > 1. Not clear about your question on "gateway" node. The >>node >> >> >>> running >> >> >>> >> > slider is part of the hadoop cluster and there are other >> >> >>>services >> >> >>> >>like >> >> >>> >> > Oozie that run on this node that utilizes hdfs and yarn. >>So >> >>if >> >> >>>your >> >> >>> >> > question is whether the node is otherwise working for HDFS >> >>and >> >> >>>Yarn >> >> >>> >> > configuration, it is working >> >> >>> >> > 2. I copied all files from HADOOP_CONF_DIR (say >> >> >>>/etc/hadoop/conf) >> >> >>> to >> >> >>> >> >the >> >> >>> >> > directory containing slider-client.xml (say >> >>/data/latest/conf) >> >> >>> >> > 3. In earlier email, I had done a mistake where >>slider-env.sh >> >> >>>file >> >> >>> >> >HADOOP_CONF_DIR >> >> >>> >> > was pointing to original directory /etc/hadoop/conf. I >>edited >> >> >>>it to >> >> >>> >> > point to same directory containing slider-client.xml & >> >> >>> slider-env.sh >> >> >>> >> >i.e. >> >> >>> >> > /data/latest/conf >> >> >>> >> > 4. I emptied slider-client.xml. It just had the >> >> >>> >> ><configuration></configuration>. >> >> >>> >> > The creation of spas worked but the Slider AM still shows >>the >> >> >>>same >> >> >>> >> >issue. >> >> >>> >> > i.e. when RM1 goes from active to standby, slider AM goes >> >>from >> >> >>> >>RUNNING >> >> >>> >> >to >> >> >>> >> > ACCPTED state with same error about TOKEN. Also NOTE that >> >>when >> >> >>> >> > slider-client.xml is empty, the "slider destroy xxx" >>command >> >> >>>still >> >> >>> >> >fails >> >> >>> >> > with Zookeeper connection errors. >> >> >>> >> > 5. I then added same parameters (as my last email - except >> >> >>> >> > HADOOP_CONF_DIR) to slider-client.xml and ran. This time >> >> >>> >>slider-env.sh >> >> >>> >> > has HADOOP_CONF_DIR pointing to /data/latest/conf and >> >> >>> >>slider-client.xml >> >> >>> >> > does not have HADOOP_CONF_DIR. The same issue exists (but >> >> >>>"slider >> >> >>> >> > destroy" does not fails) >> >> >>> >> > 6. Could you explain what do you expect to pick up from >> >>Hadoop >> >> >>> >> > configurations that will help you in RM Token ? If slider >>has >> >> >>>token >> >> >>> >> >from >> >> >>> >> > RM1, and it switches to RM2, not clear what slider does to >> >>get >> >> >>> >> >delegation >> >> >>> >> > token for RM2 communication ? >> >> >>> >> > 7. It is worth repeating again that issue happens only >>when >> >>RM1 >> >> >>>was >> >> >>> >> > active when slider app was created and then RM1 becomes >> >> >>>standby. If >> >> >>> >> >RM2 was >> >> >>> >> > active when slider app was created, then slider AM keeps >> >>running >> >> >>> for >> >> >>> >> >any >> >> >>> >> > number of switches between RM2 and RM1 back and forth ... >> >> >>> >> > >> >> >>> >> > >> >> >>> >> >On Mon, Jul 25, 2016 at 4:21 PM, Gour Saha >> >><gs...@hortonworks.com> >> >> >>> >>wrote: >> >> >>> >> > >> >> >>> >> >> The node you are running slider from, is that a gateway >>node? >> >> >>>Sorry >> >> >>> >>for >> >> >>> >> >> not being explicit. I meant copy everything under >> >> >>>/etc/hadoop/conf >> >> >>> >>from >> >> >>> >> >> your cluster into some temp directory (say >>/tmp/hadoop_conf) >> >>in >> >> >>>your >> >> >>> >> >> gateway node or local or whichever node you are running >>slider >> >> >>>from. >> >> >>> >> >>Then >> >> >>> >> >> set HADOOP_CONF_DIR to /tmp/hadoop_conf and clear >>everything >> >>out >> >> >>> from >> >> >>> >> >> slider-client.xml. >> >> >>> >> >> >> >> >>> >> >> On 7/25/16, 4:12 PM, "Manoj Samel" >><manojsamelt...@gmail.com> >> >> >>> wrote: >> >> >>> >> >> >> >> >>> >> >> >Hi Gour, >> >> >>> >> >> > >> >> >>> >> >> >Thanks for your prompt reply. >> >> >>> >> >> > >> >> >>> >> >> >FYI, issue happens when I create slider app when rm1 is >> >>active >> >> >>>and >> >> >>> >>when >> >> >>> >> >> >rm1 >> >> >>> >> >> >fails over to rm2. As soon as rm2 becomes active; the >>slider >> >>AM >> >> >>> goes >> >> >>> >> >>from >> >> >>> >> >> >RUNNING to ACCEPTED state with above error. >> >> >>> >> >> > >> >> >>> >> >> >For your suggestion, I did following >> >> >>> >> >> > >> >> >>> >> >> >1) Copied core-site, hdfs-site, yarn-site, and mapred-site >> >>from >> >> >>> >> >> >HADOOP_CONF_DIR >> >> >>> >> >> >to slider conf directory. >> >> >>> >> >> >2) Our slider-env.sh already had HADOOP_CONF_DIR set >> >> >>> >> >> >3) I removed all properties from slider-client.xml EXCEPT >> >> >>>following >> >> >>> >> >> > >> >> >>> >> >> > - HADOOP_CONF_DIR >> >> >>> >> >> > - slider.yarn.queue >> >> >>> >> >> > - slider.zookeeper.quorum >> >> >>> >> >> > - hadoop.registry.zk.quorum >> >> >>> >> >> > - hadoop.registry.zk.root >> >> >>> >> >> > - hadoop.security.authorization >> >> >>> >> >> > - hadoop.security.authentication >> >> >>> >> >> > >> >> >>> >> >> >Then I made rm1 active, installed and created slider app >>and >> >> >>> >>restarted >> >> >>> >> >>rm1 >> >> >>> >> >> >(to make rm2) active. The slider-am again went from >>RUNNING >> >>to >> >> >>> >>ACCEPTED >> >> >>> >> >> >state. >> >> >>> >> >> > >> >> >>> >> >> >Let me know if you want me to try further changes. >> >> >>> >> >> > >> >> >>> >> >> >If I make the slider-client.xml completely empty per your >> >> >>> >>suggestion, >> >> >>> >> >>only >> >> >>> >> >> >slider AM comes up but it >> >> >>> >> >> >fails to start components. The AM log shows errors trying >>to >> >> >>> >>connect to >> >> >>> >> >> >zookeeper like below. >> >> >>> >> >> >2016-07-25 23:07:41,532 >> >> >>> >> >> >[AmExecutor-006-SendThread(localhost.localdomain:2181)] >>WARN >> >> >>> >> >> >zookeeper.ClientCnxn - Session 0x0 for server null, >> >>unexpected >> >> >>> >>error, >> >> >>> >> >> >closing socket connection and attempting reconnect >> >> >>> >> >> >java.net.ConnectException: Connection refused >> >> >>> >> >> > >> >> >>> >> >> >Hence I kept minimal info in slider-client.xml >> >> >>> >> >> > >> >> >>> >> >> >FYI This is slider version 0.80 >> >> >>> >> >> > >> >> >>> >> >> >Thanks, >> >> >>> >> >> > >> >> >>> >> >> >Manoj >> >> >>> >> >> > >> >> >>> >> >> >On Mon, Jul 25, 2016 at 2:54 PM, Gour Saha >> >> >>><gs...@hortonworks.com> >> >> >>> >> >>wrote: >> >> >>> >> >> > >> >> >>> >> >> >> If possible, can you copy the entire content of the >> >>directory >> >> >>> >> >> >> /etc/hadoop/conf and then set HADOOP_CONF_DIR in >> >> >>>slider-env.sh to >> >> >>> >>it. >> >> >>> >> >> >>Keep >> >> >>> >> >> >> slider-client.xml empty. >> >> >>> >> >> >> >> >> >>> >> >> >> Now when you do the same rm1->rm2 and then the reverse >> >> >>>failovers, >> >> >>> >>do >> >> >>> >> >>you >> >> >>> >> >> >> see the same behaviors? >> >> >>> >> >> >> >> >> >>> >> >> >> -Gour >> >> >>> >> >> >> >> >> >>> >> >> >> On 7/25/16, 2:28 PM, "Manoj Samel" >> >><manojsamelt...@gmail.com> >> >> >>> >>wrote: >> >> >>> >> >> >> >> >> >>> >> >> >> >Another observation (whatever it is worth) >> >> >>> >> >> >> > >> >> >>> >> >> >> >If slider app is created and started when rm2 was >>active, >> >> >>>then >> >> >>> it >> >> >>> >> >> >>seems to >> >> >>> >> >> >> >survive switches between rm2 and rm1 (and back). I.e >> >> >>> >> >> >> > >> >> >>> >> >> >> >* rm2 is active >> >> >>> >> >> >> >* create and start slider application >> >> >>> >> >> >> >* fail over to rm1. Now the Slider AM keeps running >> >> >>> >> >> >> >* fail over to rm2 again. Slider AM still keeps running >> >> >>> >> >> >> > >> >> >>> >> >> >> >So, it seems if it starts with rm1 active, then the AM >> >>goes >> >> >>>to >> >> >>> >> >> >>"ACCEPTED" >> >> >>> >> >> >> >state when RM fails to rm2. If it starts with rm2 >>active, >> >> >>>then >> >> >>> it >> >> >>> >> >>runs >> >> >>> >> >> >> >fine >> >> >>> >> >> >> >with any switches between rm1 and rm2. >> >> >>> >> >> >> > >> >> >>> >> >> >> >Any feedback ? >> >> >>> >> >> >> > >> >> >>> >> >> >> >Thanks, >> >> >>> >> >> >> > >> >> >>> >> >> >> >Manoj >> >> >>> >> >> >> > >> >> >>> >> >> >> >On Mon, Jul 25, 2016 at 12:25 PM, Manoj Samel >> >> >>> >> >> >><manojsamelt...@gmail.com> >> >> >>> >> >> >> >wrote: >> >> >>> >> >> >> > >> >> >>> >> >> >> >> Setup >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> - Hadoop 2.6 with RM HA, Kerberos enabled >> >> >>> >> >> >> >> - Slider 0.80 >> >> >>> >> >> >> >> - In my slider-client.xml, I have added all RM HA >> >> >>>properties, >> >> >>> >> >> >>including >> >> >>> >> >> >> >> the ones mentioned in >> >> >>> >> >>http://markmail.org/message/wnhpp2zn6ixo65e3. >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> Following is the issue >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> * rm1 is active, rm2 is standby >> >> >>> >> >> >> >> * deploy and start slider application, it runs fine >> >> >>> >> >> >> >> * restart rm1, rm2 is now active. >> >> >>> >> >> >> >> * The slider-am now goes from running into "ACCEPTED" >> >> >>>mode. It >> >> >>> >> >>stays >> >> >>> >> >> >> >>there >> >> >>> >> >> >> >> till rm1 is made active again. >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> In the slider-am log, it tries to connect to RM2 and >> >> >>> connection >> >> >>> >> >>fails >> >> >>> >> >> >> >>due >> >> >>> >> >> >> >> to org.apache.hadoop.security.AccessControlException: >> >> >>>Client >> >> >>> >> >>cannot >> >> >>> >> >> >> >> authenticate via:[TOKEN]. See detailed log below >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> It seems it has some token (delegation token?) for >>RM1 >> >>but >> >> >>> >>tries >> >> >>> >> >>to >> >> >>> >> >> >>use >> >> >>> >> >> >> >> same(?) for RM2 and fails. Am I missing some >> >>configuration >> >> >>>??? >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> Thanks, >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] >>INFO >> >> >>> >> >> >> >> client.ConfiguredRMFailoverProxyProvider - Failing >> >>over to >> >> >>> rm2 >> >> >>> >> >> >> >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] >>WARN >> >> >>> >> >> >> >> security.UserGroupInformation - >> >>PriviledgedActionException >> >> >>> >> >> >>as:abc@XYZ >> >> >>> >> >> >> >> (auth:KERBEROS) >> >> >>> >> >> >>cause:org.apache.hadoop.security.AccessControlException: >> >> >>> >> >> >> >> Client cannot authenticate via:[TOKEN] >> >> >>> >> >> >> >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] >>WARN >> >> >>> >> >>ipc.Client - >> >> >>> >> >> >> >> Exception encountered while connecting to the server >>: >> >> >>> >> >> >> >> org.apache.hadoop.security.AccessControlException: >> >>Client >> >> >>> >>cannot >> >> >>> >> >> >> >> authenticate via:[TOKEN] >> >> >>> >> >> >> >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] >>WARN >> >> >>> >> >> >> >> security.UserGroupInformation - >> >>PriviledgedActionException >> >> >>> >> >> >>as:abc@XYZ >> >> >>> >> >> >> >> (auth:KERBEROS) cause:java.io.IOException: >> >> >>> >> >> >> >> org.apache.hadoop.security.AccessControlException: >> >>Client >> >> >>> >>cannot >> >> >>> >> >> >> >> authenticate via:[TOKEN] >> >> >>> >> >> >> >> 2016-07-25 19:06:48,089 [AMRM Heartbeater thread] >>INFO >> >> >>> >> >> >> >> retry.RetryInvocationHandler - Exception while >>invoking >> >> >>> >>allocate >> >> >>> >> >>of >> >> >>> >> >> >> >>class >> >> >>> >> >> >> >> ApplicationMasterProtocolPBClientImpl over rm2 after >>287 >> >> >>>fail >> >> >>> >>over >> >> >>> >> >> >> >> attempts. Trying to fail over immediately. >> >> >>> >> >> >> >> java.io.IOException: Failed on local exception: >> >> >>> >> >>java.io.IOException: >> >> >>> >> >> >> >> org.apache.hadoop.security.AccessControlException: >> >>Client >> >> >>> >>cannot >> >> >>> >> >> >> >> authenticate via:[TOKEN]; Host Details : local host >>is: >> >> >>> >>"<SliderAM >> >> >>> >> >> >> >> HOST>/<slider AM Host IP>"; destination host is: >>"<RM2 >> >> >>> >> >>HOST>":23130; >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >>>>>org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) >> >> >>> >> >> >> >> at >> >> >>>org.apache.hadoop.ipc.Client.call(Client.java:1476) >> >> >>> >> >> >> >> at >> >> >>>org.apache.hadoop.ipc.Client.call(Client.java:1403) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Proto >>>>>>>>>>>>>>>bu >> >>>>>>>>>>>>>fR >> >> >>>>>>>>>>>pcE >> >> >>> >>>>>>>>ng >> >> >>> >> >>>>>>in >> >> >>> >> >> >>>>e. >> >> >>> >> >> >> >>java:230) >> >> >>> >> >> >> >> at com.sun.proxy.$Proxy23.allocate(Unknown >> >>Source) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterP >>>>>>>>>>>>>>>ro >> >>>>>>>>>>>>>to >> >> >>>>>>>>>>>col >> >> >>> >>>>>>>>PB >> >> >>> >> >>>>>>Cl >> >> >>> >> >> >>>>ie >> >> >>> >> >> >> >> >> >>>>>ntImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) >> >> >>> >> >> >> >> at >> >> >>> sun.reflect.GeneratedMethodAccessor10.invoke(Unknown >> >> >>> >> >> >>Source) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>>>>>>>>>>>>>>th >> >>>>>>>>>>>>>od >> >> >>>>>>>>>>>Acc >> >> >>> >>>>>>>>es >> >> >>> >> >>>>>>so >> >> >>> >> >> >>>>rI >> >> >>> >> >> >> >>mpl.java:43) >> >> >>> >> >> >> >> at >> >>java.lang.reflect.Method.invoke(Method.java:497) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >>>>>>>>>>>>>>>od >> >>>>>>>>>>>>>(R >> >> >>>>>>>>>>>etr >> >> >>> >>>>>>>>yI >> >> >>> >> >>>>>>nv >> >> >>> >> >> >>>>oc >> >> >>> >> >> >> >>ationHandler.java:252) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >>>>>>>>>>>>>>>ry >> >>>>>>>>>>>>>In >> >> >>>>>>>>>>>voc >> >> >>> >>>>>>>>at >> >> >>> >> >>>>>>io >> >> >>> >> >> >>>>nH >> >> >>> >> >> >> >>andler.java:104) >> >> >>> >> >> >> >> at com.sun.proxy.$Proxy24.allocate(Unknown >> >>Source) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.alloca >>>>>>>>>>>>>>>te >> >>>>>>>>>>>>>(A >> >> >>>>>>>>>>>MRM >> >> >>> >>>>>>>>Cl >> >> >>> >> >>>>>>ie >> >> >>> >> >> >>>>nt >> >> >>> >> >> >> >>Impl.java:278) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsync >>>>>>>>>>>>>>>Im >> >>>>>>>>>>>>>pl >> >> >>>>>>>>>>>$He >> >> >>> >>>>>>>>ar >> >> >>> >> >>>>>>tb >> >> >>> >> >> >>>>ea >> >> >>> >> >> >> >>tThread.run(AMRMClientAsyncImpl.java:224) >> >> >>> >> >> >> >> Caused by: java.io.IOException: >> >> >>> >> >> >> >> org.apache.hadoop.security.AccessControlException: >> >>Client >> >> >>> >>cannot >> >> >>> >> >> >> >> authenticate via:[TOKEN] >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >>>>>org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682) >> >> >>> >> >> >> >> at >> >> >>>java.security.AccessController.doPrivileged(Native >> >> >>> >> >>Method) >> >> >>> >> >> >> >> at >> >> >>>javax.security.auth.Subject.doAs(Subject.java:422) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>>>>>>>>>>>>>>up >> >>>>>>>>>>>>>In >> >> >>>>>>>>>>>for >> >> >>> >>>>>>>>ma >> >> >>> >> >>>>>>ti >> >> >>> >> >> >>>>on >> >> >>> >> >> >> >>.java:1671) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.ipc.Client$Connection.handleSaslConnection >>>>>>>>>>>>>>>Fa >> >>>>>>>>>>>>>il >> >> >>>>>>>>>>>ure >> >> >>> >>>>>>>>(C >> >> >>> >> >>>>>>li >> >> >>> >> >> >>>>en >> >> >>> >> >> >> >>t.java:645) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client. >>>>>>>>>>>>>ja >> >>>>>>>>>>>va >> >> >>>>>>>>>:73 >> >> >>> >>>>>>3) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >>>>>>>>>org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:37 >>>>>>>>>0) >> >> >>> >> >> >> >> at >> >> >>> >> >> >>>>org.apache.hadoop.ipc.Client.getConnection(Client.java:1525) >> >> >>> >> >> >> >> at >> >> >>>org.apache.hadoop.ipc.Client.call(Client.java:1442) >> >> >>> >> >> >> >> ... 12 more >> >> >>> >> >> >> >> Caused by: >> >> >>>org.apache.hadoop.security.AccessControlException: >> >> >>> >> >>Client >> >> >>> >> >> >> >> cannot authenticate via:[TOKEN] >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.security.SaslRpcClient.selectSaslClient(Sa >>>>>>>>>>>>>>>sl >> >>>>>>>>>>>>>Rp >> >> >>>>>>>>>>>cCl >> >> >>> >>>>>>>>ie >> >> >>> >> >>>>>>nt >> >> >>> >> >> >>>>.j >> >> >>> >> >> >> >>ava:172) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpc >>>>>>>>>>>>>>>Cl >> >>>>>>>>>>>>>ie >> >> >>>>>>>>>>>nt. >> >> >>> >>>>>>>>ja >> >> >>> >> >>>>>>va >> >> >>> >> >> >>>>:3 >> >> >>> >> >> >> >>96) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.ipc.Client$Connection.setupSaslConnection( >>>>>>>>>>>>>>>Cl >> >>>>>>>>>>>>>ie >> >> >>>>>>>>>>>nt. >> >> >>> >>>>>>>>ja >> >> >>> >> >>>>>>va >> >> >>> >> >> >>>>:5 >> >> >>> >> >> >> >>55) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >>>>>>>>>org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:37 >>>>>>>>>0) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >>>>>org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >>>>>org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721) >> >> >>> >> >> >> >> at >> >> >>>java.security.AccessController.doPrivileged(Native >> >> >>> >> >>Method) >> >> >>> >> >> >> >> at >> >> >>>javax.security.auth.Subject.doAs(Subject.java:422) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>>>>>>>>>>>>>>up >> >>>>>>>>>>>>>In >> >> >>>>>>>>>>>for >> >> >>> >>>>>>>>ma >> >> >>> >> >>>>>>ti >> >> >>> >> >> >>>>on >> >> >>> >> >> >> >>.java:1671) >> >> >>> >> >> >> >> at >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >>>>>>>>>>>>>org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client. >>>>>>>>>>>>>ja >> >>>>>>>>>>>va >> >> >>>>>>>>>:72 >> >> >>> >>>>>>0) >> >> >>> >> >> >> >> ... 15 more >> >> >>> >> >> >> >> 2016-07-25 19:06:48,089 [AMRM Heartbeater thread] >>INFO >> >> >>> >> >> >> >> client.ConfiguredRMFailoverProxyProvider - Failing >> >>over to >> >> >>> rm1 >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >> >> >> >> >> >>
