Lets change it in both places. Please file issues. Lets try minimize the freakout incidents running your hbase/hadoop cluster. Thanks Takeshi, St.Ack
On Thu, Jan 16, 2014 at 9:57 PM, takeshi <[email protected]> wrote: > Hi All, > > Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" > while we are using the HBase client API to put data into HBase-0.94.16 with > krb5-1.6.1 enabled. The related msg as follows... > {code} > [2014-01-15 > > 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)): > PriviledgedActionException > as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS > initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Request is a > replay (34) - PROCESS_TGS)] > [2014-01-15 > > 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)): > PriviledgedAction > as:takeshi_miao@LABfrom > :sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > [2014-01-15 > > 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)): > Exception encountered while connecting to the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Request is a > replay (34) - PROCESS_TGS)] > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO > > ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)): > Initiating logout for takeshi_miao@LAB > [2014-01-15 > > 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)): > hadoop logout > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO > > ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)): > Initiating re-login for takeshi_miao@LAB > [2014-01-15 > > 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)): > hadoop login > [2014-01-15 > > 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)): > hadoop login commit > [2014-01-15 > > 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)): > using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao, > UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]: > 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB > ] > {code} > > At the beginning, we were worry about the "data loss" occurring while we > found the "Request is a replay (34) - PROCESS_TGS" (especially it is the > ERROR level) in log, but after code study, this is basically *NOT* a data > loss issue due to HBase client API would try 5 times internally > (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of > retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all > failed thread), The HTable API would also throw IOEcxeption to client code > if any thread still fail after these retries. > > Based on HBase users' viewpoint as us, we think this is better to change > the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been > confused us for a while...But this code change may need to change boht > HBase code and Hadoop code as well; so I am wondering how community think > about this small thing but may be important to the pure HBase users. > > P.S. more details are in this article > > http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html > > Tks > > Best regards > > takeshi >
