Hi Stack, https://issues.apache.org/jira/browse/HBASE-10379 ticket created, tks for your response~
Best regards takeshi 2014/1/18 Stack <[email protected]> > Lets change it in both places. Please file issues. Lets try minimize the > freakout incidents running your hbase/hadoop cluster. > Thanks Takeshi, > St.Ack > > > On Thu, Jan 16, 2014 at 9:57 PM, takeshi <[email protected]> wrote: > > > Hi All, > > > > Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" > > while we are using the HBase client API to put data into HBase-0.94.16 > with > > krb5-1.6.1 enabled. The related msg as follows... > > {code} > > [2014-01-15 > > > > > 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)): > > PriviledgedActionException > > as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS > > initiate failed [Caused by > > GSSException: No valid credentials provided (Mechanism level: Request is > a > > replay (34) - PROCESS_TGS)] > > [2014-01-15 > > > > > 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)): > > PriviledgedAction > > as:takeshi_miao@LABfrom > > :sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > [2014-01-15 > > > > > 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)): > > Exception encountered while connecting to the server : > > javax.security.sasl.SaslException: GSS initiate failed [Caused by > > GSSException: No valid credentials provided (Mechanism level: Request is > a > > replay (34) - PROCESS_TGS)] > > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO > > > > > ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)): > > Initiating logout for takeshi_miao@LAB > > [2014-01-15 > > > > > 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)): > > hadoop logout > > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO > > > > > ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)): > > Initiating re-login for takeshi_miao@LAB > > [2014-01-15 > > > > > 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)): > > hadoop login > > [2014-01-15 > > > > > 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)): > > hadoop login commit > > [2014-01-15 > > > > > 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)): > > using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao, > > UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]: > > 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, > takeshi_miao@LAB > > ] > > {code} > > > > At the beginning, we were worry about the "data loss" occurring while we > > found the "Request is a replay (34) - PROCESS_TGS" (especially it is the > > ERROR level) in log, but after code study, this is basically *NOT* a data > > loss issue due to HBase client API would try 5 times internally > > (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of > > retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all > > failed thread), The HTable API would also throw IOEcxeption to client > code > > if any thread still fail after these retries. > > > > Based on HBase users' viewpoint as us, we think this is better to change > > the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had > been > > confused us for a while...But this code change may need to change boht > > HBase code and Hadoop code as well; so I am wondering how community think > > about this small thing but may be important to the pure HBase users. > > > > P.S. more details are in this article > > > > > http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html > > > > Tks > > > > Best regards > > > > takeshi > > >
