[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161610#comment-14161610 ] Hudson commented on HBASE-10411: SUCCESS: Integrated in HBase-1.0 #280 (See [https://builds.apache.org/job/HBase-1.0/280/]) HBASE-10411 Add a kerberos 'request is a replay (34)' issue in the Troubleshooting chapter (Takeshi Miao) (mstanleyjones: rev 24cb52be3713ce183a69b5e83c0fad076c7a8639) * src/main/docbkx/troubleshooting.xml > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Fix For: 2.0.0, 0.99.1 > > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161483#comment-14161483 ] Hudson commented on HBASE-10411: SUCCESS: Integrated in HBase-TRUNK #5626 (See [https://builds.apache.org/job/HBase-TRUNK/5626/]) HBASE-10411 Add a kerberos 'request is a replay (34)' issue in the Troubleshooting chapter (Takeshi Miao) (mstanleyjones: rev b9701d05690acc505580ea9dee167d62986a84e3) * src/main/docbkx/troubleshooting.xml > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Fix For: 2.0.0, 0.99.1 > > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912542#comment-13912542 ] takeshi.miao commented on HBASE-10411: -- Hi [~apurtell] How do you think about this patch ? > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903078#comment-13903078 ] takeshi.miao commented on HBASE-10411: -- [~xieliang007] :) > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902985#comment-13902985 ] Liang Xie commented on HBASE-10411: --- np:) it's just fine, i put here just in case other guys who hit the similar issue could search for refer. Go ahead, good job! > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902980#comment-13902980 ] takeshi.miao commented on HBASE-10411: -- Hi [~xieliang007] I am considering still not put your info. into the docs due to we did not suffer this kind of issue you mentioned, we can put these info. into docs if some other guys would still suffer the it after upgrading to new version of krb5-server, in the future. How do you think ? > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902969#comment-13902969 ] Liang Xie commented on HBASE-10411: --- changing hrb5-server side was not done by me, was our ops guys, probably should be setting the env variable per document directly. > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902927#comment-13902927 ] takeshi.miao commented on HBASE-10411: -- [~xieliang007] Just for curious, how did you disable the replay caching for your krb5-server ? Due to we did search for this property but not found any useful info. tks~ > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902920#comment-13902920 ] Liang Xie commented on HBASE-10411: --- yep, when we used the modified KrbApReq class via "-Xbootclasspath/p" parameter, the "request is a replay" warning gone away at all(our krb5-server side disabled the replay caching already). just for your refer:) > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902919#comment-13902919 ] Liang Xie commented on HBASE-10411: --- yep, when we used the modified KrbApReq class via "-Xbootclasspath/p" parameter, the "request is a replay" warning gone away at all(our krb5-server side disabled the replay caching already). just for your refer:) > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[
https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902871#comment-13902871
]
takeshi.miao commented on HBASE-10411:
--
[~xieliang007]
This is a surprising news for me !! Did you ever reproduce it in your env. ?
due to we eliminate this issue with following combination of JDK6 and
krb5-server versions
{code}
[ec2-user@aws-scottm-hbase-1 opt]$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
[ec2-user@aws-scottm-hbase-1 opt]$ rpm -qa | egrep krb5
pam_krb5-2.3.11-9.el6.x86_64
krb5-server-1.10.3-10.el6_4.6.x86_64
krb5-libs-1.10.3-10.el6_4.6.x86_64
krb5-workstation-1.10.3-10.el6_4.6.x86_64
{code}
I never try JDK7 before due to it still not in our scope for PROD usage.
> [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting
> section
> -
>
> Key: HBASE-10411
> URL: https://issues.apache.org/jira/browse/HBASE-10411
> Project: HBase
> Issue Type: Improvement
> Components: documentation, security
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
> Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt
>
>
> For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the
> troubleshooting section in HBase book
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[ https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897618#comment-13897618 ] Liang Xie commented on HBASE-10411: --- Hmm, seems my above said is only for JDK6, just found JDK-7085018 and JDK-6882687... > [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting > section > - > > Key: HBASE-10411 > URL: https://issues.apache.org/jira/browse/HBASE-10411 > Project: HBase > Issue Type: Improvement > Components: documentation, security >Reporter: takeshi.miao >Assignee: takeshi.miao >Priority: Minor > Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt > > > For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the > troubleshooting section in HBase book -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[
https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897597#comment-13897597
]
Liang Xie commented on HBASE-10411:
---
Disable replay caching in krb5-server side seems not enough per my
understanding, since there's another replay checking mechanism in JDK side as
well, see "sun/security/krb5/KrbApReq.java" from jdk source.
One possible solution is to impl another sun.security.krb5.KrbApReq, then build
a jar. With the "-Xbootclasspath/p" parameter, we can utilize the modified
KrbApReq Class(The modification is delete the table get/put related operation)
over the original class from rt.jar :)
Another thing need to know about "-Xbootclasspath" is:
{code}
Note: Applications that use this option for the purpose of overriding a class
in rt.jar should not be deployed as doing so would contravene the Java 2
Runtime Environment binary code license.
{code}
Another more elegant solution is to make cache table checking configurable in
KrbApReq Class, I'll throw a trivial patch to OpenJDK community.
> [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting
> section
> -
>
> Key: HBASE-10411
> URL: https://issues.apache.org/jira/browse/HBASE-10411
> Project: HBase
> Issue Type: Improvement
> Components: documentation, security
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
> Attachments: HBASE-10411-trunk-v01.patch, HBASE-10411-v01.odt
>
>
> For kerberos 'request is a replay (34)' issue (HBASE-10379), adding it to the
> troubleshooting section in HBase book
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10411) [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting section
[
https://issues.apache.org/jira/browse/HBASE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891964#comment-13891964
]
Hadoop QA commented on HBASE-10411:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12627091/HBASE-10411-trunk-v01.patch
against trunk revision .
ATTACHMENT ID: 12627091
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Secure Client Connect ([Caused by GSSException: No
valid credentials provided (Mechanism level: Request is a replay (34) V
PROCESS_TGS)])
+This issue is caused by the Kerberos bugs for its replay_cache component,
http://krbdev.mit.edu/rt/Ticket/Display.html?id=1201";>#1201
and http://krbdev.mit.edu/rt/Ticket/Display.html?id=5924";>#5924,
whom were talking about that the old version of krb5-server would
false-positively blocks subsequent requests sent from a Principal.
+In this case, which means that the krb5-server would sometimes block the
connections sent from one Client (one HTable instance with multi-threading
connection instances for each regionserver); You would see such message
'Request is a replay (34)' in your client log if you hit this bug. You can just
ignore this message due to the implementation of HTable will retry 5 * 10 for
each failed connection by default. The HTable will throw IOException if any
connection to regionserver was failed after the retries, therefore, the user
client code for HTable instance still can handle it further.
+Otherwise, you can pick a new version of krb5-server to solve this issue
gracefully. We tested and mitigated this issue in the environment:
hbase-0.94.16 with krb5-server-1.10.3 on RHEL-6.4_GA-x86_64-10-Hourly2 on AWS,
compared to the old environment: HBase-0.94.16 with krb5-server-1.6.1 on
CentOS-5.3. Please refer to JIRA https://issues.apache.org/jira/browse/HBASE-10379";>HBASE-10379
for more details.
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8603//console
This message is automatically generated.
> [Book] Add a kerberos 'request is a replay (34)' issue at troubleshooting
> section
> -
>
> Key: HBASE-10411
> URL: https://issues.apache.org/jira/browse/HBASE-10411
> Project: HBase
> Issue Type: Improvement
>
