[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304244#comment-16304244
]
Mukul Kumar Singh commented on HADOOP-15143:
I have raised HADOOP-15144 for failure of TestRaceWhenRelogin
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15144) TestRaceWhenRelogin fails because of incorrect order of tickets
Mukul Kumar Singh created HADOOP-15144:
--
Summary: TestRaceWhenRelogin fails because of incorrect order of
tickets
Key: HADOOP-15144
URL: https://issues.apache.org/jira/browse/HADOOP-15144
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Mukul Kumar Singh
TestRaceWhenRelogin fails because of incorrect order of tickets. The test fails
because of the following stack trace.
{code}
java.lang.AssertionError: tgt is not the first ticket after relogin
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.security.TestRaceWhenRelogin.test(TestRaceWhenRelogin.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304238#comment-16304238
]
Mukul Kumar Singh commented on HADOOP-15143:
Failure of TestRaceWhenRelogin is not related to the patch. The test fails
without the patch as well.
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15117) open(PathHandle) contract test should be exhaustive for default options
[
https://issues.apache.org/jira/browse/HADOOP-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304232#comment-16304232
]
genericqa commented on HADOOP-15117:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 21 new + 7 unchanged -
0 fixed = 28 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 14s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
30s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15117 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903749/HADOOP-15117.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a540065c2163 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64
[jira] [Updated] (HADOOP-15117) open(PathHandle) contract test should be exhaustive for default options
[
https://issues.apache.org/jira/browse/HADOOP-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-15117:
---
Assignee: Chris Douglas
Status: Patch Available (was: Open)
> open(PathHandle) contract test should be exhaustive for default options
> ---
>
> Key: HADOOP-15117
> URL: https://issues.apache.org/jira/browse/HADOOP-15117
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Chris Douglas
>Assignee: Chris Douglas
> Attachments: HADOOP-15117.000.patch
>
>
> The current {{AbstractContractOpenTest}} covers many, but not all of the
> permutations of the default {{HandleOpt}}. It could also be refactored to be
> clearer as documentation
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15117) open(PathHandle) contract test should be exhaustive for default options
[
https://issues.apache.org/jira/browse/HADOOP-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-15117:
---
Attachment: HADOOP-15117.000.patch
Added a new contract test to validate the {{PathHandle}} defaults
/cc [~elgoiri], [~ste...@apache.org]
> open(PathHandle) contract test should be exhaustive for default options
> ---
>
> Key: HADOOP-15117
> URL: https://issues.apache.org/jira/browse/HADOOP-15117
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Chris Douglas
> Attachments: HADOOP-15117.000.patch
>
>
> The current {{AbstractContractOpenTest}} covers many, but not all of the
> permutations of the default {{HandleOpt}}. It could also be refactored to be
> clearer as documentation
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304081#comment-16304081
]
Rushabh S Shah commented on HADOOP-14445:
-
Thanks [~jojochuang], [~yzhangal] and [~xiaochen] for the great discussion and
review comments on the first patch.
I have attached trunk and branch-2.8 version of the patch.
Except for checkstyle warnings, the patch is ready to review.
I thought the trunk and branch-2.8 patch would be identical but that is not the
case.
There is some difference in {{KMSClientProvider}} because of HADOOP-14987.
The change in HADOOP-14987 is only for debugging purpose and there is no
feature or major enhancement.
@Xiao: Since you are one of the contributor in that jira, we can backport that
jira all the way back to branch-2.8 if you don't mind.
I implemented what we discussed in previous comments.
To summarize let me write below.
* While instantiating {{KMSClientProvider}}, the token service will be
initialized to be the same as
{{CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH}}.
* {{KMSTokenRenewer}} will try to instantiate {{KeyProvider}} from token
service field.
If it fails to instantiate, it will fallback to the configuration.
I think that should take care of backward compatibility and key shell use cases.
Also added a test case to test backward compatibility for token renewal and
cancellation.
I have tried to incorporate all the previous review comments from everyone.
Please let me know if I missed one.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304070#comment-16304070
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-common-project: The patch generated 8 new
+ 122 unchanged - 4 fixed = 130 total (was 126) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 14s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903738/HADOOP-14445.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 49f3f4226fe0 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 52babbb |
| maven |
[jira] [Resolved] (HADOOP-14134) KMSTokenRenewer should renew token from the service address in token.
[ https://issues.apache.org/jira/browse/HADOOP-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah resolved HADOOP-14134. - Resolution: Duplicate > KMSTokenRenewer should renew token from the service address in token. > - > > Key: HADOOP-14134 > URL: https://issues.apache.org/jira/browse/HADOOP-14134 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > > Currently KMSTokenRenewer renews token from the address mentioned in the conf. > Ideally it should go to the service address mentioned in the token. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14445:
Attachment: HADOOP-14445.002.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303950#comment-16303950
]
genericqa commented on HADOOP-15143:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15143 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903715/HADOOP-15143.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 68d54bb9c28f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 52babbb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13888/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13888/testReport/ |
| Max. process+thread count | 1717 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303848#comment-16303848
]
Mukul Kumar Singh commented on HADOOP-15143:
This will happen when a destroyed ticket is being renewed by
checkTGTAndReloginFromKeytab.
We need to add a check similar to the following jdk check introduced as part of
JDK-8163103.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-15143:
---
Status: Patch Available (was: Open)
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-15143:
---
Attachment: HADOOP-15143.001.patch
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303790#comment-16303790
]
Yonger commented on HADOOP-14943:
-
[~ste...@apache.org] As Hadoop running on s3-compatible storage, compute and
storage separately, so I called the host that running MR/Spark/Hive application
"compute host".
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15142) Register FTP and SFTP as FS services
[ https://issues.apache.org/jira/browse/HADOOP-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303725#comment-16303725 ] Mario Molina commented on HADOOP-15142: --- This build error has nothing to do with that patch applied. In fact, it's an error related to a file not found (debug.log) within the test. > Register FTP and SFTP as FS services > > > Key: HADOOP-15142 > URL: https://issues.apache.org/jira/browse/HADOOP-15142 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.0.0 >Reporter: Mario Molina >Priority: Minor > Fix For: 3.0.1 > > Attachments: HADOOP-15142.001.patch > > > SFTPFileSystem and FTPFileSystem are not registered as a FS services. > When calling the 'get' or 'newInstance' methods of the FileSystem class, the > FS instance cannot be created due to the schema is not registered as a > service FS. > Also, the SFTPFileSystem class doesn't have the getScheme method implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14178) Move Mockito up to version 2.x
[ https://issues.apache.org/jira/browse/HADOOP-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303643#comment-16303643 ] Akira Ajisaka commented on HADOOP-14178: bq. The patch becomes bigger and bigger. I'd like to split the patch for each modules for reviewers to review the patches easily. This is not simple. Mockito 1.x and 2.x use the same package name, so we need to exclude the dependency appropriately and this it is very hard work. > Move Mockito up to version 2.x > -- > > Key: HADOOP-14178 > URL: https://issues.apache.org/jira/browse/HADOOP-14178 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Akira Ajisaka > Attachments: HADOOP-14178.001.patch, HADOOP-14178.002.patch, > HADOOP-14178.003.patch, HADOOP-14178.004.patch, HADOOP-14178.005-wip.patch, > HADOOP-14178.005-wip2.patch, HADOOP-14178.005-wip3.patch, > HADOOP-14178.005-wip4.patch, HADOOP-14178.005.patch > > > I don't know when Hadoop picked up Mockito, but it has been frozen at 1.8.5 > since the switch to maven in 2011. > Mockito is now at version 2.1, [with lots of Java 8 > support|https://github.com/mockito/mockito/wiki/What%27s-new-in-Mockito-2]. > That' s not just defining actions as closures, but in supporting Optional > types, mocking methods in interfaces, etc. > It's only used for testing, and, *provided there aren't regressions*, cost of > upgrade is low. The good news: test tools usually come with good test > coverage. The bad: mockito does go deep into java bytecodes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
