[jira] [Commented] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648249#comment-16648249
]
Rushabh S Shah commented on HADOOP-14445:
-
{quote}I have committed this to trunk.
{quote}
Great to see this Jira finally resolved. Thanks [~xiaochen] for staying on top
of this. :)
Thanks [~daryn] for throwing all ideas and then rejecting after. :P
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508483#comment-16508483
]
Rushabh S Shah commented on HADOOP-14445:
-
Thanks [~xiaochen] for revised patch. I will review this week.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15459:
Resolution: Invalid
Status: Resolved (was: Patch Available)
Gave a little bit more thought to this jira.
This change will bring semantic change to how acls are handled today.
Due to security reasons, if an admin granted access to one opType per key acl,
(s)he has to grant access to all opTypes.
If an admin makes a typo in one opType, after this change it will fallback to
default for that opType which is not desirable.
So closing this ticket as Invalid.
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HADOOP-15459.001.patch, HADOOP-15459.002.patch
>
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479201#comment-16479201
]
Rushabh S Shah commented on HADOOP-15459:
-
Not fixing the checkstyle warning to maintain readability.
[~jojochuang], [~xiaochen], [~RANith]: can someone help with reviewing this
patch ?
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HADOOP-15459.001.patch, HADOOP-15459.002.patch
>
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15459:
Attachment: HADOOP-15459.002.patch
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HADOOP-15459.001.patch, HADOOP-15459.002.patch
>
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15459:
Attachment: HADOOP-15459.001.patch
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HADOOP-15459.001.patch
>
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15459:
Status: Patch Available (was: Open)
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HADOOP-15459.001.patch
>
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah reassigned HADOOP-15459:
---
Assignee: Rushabh S Shah
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477483#comment-16477483
]
Rushabh S Shah commented on HADOOP-15459:
-
bq. Rushabh S Shah No problem.
Thank you!
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Critical
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477469#comment-16477469
]
Rushabh S Shah commented on HADOOP-15459:
-
Hi [~RANith], We have an internal release which is blocked on this jira.
Do you mind if I provide a patch to this jira ?
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Ranith Sardar
>Priority: Critical
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473728#comment-16473728
]
Rushabh S Shah edited comment on HADOOP-15459 at 5/14/18 3:42 PM:
--
Below is the relevant piece of code which IMO is buggy.
{code:title=Bar.java|borderStyle=solid}
private boolean checkKeyAccess(String keyName, UserGroupInformation ugi,
KeyOpType opType) {
Map keyAcl = keyAcls.get(keyName);
if (keyAcl == null) {// This should be if(keyAcl == null ||
keyAcl.get(opType) == null)
// If No key acl defined for this key, check to see if
// there are key defaults configured for this operation
LOG.debug("Key: {} has no ACLs defined, using defaults.", keyName);
keyAcl = defaultKeyAcls;
}
boolean access = checkKeyAccess(keyAcl, ugi, opType);
...
{code}
Instead of key'ing just on keyname, it should consider opType also.
was (Author: shahrs87):
Below is the relevant piece of code which IMO is buggy.
{code:title=Bar.java|borderStyle=solid}
private boolean checkKeyAccess(String keyName, UserGroupInformation ugi,
KeyOpType opType) {
Map keyAcl = keyAcls.get(keyName);
if (keyAcl == null) {// This should be if(keyAcl == null ||
keyAcl.get(opType) == null)
// If No key acl defined for this key, check to see if
// there are key defaults configured for this operation
LOG.debug("Key: {} has no ACLs defined, using defaults.", keyName);
keyAcl = defaultKeyAcls;
}
boolean access = checkKeyAccess(keyAcl, ugi, opType);
...
{code}
Instead of key'ing just on keyname, it should also consider opType also.
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Ranith Sardar
>Priority: Critical
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
[
https://issues.apache.org/jira/browse/HADOOP-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473728#comment-16473728
]
Rushabh S Shah commented on HADOOP-15459:
-
Below is the relevant piece of code which IMO is buggy.
{code:title=Bar.java|borderStyle=solid}
private boolean checkKeyAccess(String keyName, UserGroupInformation ugi,
KeyOpType opType) {
Map keyAcl = keyAcls.get(keyName);
if (keyAcl == null) {// This should be if(keyAcl == null ||
keyAcl.get(opType) == null)
// If No key acl defined for this key, check to see if
// there are key defaults configured for this operation
LOG.debug("Key: {} has no ACLs defined, using defaults.", keyName);
keyAcl = defaultKeyAcls;
}
boolean access = checkKeyAccess(keyAcl, ugi, opType);
...
{code}
Instead of key'ing just on keyname, it should also consider opType also.
> KMSACLs will fail for other optype if acls is defined for one optype.
> -
>
> Key: HADOOP-15459
> URL: https://issues.apache.org/jira/browse/HADOOP-15459
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Critical
>
> Assume subset of kms-acls xml file.
> {noformat}
>
> default.key.acl.DECRYPT_EEK
>
>
> default ACL for DECRYPT_EEK operations for all key acls that are not
> explicitly defined.
>
>
>
>
> key.acl.key1.DECRYPT_EEK
> user1
>
>
> default.key.acl.READ
> *
>
> default ACL for READ operations for all key acls that are not
> explicitly defined.
>
>
>
> whitelist.key.acl.READ
> hdfs
>
> Whitelist ACL for READ operations for all keys.
>
>
> {noformat}
> For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
> For other {{READ}} operation(like getMetadata), by default I still want
> everyone to access all keys via {{default.key.acl.READ}}
> But it doesn't allow anyone to access {{key1}} for any other READ operations.
> As a result of this, if the admin restricted access for one opType then
> (s)he has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15459) KMSACLs will fail for other optype if acls is defined for one optype.
Rushabh S Shah created HADOOP-15459:
---
Summary: KMSACLs will fail for other optype if acls is defined for
one optype.
Key: HADOOP-15459
URL: https://issues.apache.org/jira/browse/HADOOP-15459
Project: Hadoop Common
Issue Type: Bug
Components: kms
Affects Versions: 2.8.3
Reporter: Rushabh S Shah
Assume subset of kms-acls xml file.
{noformat}
default.key.acl.DECRYPT_EEK
default ACL for DECRYPT_EEK operations for all key acls that are not
explicitly defined.
key.acl.key1.DECRYPT_EEK
user1
default.key.acl.READ
*
default ACL for READ operations for all key acls that are not
explicitly defined.
whitelist.key.acl.READ
hdfs
Whitelist ACL for READ operations for all keys.
{noformat}
For key {{key1}}, we restricted {{DECRYPT_EEK}} operation to only {{user1}}.
For other {{READ}} operation(like getMetadata), by default I still want
everyone to access all keys via {{default.key.acl.READ}}
But it doesn't allow anyone to access {{key1}} for any other READ operations.
As a result of this, if the admin restricted access for one opType then (s)he
has to define access for all other opTypes also, which is not desirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15441) Log kms url and token service at debug level.
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15441:
Fix Version/s: 3.2.0
> Log kms url and token service at debug level.
> -
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.8.5
>
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch,
> HADOOP-15441.003.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15441) Log kms url and token service at debug level.
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15441:
Resolution: Fixed
Fix Version/s: 2.8.5
3.0.3
2.9.2
3.1.1
2.10.0
Status: Resolved (was: Patch Available)
Thanks [~gabor.bota] for the contribution.
Committed to trunk all the way back to branch-2.8
> Log kms url and token service at debug level.
> -
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 2.10.0, 3.1.1, 2.9.2, 3.0.3, 2.8.5
>
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch,
> HADOOP-15441.003.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15441) Log kms url and token service at debug level.
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15441:
Summary: Log kms url and token service at debug level. (was: After
HADOOP-14987, encryption zone operations print unnecessary INFO logs)
> Log kms url and token service at debug level.
> -
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch,
> HADOOP-15441.003.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15441) After HADOOP-14987, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472048#comment-16472048
]
Rushabh S Shah commented on HADOOP-15441:
-
[~gabor.bota]: The patch doesn't apply cleanly after revert of hadoop-14445.
Can you please rebase the patch.
> After HADOOP-14987, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15441) After HADOOP-14987, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471012#comment-16471012
]
Rushabh S Shah commented on HADOOP-15441:
-
{quote}Since log message is parametrized it will not be evaluated if logging
level is not DEBUG.
{quote}
Thanks [~ajayydv] !
> After HADOOP-14987, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15441) After HADOOP-14987, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470599#comment-16470599
]
Rushabh S Shah commented on HADOOP-15441:
-
Sorry for not committing yesterday.
bq. Since we are using slf4j, we don't really need if(LOG.isDebugEnabled()).
[~xyao]: can you please elaborate why {{isDebugEnabled}} check is not enabled ?
> After HADOOP-14987, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch
>
>
> It looks like after HADOOP-14987, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15441) After HADOOP-14445, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468974#comment-16468974
]
Rushabh S Shah edited comment on HADOOP-15441 at 5/9/18 3:41 PM:
-
Thanks [~gabor.bota] for the updated patch.
{quote}Since we are using slf4j, we don't really need
{{if(LOG.isDebugEnabled())}}.
{quote}
Thanks [~xyao] for pointing this out.
I am happy with v1 of the patch. If everyone is ok, I will go ahead and commit
v1 of patch today evening.
Gabor: can you please update the description and title of the jira.
was (Author: shahrs87):
Thanks [~gabor.bota] for the updated patch.
{quote}Since we are using slf4j, we don't really need
{{if(LOG.isDebugEnabled())}}.
{quote}
Thanks [~xyao] for pointing this out.
I am happy with v1 of the patch. If everyone is ok, I will go ahead and commit
v1 of patch today evening.
Gabor: can you please update the description and title of the patch.
> After HADOOP-14445, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch
>
>
> It looks like after HADOOP-14445, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15441) After HADOOP-14445, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468974#comment-16468974
]
Rushabh S Shah commented on HADOOP-15441:
-
Thanks [~gabor.bota] for the updated patch.
{quote}Since we are using slf4j, we don't really need
{{if(LOG.isDebugEnabled())}}.
{quote}
Thanks [~xyao] for pointing this out.
I am happy with v1 of the patch. If everyone is ok, I will go ahead and commit
v1 of patch today evening.
Gabor: can you please update the description and title of the patch.
> After HADOOP-14445, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch, HADOOP-15441.002.patch
>
>
> It looks like after HADOOP-14445, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14841:
Resolution: Fixed
Fix Version/s: 3.0.3
2.9.2
2.8.4
3.1.0
Status: Resolved (was: Patch Available)
Thanks [~daryn], [~xiaochen] and [~jojochuang] for the review.
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 3.1.0, 2.8.4, 2.9.2, 3.0.3
>
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch, HADOOP-14841.05.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
>
[jira] [Commented] (HADOOP-15441) After HADOOP-14445, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463987#comment-16463987
]
Rushabh S Shah commented on HADOOP-15441:
-
Thanks [~gabor.bota] for the patch.
One minor nit.
Use the pattern.
{noformat}
if (LOG.isDebugEnabled()) {
LOG.debug("Response={}({}), resetting authToken",
conn.getResponseCode(), conn.getResponseMessage());
}
{noformat}
> After HADOOP-14445, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15441.001.patch
>
>
> It looks like after HADOOP-14445, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462944#comment-16462944
]
Rushabh S Shah commented on HADOOP-14841:
-
{quote}
I'd suggest to explicitly say "server HTTP 500 error". And we can probably drop
"// Refer to HADOOP-14841 for more details." because git blame would easily
show it.
{quote}
Done in v5 of patch.
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch, HADOOP-14841.05.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
>
[jira] [Updated] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14841:
Attachment: HADOOP-14841.05.patch
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch, HADOOP-14841.05.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
>
[jira] [Comment Edited] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462900#comment-16462900
]
Rushabh S Shah edited comment on HADOOP-14841 at 5/3/18 6:30 PM:
-
{quote}+1 pending the comment change.
{quote}
Thanks [~daryn], [~xiaochen], [~jojochuang] for the review and comment.
Updated the patch with generic comment. Also updated the title.
Daryn, Xiao: Even if the revised patch has just the comment change, will wait
for either of yours final +1 on v4 of patch before committing.
Also same patch applies fine to all branches until branch-2.8
was (Author: shahrs87):
bq. +1 pending the comment change.
Thanks [~daryn], [~xiaochen], [~jojochuang] for the review and comment.
Updated the patch with generic comment. Also updated the title.
Daryn, Xiao: Even if the revised patch has just the comment change, will wait
for either of yours final +1 on v4 of patch.
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
>
[jira] [Commented] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462900#comment-16462900
]
Rushabh S Shah commented on HADOOP-14841:
-
bq. +1 pending the comment change.
Thanks [~daryn], [~xiaochen], [~jojochuang] for the review and comment.
Updated the patch with generic comment. Also updated the title.
Xiao: Even if the revised patch has just the comment change, will wait for your
final +1 on v4 of patch.
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
>
[jira] [Comment Edited] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462900#comment-16462900
]
Rushabh S Shah edited comment on HADOOP-14841 at 5/3/18 6:29 PM:
-
bq. +1 pending the comment change.
Thanks [~daryn], [~xiaochen], [~jojochuang] for the review and comment.
Updated the patch with generic comment. Also updated the title.
Daryn, Xiao: Even if the revised patch has just the comment change, will wait
for either of yours final +1 on v4 of patch.
was (Author: shahrs87):
bq. +1 pending the comment change.
Thanks [~daryn], [~xiaochen], [~jojochuang] for the review and comment.
Updated the patch with generic comment. Also updated the title.
Xiao: Even if the revised patch has just the comment change, will wait for your
final +1 on v4 of patch.
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
>
[jira] [Updated] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14841:
Attachment: HADOOP-14841.04.patch
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch, HADOOP-14841.04.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:579)
> at
[jira] [Updated] (HADOOP-14841) Kms client should disconnect if unable to get output stream from connection.
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14841:
Summary: Kms client should disconnect if unable to get output stream from
connection. (was: Let KMS Client retry 'No content to map' EOFExceptions)
> Kms client should disconnect if unable to get output stream from connection.
>
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
>
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462875#comment-16462875
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. Fine as in "won't NPE"
It won't throw NPE.
bq. "tested spark can get the new token once the current one reaches 75% of its
expiration time"?
IIUC spark is going through all the tokens in the credentials cache which will
contain {{KMS_D_T}} and {{kms-dt}}.
Spark is reading {{identifier.getIssueDate}} from the token and both tokens
will have the same {{issueDate}} so we can safely _not implement_
{{KMSLegacyDelegationTokenIdentifier}}.
As long as spark doesn't throw NPE, I think we are good here.
Hope it makes sense.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch,
> HADOOP-15408.trunk.poc.patch, split.patch, split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462669#comment-16462669
]
Rushabh S Shah commented on HADOOP-14841:
-
bq. Besides Daryn's comment comment,
I will update the patch once we are in agreement about what to do with
connection object once it goes in a bad state.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
>
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462666#comment-16462666
]
Rushabh S Shah commented on HADOOP-14841:
-
{quote}based on testing from the stress program?
{quote}
Yes the stress program recovered after applying this patch and we didn't see
any exceptions on the server side.
bq. but I think the happy path didn't disconnect,
Yes, the happy path didn't disconnect.
But if the client is not able to {{getOutputStream}} from connection, then the
connection is in an inconsistent state.
I don't see any valid reason keeping that connection alive. I am also not an
expert on HttpUrlConnection. So any more feedbacks/discussions are welcome.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
>
[jira] [Commented] (HADOOP-15441) After HADOOP-14445, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462637#comment-16462637
]
Rushabh S Shah commented on HADOOP-15441:
-
bq. Also we have other logging in KMSTokenRenewer which is filling disk on our
test cluster.
Turns out that this excessive logging is fixed by HADOOP-15390
> After HADOOP-14445, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> It looks like after HADOOP-14445, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15441) After HADOOP-14445, encryption zone operations print unnecessary INFO logs
[
https://issues.apache.org/jira/browse/HADOOP-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462560#comment-16462560
]
Rushabh S Shah commented on HADOOP-15441:
-
Also we have other logging in KMSTokenRenewer which is filling disk on our test
cluster.
We need to make them debug too.
Will post a patch soon.
> After HADOOP-14445, encryption zone operations print unnecessary INFO logs
> --
>
> Key: HADOOP-15441
> URL: https://issues.apache.org/jira/browse/HADOOP-15441
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> It looks like after HADOOP-14445, any encryption zone operations prints extra
> INFO log messages as follows:
> {code:java}
> $ hdfs dfs -copyFromLocal /etc/krb5.conf /scale/
> 18/05/02 11:54:55 INFO kms.KMSClientProvider: KMSClientProvider for KMS url:
> https://hadoop3-1.example.com:16000/kms/v1/ delegation token service:
> kms://ht...@hadoop3-1.example.com:16000/kms created.
> {code}
> It might make sense to make it a DEBUG message instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15431) KMSTokenRenewer should work with KMS_DELEGATION_TOKEN which has ip:port as service
[
https://issues.apache.org/jira/browse/HADOOP-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461519#comment-16461519
]
Rushabh S Shah commented on HADOOP-15431:
-
bq. Could you share some details?
What Daryn meant is: Internally we have whole provideruri in kms token service
since we started using EZ.
After HADOOP-14445, kms server started issuing token with kind
"KMS_DELEGATION_TOKEN" but we internally changed it back to "kms-dt" because we
have no motivation to go to "KMS_D_T" kind. Also we have that config for
copying Legacy Token also switched off.
So we have to rebase our code everytime any change is made in
KMSClientProvider's token area.
> KMSTokenRenewer should work with KMS_DELEGATION_TOKEN which has ip:port as
> service
> --
>
> Key: HADOOP-15431
> URL: https://issues.apache.org/jira/browse/HADOOP-15431
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-15431.01.patch, HADOOP-15431.02.patch
>
>
> Seen a test failure where a MR job failed to submit.
> RM log has:
> {noformat}
> 2018-04-30 15:00:17,864 WARN
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Unable to add the application to the delegation token renewer.
> java.lang.IllegalArgumentException: Invalid token service IP_ADDR:16000
> at
> org.apache.hadoop.util.KMSUtil.createKeyProviderFromTokenService(KMSUtil.java:237)
> at
> org.apache.hadoop.crypto.key.kms.KMSTokenRenewer.createKeyProvider(KMSTokenRenewer.java:100)
> at
> org.apache.hadoop.crypto.key.kms.KMSTokenRenewer.renew(KMSTokenRenewer.java:57)
> at org.apache.hadoop.security.token.Token.renew(Token.java:414)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:590)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:587)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:585)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:894)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:871)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> while client log has
> {noformat}
> 18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1525128478242_0001
> 18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
> Service: ha-hdfs:ns1, Ident: (token for systest: HDFS_DELEGATION_TOKEN
> owner=syst...@example.com, renewer=yarn, realUser=, issueDate=1525128807236,
> maxDate=1525733607236, sequenceNumber=1038, masterKeyId=20)
> 18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: HBASE_AUTH_TOKEN,
> Service: 621a942b-292f-493d-ba50-f9b783704359, Ident:
> (org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@0)
> 18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: KMS_DELEGATION_TOKEN,
> Service: IP_ADDR:16000, Ident: 00 07 73 79 73 74 65 73 74 04 79 61 72 6e 00
> 8a 01 63 18 c2 c3 d5 8a 01 63 3c cf 47 d5 8e 01 ec 10
> 18/04/30 15:53:29 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/systest/.staging/job_1525128478242_0001
> 18/04/30 15:53:29 WARN security.UserGroupInformation:
> PriviledgedActionException as:syst...@example.com (auth:KERBEROS)
> cause:java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException:
> Failed to submit application_1525128478242_0001 to YARN : Invalid token
> service IP_ADDR:16000
> 18/04/30 15:53:29 INFO client.ConnectionManager$HConnectionImplementation:
> Closing master protocol: MasterService
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461426#comment-16461426
]
Rushabh S Shah commented on HADOOP-14841:
-
bq.KMS client side retry login should help with this scenario, I guess?
In this case, server will throw 500 Internal server error and client will not
retry on that exception.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
>
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461216#comment-16461216
]
Rushabh S Shah commented on HADOOP-15408:
-
{quote}
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala#L59
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L353
{quote}
It turns out that in scala {{null.instanceof(Object)}} returns false.
So I think {{decodeIdentifier}} returning null should be fine.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch,
> HADOOP-15408.trunk.poc.patch, split.patch, split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461000#comment-16461000
]
Rushabh S Shah commented on HADOOP-14841:
-
[~xiaochen]: Not trying to hijack your jira. please don't feel that way.
Uploading a simple patch. Lets see what hadoopqa thinks.
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
>
[jira] [Updated] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-14841:
Attachment: HADOOP-14841.03.patch
> Let KMS Client retry 'No content to map' EOFExceptions
> --
>
> Key: HADOOP-14841
> URL: https://issues.apache.org/jira/browse/HADOOP-14841
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14841.01.patch, HADOOP-14841.02.patch,
> HADOOP-14841.03.patch
>
>
> We have seen quite some occurrences when the KMS server is stressed, some of
> the requests would end up getting a 500 return code, with this in the server
> log:
> {noformat}
> 2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> User impala/HOSTNAME@REALM (auth:KERBEROS) request POST
> https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
> caused exception.
> java.io.EOFException: No content to map to Object due to end of input
> at
> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
> at
> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
> at
> com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
> at
> com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:579)
> at
>
[jira] [Commented] (HADOOP-14841) Let KMS Client retry 'No content to map' EOFExceptions
[
https://issues.apache.org/jira/browse/HADOOP-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460968#comment-16460968
]
Rushabh S Shah commented on HADOOP-14841:
-
We (Daryn and I) were able to reproduce the issue and found the root cause also.
Here is the test program that we used to reproduce.
{code:title=TestStressKMS.java|borderStyle=solid}
public class TestStressKMS extends Thread {
public static final Log LOG = LogFactory.getLog(TestStressKMS.class);
public void run() {
final Path path = new Path("/EZ1/test.pig");
final Configuration conf = new Configuration();
try {
FileSystem fileSystem = path.getFileSystem(conf);
InputStream inStream = fileSystem.open(path)
inStream.read();
} catch (Throwable ioe) {
System.out.println(ioe);
}
}
public static void main(String[] args) {
int numThreads = Integer.parseInt(args[0]);
Thread[] threads = new Thread[numThreads];
for(int i = 0; i < numThreads; i++) {
threads[i] = new TestStressKMS();
threads[i].setName("TestStressKMS " + i);
System.out.println(threads[i].getName());
threads[i].start();
}
try {
for(int i = 0; i < numThreads; i++) {
threads[i].join();
}
} catch (InterruptedException ie) {
}
}
}
{code}
We ran the above progeam with 128 threads and multiple such processes from
different nodes.
+What we saw on server:+
After few minutes, the server logged following exception.
{noformat}
2018-04-27 03:25:26,321 WARN KMSExceptionsProvider - User:'hadoopqa
(auth:KERBEROS)' Method:POST
URL:http://:4443/kms/v1/keyversion/key1%400/_eek?eek_op=decrypt
Response:Internal Server Error-No content to map to Object due to end of input
java.io.EOFException: No content to map to Object due to end of input
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2775)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
at
org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:57)
at
org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:35)
at
com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
at
com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
at
com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDi
spatchProvider.java:153)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDisp
atchProvider.java:203)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at
org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
at
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459975#comment-16459975
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. I want this fixed asap too,
We have fixed it internally by removing {{KMSLegacyTokenIdentifier}} class and
requested one of our spark devs to verify whether their jobs are running
successfully.
I got a +1 from him. We believe we are running the latest version of spark
which has that change.
I have again requested him to add logging in that specific area and verifies
that scala is not somehow filtering nulls.
Will post the result later today.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458911#comment-16458911
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. In other words, even though the bytes are the same for both kinds,
Token#decodeIdentifier() is broken by patch 1,
Trying to understand why we want to decode TokenIdentifier ?
{quote}
In other words, even though the bytes are the same for both kinds,
Token#decodeIdentifier() is broken by patch 1, because service loader cannot
find a class to decode kms-dt.
{quote}
I assume you are saying this because of the stack trace that I pasted in
description.
It failed in our build because we had {{KMSLegacyDelegationTokenIdentifier}} in
{{hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier}}
file and service loader was unable to find
{{KMSLegacyDelegationTokenIdentifier}}.
But after removing {{KMSLegacyDelegationTokenIdentifier}} from
{{hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier}},
the job is able to run successfully.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458803#comment-16458803
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. so I assumed that was the st you saw...
Yes.
bq. patch 1 fixed the spark issue
Yes.
Just for clarification, patch 1 = HADOOP-15408-trunk.001.patch
bq. patch 1 does not work with kms-dt
It does work with kms-dt. Just for reference, internally we only use {{kms-dt}}
since we have uri in the token service since the beginning of EZ.
bq. split.patch does not fix the spark issue
I haven't tested internally with {{split.prelim.patch}}
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458731#comment-16458731
]
Rushabh S Shah commented on HADOOP-15408:
-
{quote}What exception did you see with this case?
{quote}
I didn't see any exceptions with this case.
{quote}Just going back to old approach.
{quote}
Sorry I wasn't clear with my previous comment. Let me be more clear this time.
We internally fixed this issue with {{HADOOP-15408-trunk.001.patch}} and the
spark jobs ran successfully after this fix.
Going back to [this
comment|https://issues.apache.org/jira/browse/HADOOP-15408?focusedCommentId=16452594=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16452594],
I was asking that where did you see this stack trace ?
I assume you applied {{HADOOP-15408-trunk.001.patch}} patch and you
encountered some failure which showed that stack trace.
Let me know if still I am not clear.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455478#comment-16455478
]
Rushabh S Shah commented on HADOOP-15408:
-
{quote}
java.util.ServiceConfigurationError:
org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
KMSLegacyDelegationTokenIdentifier could not be instantiated
{quote}
Just going back to old approach. Now if I see the stack trace again, how did
the application know about {{KMSLegacyDelegationTokenIdentifier}}.
It seems like you forgot to remove
{{org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier}}
from
{{hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier}}.
I have put {{HADOOP-15408-trunk.001.patch}} internally and will update tomorrow.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[
https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454183#comment-16454183
]
Rushabh S Shah commented on HADOOP-15414:
-
bq. IF on HDFS Federation, all delegation tokens for NameNode can collection
correctly BUT delegation token for KMS not collect reference
You need to debug why the job client was not able to collect kms delegation
token.
You need to have kms delegation token to talk to kms to decrypt edeks.
> Job submit not work well on HDFS Federation with Transparent Encryption
> feature
> ---
>
> Key: HADOOP-15414
> URL: https://issues.apache.org/jira/browse/HADOOP-15414
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: He Xiaoqiao
>Priority: Major
>
> When submit sample MapReduce job WordCount which read/write path under
> encryption zone on HDFS Federation in security mode to YARN, task throws
> exception as below:
> {code:java}
> 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0,
> Status : FAILED
> Error: java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793)
> at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
> ... 21 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
>
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452992#comment-16452992
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. Would you be able to test whether the split approach helps on the failing
spark job?
I am planning to put this patch in my internal build tonight and will request
spark team to test it.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452744#comment-16452744
]
Rushabh S Shah commented on HADOOP-15408:
-
{quote}And because the new jar will meet kms-dt (e.g. rolling upgrade), we need
to fix it.
{quote}
My bad. Ignore my patch then.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452551#comment-16452551
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. the new jar alone won't be able to decode kms-dt?
We do we want to decode in the first place ?
We just want TokenRenewer to renew the token and the kind field in Token class
will be able to find the right TokenRenewer (i.e KMSLegacyTokenRenewer).
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452519#comment-16452519
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. Thanks for testing. The patch was to express the idea - seems it won't
compile on trunk.
Thanks [~xiaochen] for the patch. I should have been more clear. I didn't test
with your patch (split.patch).
{quote}
Identifier for both the tokens (i.e KMS_DELEGATION_TOKEN and kms-dt) are the
same (byte-to-byte) so we don't need to have another class
KMSLegacyDelegationTokenIdentifier for legacy token identifier.
{quote}
I have a different idea. I am uploading the patch (HADOOP-15408-001.patch) .
Please don't think I am intruding.
Lets see if we agree on one approach.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15408:
Attachment: HADOOP-15408-trunk.001.patch
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: HADOOP-15408-trunk.001.patch, split.patch,
> split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451504#comment-16451504
]
Rushabh S Shah edited comment on HADOOP-15408 at 4/25/18 1:44 AM:
--
Thanks [~xiaochen] for the patch.
Had an offline chat with [~daryn] on the proposed fix.
Below is the summary.
Identifier for both the tokens (i.e KMS_DELEGATION_TOKEN and kms-dt) are the
same (byte-to-byte) so we don't need to have another class
{{KMSLegacyDelegationTokenIdentifier}} for legacy token identifier.
Kind in Identifier doesn't mean much.
After removing {{KMSLegacyDelegationTokenIdentifier}} class and
{{KMSLegacyDelegationTokenIdentifier}} from
{{org.apache.hadoop.security.token.TokenIdentifier}}, 4 tests are failing in
TestKMS because they are trying to decodeIdentifier with kind {{kms-dt}} and
Serviceloader is not able to find any Identifier which corresponds to
{{kms-dt}} kind.
Since it is test-only code, we can change the test.
Let me know if this makes sense.
was (Author: shahrs87):
Thanks [~xiaochen] for the patch.
Had an offline chat with [~daryn] on the proposed fix.
Identifier for both the tokens (i.e KMS_DELEGATION_TOKEN and kms-dt) are the
same (byte-to-byte) so we don't need to have another class
{{KMSLegacyDelegationTokenIdentifier}} for legacy token identifier.
Kind in Identifier doesn't mean much.
After removing {{KMSLegacyDelegationTokenIdentifier}} class and
{{KMSLegacyDelegationTokenIdentifier}} from
{{org.apache.hadoop.security.token.TokenIdentifier}}, 4 tests are failing in
TestKMS because they are trying to decodeIdentifier with kind {{kms-dt}} and
Serviceloader is not able to find any Identifier which corresponds to
{{kms-dt}} kind.
Since it is test-only code, we can change the test.
Let me know if this makes sense.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: split.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451504#comment-16451504
]
Rushabh S Shah commented on HADOOP-15408:
-
Thanks [~xiaochen] for the patch.
Had an offline chat with [~daryn] on the proposed fix.
Identifier for both the tokens (i.e KMS_DELEGATION_TOKEN and kms-dt) are the
same (byte-to-byte) so we don't need to have another class
{{KMSLegacyDelegationTokenIdentifier}} for legacy token identifier.
Kind in Identifier doesn't mean much.
After removing {{KMSLegacyDelegationTokenIdentifier}} class and
{{KMSLegacyDelegationTokenIdentifier}} from
{{org.apache.hadoop.security.token.TokenIdentifier}}, 4 tests are failing in
TestKMS because they are trying to decodeIdentifier with kind {{kms-dt}} and
Serviceloader is not able to find any Identifier which corresponds to
{{kms-dt}} kind.
Since it is test-only code, we can change the test.
Let me know if this makes sense.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
> Attachments: split.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450294#comment-16450294
]
Rushabh S Shah commented on HADOOP-15408:
-
{quote}This fix also broke Ranger, for the same reason.
{quote}
Thanks Arpit for chiming in. Since there are multiple components affected by
this change, it makes sense to fix in hadoop itself.
bq. I didn't check how the class loader would work because it would see
'kms-dt' and be able to map to both the legacy identifier from the new jar, and
the only identifier from the old jar.
Thanks [~xiaochen] for the comment.
I don't completely grasp the idea. I would really appreciate if you can share
sample pseudo code explaining the idea.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450184#comment-16450184
]
Rushabh S Shah edited comment on HADOOP-15408 at 4/24/18 4:40 PM:
--
Thanks [~jojochuang] for taking a look.
bq. Is it a valid use case?
I think yes.
If we can come up with a simple fix the IMO we should fix it.
In our cluster, there are many services (like oozie, hive) which can bundle an
older versions of hadoop jar and they expect backwards compatibility between
minor hadoop versions.
Would like to hear more opinions.
was (Author: shahrs87):
bq. Is it a valid use case?
I think yes.
If we can come up with a simple fix the IMO we should fix it.
In our cluster, there are many services (like oozie, hive) which can bundle an
older versions of hadoop jar and they expect backwards compatibility between
minor hadoop versions.
Would like to hear more opinions.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15408) HADOOP-14445 broke Spark.
[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450184#comment-16450184
]
Rushabh S Shah commented on HADOOP-15408:
-
bq. Is it a valid use case?
I think yes.
If we can come up with a simple fix the IMO we should fix it.
In our cluster, there are many services (like oozie, hive) which can bundle an
older versions of hadoop jar and they expect backwards compatibility between
minor hadoop versions.
Would like to hear more opinions.
> HADOOP-14445 broke Spark.
> -
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.4
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15408) HADOOP-14445 broke Spark.
Rushabh S Shah created HADOOP-15408:
---
Summary: HADOOP-14445 broke Spark.
Key: HADOOP-15408
URL: https://issues.apache.org/jira/browse/HADOOP-15408
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.8.4
Reporter: Rushabh S Shah
Spark bundles hadoop related jars in their package.
Spark expects backwards compatibility between minor versions.
Their job failed after we deployed HADOOP-14445 in our test cluster.
{noformat}
2018-04-20 21:09:53,245 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2018-04-20 21:09:53,273 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.util.ServiceConfigurationError:
org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
KMSLegacyDelegationTokenIdentifier could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
at org.apache.hadoop.security.token.Token.toString(Token.java:413)
at java.lang.String.valueOf(String.java:2994)
at
org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
at
org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.(KMSDelegationToken.java:64)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 10 more
2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{noformat}
Their classpath looks like
{{\{...:hadoop-common-pre-HADOOP-14445.jar:.:hadoop-common-with-HADOOP-14445.jar:\}}}
This is because the container loaded {{KMSDelegationToken}} class from an older
jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it fails when
{{KMSLegacyDelegationTokenIdentifier}} wants to read {{TOKEN_LEGACY_KIND}} from
{{KMSDelegationToken}} which doesn't exist before.
Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449855#comment-16449855
]
Rushabh S Shah commented on HADOOP-15385:
-
Thanks [~jlowe] for the patch and [~tasanuma0829] for the review.
+1 (binding).
There are 3 test failures in {{TestDistCpSystem}}.
Since this patch is just a test code change, I can't think this patch broke
tests in {{TestDistCpSystem}}.
Jason, can you please verify that these failures are unrelated and if possible
create a tracking jira.
I will commit this patch shortly.
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.2
>Reporter: Rushabh S Shah
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15385-branch-2.001.patch
>
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
> c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
> format option. Contributed by Erik Krogen
> 1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
> fails. Contributed by Jason Lowe
> c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
> leveldb logger. Contributed by Jason Lowe.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445219#comment-16445219
]
Rushabh S Shah commented on HADOOP-15385:
-
{quote} Are you going on work on this JIRA?
{quote}
Hi Sammi, I am _not_ planning to work on this soon. I am occupied with some
other stuff.
Maybe we can ping who broke/introduced these failures and request them to take
a look ?
These failures are not seen in trunk or branch-3.1. Haven't checked branch-3.0.
bq. Otherwise, I might consider leave it to next release. Your thoughts?
Since the junits are failing, I think there is a hidden bug somewhere and
distcp is too important tool to ignore the failures. If there is a bug, then it
can result in data loss/corruption.
Would like to hear from other community members viewpoints.
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
> c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
> format option. Contributed by Erik Krogen
> 1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
> fails. Contributed by Jason Lowe
> c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
> leveldb logger. Contributed by Jason Lowe.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15385:
Target Version/s: 2.9.1, 2.8.4 (was: 2.8.4)
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
> c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
> format option. Contributed by Erik Krogen
> 1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
> fails. Contributed by Jason Lowe
> c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
> leveldb logger. Contributed by Jason Lowe.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444808#comment-16444808
]
Rushabh S Shah commented on HADOOP-15385:
-
Added blocker for 2.9.1 also. Please remove if you feel its not a blocker.
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
> c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
> format option. Contributed by Erik Krogen
> 1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
> fails. Contributed by Jason Lowe
> c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
> leveldb logger. Contributed by Jason Lowe.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444805#comment-16444805
]
Rushabh S Shah commented on HADOOP-15385:
-
These tests are failing in branch-2.9 also.
{noformat}
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
expected:<4> but was:<5>
[ERROR]
TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
expected:<2> but was:<3>
[ERROR]
TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
expected:<6> but was:<8>
[ERROR]
TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
expected:<4> but was:<5>
[ERROR] TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
[ERROR] TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
[ERROR] TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
[ERROR] TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
[ERROR] TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
[ERROR] TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
[ERROR]
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
[INFO]
[ERROR] Tests run: 73, Failures: 16, Errors: 0, Skipped: 0
{noformat}
{noformat}
C02QD8LYG8WP-lm:hadoop-distcp rushabhs$ git branch
* branch-2.9.1
C02QD8LYG8WP-lm:hadoop-distcp rushabhs$ git status
On branch branch-2.9.1
Your branch is up-to-date with 'origin/branch-2.9.1'.
nothing to commit, working tree clean
{noformat}
Cc [~Sammi].
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
>
[jira] [Resolved] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah resolved HADOOP-15399.
-
Resolution: Invalid
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443064#comment-16443064
]
Rushabh S Shah commented on HADOOP-15399:
-
Thanks [~jojochuang] for jira pointers.
Will test and close if necessary.
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Fix Version/s: 2.8.4
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Description:
KMSACLs uses {{AccessControlList}} for authorization.
For creating groups membership, the group implementation class that will be
instantiated is configured by {{hadoop.security.group.mapping}}.
Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
{{AccessControlList}}.
{{kms-acls.xml}} doesn't look the right place add the above config.
So KMSAcls should read either kms-site.
[~xiaochen]: Any preference which file should acls load ?
IMO it should be kms-site because that file is mandatory. But all the
properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit inclined
towards core-site.xml.
was:
KMSACLs create {{AccessControlList}} for authorization.
For creating groups membership, the group implementation class that will be
instantiated is configured by {{hadoop.security.group.mapping}}.
{{kms-acls.xml}} doesn't look the right place add the above config.
So KMSAcls should read either kms-site.
[~xiaochen]: Any preference which file should acls load ?
IMO it should be kms-site because that file is mandatory. But all the
properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit inclined
towards core-site.xml.
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Component/s: kms
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
>
> KMSACLs create {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah moved YARN-8176 to HADOOP-15399:
---
Key: HADOOP-15399 (was: YARN-8176)
Project: Hadoop Common (was: Hadoop YARN)
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
>
> KMSACLs create {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
[
https://issues.apache.org/jira/browse/HADOOP-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15385:
Priority: Blocker (was: Critical)
Target Version/s: 2.8.4 (was: 2.8.5)
Marking this Jira as a blocker for 2.8.4 release.
Cc [~djp].
> Many tests are failing in hadoop-distcp project in branch-2.8
> -
>
> Key: HADOOP-15385
> URL: https://issues.apache.org/jira/browse/HADOOP-15385
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Priority: Blocker
>
> Many tests are failing in hadoop-distcp project in branch-2.8
> Below are the failing tests.
> {noformat}
> Failed tests:
>
> TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
> expected:<4> but was:<5>
> TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
> expected:<2> but was:<3>
> TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
> expected:<4> but was:<5>
> TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
> expected:<2> but was:<3>
> TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
> expected:<6> but was:<8>
> Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
> {noformat}
> {noformat}
> rushabhs$ pwd
> /Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
> rushabhs$ git branch
> branch-2
> branch-2.7
> * branch-2.8
> branch-2.9
> branch-3.0
> rushabhs$ git log --oneline | head -n3
> c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
> format option. Contributed by Erik Krogen
> 1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
> fails. Contributed by Jason Lowe
> c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
> leveldb logger. Contributed by Jason Lowe.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15385) Many tests are failing in hadoop-distcp project in branch-2.8
Rushabh S Shah created HADOOP-15385:
---
Summary: Many tests are failing in hadoop-distcp project in
branch-2.8
Key: HADOOP-15385
URL: https://issues.apache.org/jira/browse/HADOOP-15385
Project: Hadoop Common
Issue Type: Bug
Components: tools/distcp
Affects Versions: 2.8.3
Reporter: Rushabh S Shah
Many tests are failing in hadoop-distcp project in branch-2.8
Below are the failing tests.
{noformat}
Failed tests:
TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel:326->checkResult:428
expected:<4> but was:<5>
TestDistCpViewFs.testGlobTargetMissingMultiLevel:346->checkResult:428
expected:<4> but was:<5>
TestDistCpViewFs.testGlobTargetMissingSingleLevel:306->checkResult:428
expected:<2> but was:<3>
TestDistCpViewFs.testUpdateGlobTargetMissingMultiLevel:367->checkResult:428
expected:<6> but was:<8>
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
TestIntegration.testUpdateGlobTargetMissingSingleLevel:431->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingMultiLevel:454->checkResult:577
expected:<4> but was:<5>
TestIntegration.testGlobTargetMissingSingleLevel:408->checkResult:577
expected:<2> but was:<3>
TestIntegration.testUpdateGlobTargetMissingMultiLevel:478->checkResult:577
expected:<6> but was:<8>
Tests run: 258, Failures: 16, Errors: 0, Skipped: 0
{noformat}
{noformat}
rushabhs$ pwd
/Users/rushabhs/hadoop/apacheHadoop/hadoop/hadoop-tools/hadoop-distcp
rushabhs$ git branch
branch-2
branch-2.7
* branch-2.8
branch-2.9
branch-3.0
rushabhs$ git log --oneline | head -n3
c4ea1c8bb73 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of
format option. Contributed by Erik Krogen
1548205a845 YARN-8147. TestClientRMService#testGetApplications sporadically
fails. Contributed by Jason Lowe
c01b425ba31 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom
leveldb logger. Contributed by Jason Lowe.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15381) KmsAcls should be reloaded periodically.
Rushabh S Shah created HADOOP-15381:
---
Summary: KmsAcls should be reloaded periodically.
Key: HADOOP-15381
URL: https://issues.apache.org/jira/browse/HADOOP-15381
Project: Hadoop Common
Issue Type: Bug
Components: kms
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Currently {{KMSACLs}} is getting reloaded depending on whether {{kms-acls}}
file is modified or not.
Since the access control is specified by {{AcccessControlList}}, the member may
contain {{unix-group}} or {{net-group}}.
If we add a new user to any group, technically the file is not modified but the
membership of a group is modified, so we need to reload the {{kms-acls}} file.
There are couple of solutions for this.
1. short-term solution: Reload the file periodically (like lets say every 15
minutes).
2. long-term solution: Implement a {{refreshUserGroupMapping}} like
protocol/servlet so whenever we change membership we can invoke that protocol
and we get the updates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432894#comment-16432894
]
Rushabh S Shah commented on HADOOP-14445:
-
{quote}The branch-2 and branch-2.8 conflicts are minor. Rushabh S Shah do you
want to give a final pass on them?
{quote}
+1 binding on branch-2 and branch-2.8 patches.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430948#comment-16430948
]
Rushabh S Shah commented on HADOOP-14445:
-
+1 binding for the latest trunk patch (v13) pending jenkins.
Thanks [~xiaochen].
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430803#comment-16430803
]
Rushabh S Shah commented on HADOOP-14445:
-
[~xiaochen]: Thanks for the latest patch.
bq. Testing this in real clusters revealed some issues in the patch.
Good find. The problem in creating the patch is there are 3 moving components
and making sure the patch covers all the cases is pretty difficult.
Overall the patch looks good.
Just one minor comment in new added test class.
+TestKMSClientProvider.java+
1.
{code}
public void testNotCopyFromLegacyToken() throws Exception {
...
assertNotEquals(uriString, tokens[0].getService().toString());
...
{code}
Instead of asserting what it is not equal to, we should assert what it should
be equal to.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.branch-2.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428485#comment-16428485
]
Rushabh S Shah commented on HADOOP-14445:
-
bq. Rushabh, do you mind giving another pass?
Sure. Will try my best to review today and if not, then surely on Monday.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.branch-2.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15344) LoadBalancingKMSClientProvider#close should not swallow exceptions
[
https://issues.apache.org/jira/browse/HADOOP-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427598#comment-16427598
]
Rushabh S Shah commented on HADOOP-15344:
-
bq. The exception from the failure is just logged and won't be thrown out.
Yes, you are correct. LBKMSCP#doOp will succeed.
{code}
finally {
if (keyProvider != null) {
keyProvider.close();
}
{code}
Lets say if one of kmscp threw an exception on close, then it will appear to
renew client that the renew operation failed but it succeeded in reality.
Am I making sense ?
> LoadBalancingKMSClientProvider#close should not swallow exceptions
> --
>
> Key: HADOOP-15344
> URL: https://issues.apache.org/jira/browse/HADOOP-15344
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Xiao Chen
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HADOOP-15344.001.patch
>
>
> As [~shahrs87]'s comment on HADOOP-14445 says:
> {quote}
> LoadBalancingKMSCP never throws IOException back. It just swallows all the
> IOException and just logs it.
> ...
> Maybe we might want to return MultipleIOException from
> LoadBalancingKMSCP#close.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424395#comment-16424395 ] Rushabh S Shah commented on HADOOP-14987: - [~xiaochen]:Do you mind attaching the latest patch that you committed for branch-2* ? > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.0.0, 2.10.0, 2.8.4, 2.9.2 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424166#comment-16424166
]
Rushabh S Shah commented on HADOOP-14445:
-
Thanks [~xiaochen] for the latest patch.
It looks good.
bq. If KMS server is old, you'd get an old token.
Thanks for catching that. I totally missed that.
There is one test failure in the latest run.
{noformat}
org.apache.hadoop.conf.TestCommonConfigurationFields.testCompareXmlAgainstConfigurationClass
Failing for the past 1 build (Since Failed#14425 )
Took 0.2 sec.
Error Message
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys class
org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys class
org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort class
org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController class
org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping class
org.apache.hadoop.io.erasurecode.CodecUtil class
org.apache.hadoop.security.RuleBasedLdapGroupsMapping Entries:
hadoop.security.key.default.bitlength hadoop.security.key.default.cipher
expected:<0> but was:<2>
{noformat}
I can't think of a way that your latest patch can introduce this failure.
The hadoop-common build is fairly stable compared to hadoop-hdfs. Can you
please double check whether your patch introduced this failure.
If not, can you please find out which jira is responsible ?
Also there are couple of checkstyle warnings in TestKMS.java regarding unused
import.
If the test failure is not related, then you can make the checkstyle changes
while committing.
Also can you upload the new patch after committing and resolving the jira.
I know some people had concerns that it is difficult to co-relate the commit
with the last patch if they are not the same.
+1 (non-binding) pending confirming test failure.
Thanks a lot for the good work here.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420784#comment-16420784
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/31/18 6:19 PM:
--
Thanks [~xiaochen] for revising the patch.
The patch looks really good and almost close.
Just few minor comments.
+TestKMS.java+
{code:java}
public void doKMSHAZKWithDelegationTokenAccess() throws Exception {
...
...
...
doAs("client",
new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// verify both tokens can be used to authenticate
for (Token t : credentials.getAllTokens()) {
assertTokenAccess(lbkp1, keyName, t);
}
return null;
}
});
}
private void assertTokenAccess(final LoadBalancingKMSClientProvider lbkp,
final String keyName, final Token token) throws Exception {
UserGroupInformation tokenUgi =
UserGroupInformation.createUserForTesting("test", new String[] {});
// Verify the tokens can authenticate to any KMS
tokenUgi.addToken(token);
tokenUgi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
{code}
I don't understand much of UGI so I apologize in advance if this question
doesn't make sense.
Why are we doing {{doAs("client")}} when we are again going to do {{doAs
("test")}} user in {{assertTokenAccess}} ?
+KMSClientProvider.java+
{code:java}
assert KMSDelegationToken.TOKEN_KIND.equals(token.getKind());
if (!copyLegacyToken || !KMSDelegationToken.TOKEN_KIND
.equals(token.getKind())) {
LOG.debug("Not creating legacy token because copyLegacyToken={}, "
+ "token={}", copyLegacyToken, token);
return null;
}
{code}
Is it ever possible that {{token.getKind != KMSDelegationToken.TOKEN_KIND}} ?
I would replace {{assert}} with {{PreConditions}} and remove
{{!KMSDelegationToken.TOKEN_KIND.equals(token.getKind())}} check from if
statement.
+core-default.xml+
typo
{noformat}
confronting to kms-dt. All other parts of the token remain the same.
{noformat}
Instead of {{confronting}}, I think you meant {{confirming}} ?
was (Author: shahrs87):
Thanks [~xiaochen] for revising the patch.
The patch looks really good and almost close.
Just few minor comments.
+TestKMS.java+
{code}
public void doKMSHAZKWithDelegationTokenAccess() throws Exception {
...
...
...
doAs("client",
new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// verify both tokens can be used to authenticate
for (Token t : credentials.getAllTokens()) {
assertTokenAccess(lbkp1, keyName, t);
}
return null;
}
});
}
private void assertTokenAccess(final LoadBalancingKMSClientProvider lbkp,
final String keyName, final Token token) throws Exception {
UserGroupInformation tokenUgi =
UserGroupInformation.createUserForTesting("test", new String[] {});
// Verify the tokens can authenticate to any KMS
tokenUgi.addToken(token);
tokenUgi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
{code}
I don't understand much of UGI so I apologize in advance if this question
doesn't make sense.
Why are we doing {{doAs("client")}} when we are again going to do
{{doAs("test")} user in {{assertTokenAccess}}} ?
+KMSClientProvider.java+
{code}
assert KMSDelegationToken.TOKEN_KIND.equals(token.getKind());
if (!copyLegacyToken || !KMSDelegationToken.TOKEN_KIND
.equals(token.getKind())) {
LOG.debug("Not creating legacy token because copyLegacyToken={}, "
+ "token={}", copyLegacyToken, token);
return null;
}
{code}
Is it ever possible that {{token.getKind != KMSDelegationToken.TOKEN_KIND}} ?
I would replace {{assert}} with {{PreConditions}} and remove
{{!KMSDelegationToken.TOKEN_KIND.equals(token.getKind())}} check from if
statement.
+core-default.xml+
typo
{noformat}
confronting to kms-dt. All other parts of the token remain the same.
{noformat}
Instead of {{confronting}}, I think you meant {{confirming}} ?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments:
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420784#comment-16420784
]
Rushabh S Shah commented on HADOOP-14445:
-
Thanks [~xiaochen] for revising the patch.
The patch looks really good and almost close.
Just few minor comments.
+TestKMS.java+
{code}
public void doKMSHAZKWithDelegationTokenAccess() throws Exception {
...
...
...
doAs("client",
new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// verify both tokens can be used to authenticate
for (Token t : credentials.getAllTokens()) {
assertTokenAccess(lbkp1, keyName, t);
}
return null;
}
});
}
private void assertTokenAccess(final LoadBalancingKMSClientProvider lbkp,
final String keyName, final Token token) throws Exception {
UserGroupInformation tokenUgi =
UserGroupInformation.createUserForTesting("test", new String[] {});
// Verify the tokens can authenticate to any KMS
tokenUgi.addToken(token);
tokenUgi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
{code}
I don't understand much of UGI so I apologize in advance if this question
doesn't make sense.
Why are we doing {{doAs("client")}} when we are again going to do
{{doAs("test")} user in {{assertTokenAccess}}} ?
+KMSClientProvider.java+
{code}
assert KMSDelegationToken.TOKEN_KIND.equals(token.getKind());
if (!copyLegacyToken || !KMSDelegationToken.TOKEN_KIND
.equals(token.getKind())) {
LOG.debug("Not creating legacy token because copyLegacyToken={}, "
+ "token={}", copyLegacyToken, token);
return null;
}
{code}
Is it ever possible that {{token.getKind != KMSDelegationToken.TOKEN_KIND}} ?
I would replace {{assert}} with {{PreConditions}} and remove
{{!KMSDelegationToken.TOKEN_KIND.equals(token.getKind())}} check from if
statement.
+core-default.xml+
typo
{noformat}
confronting to kms-dt. All other parts of the token remain the same.
{noformat}
Instead of {{confronting}}, I think you meant {{confirming}} ?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419283#comment-16419283
]
Rushabh S Shah commented on HADOOP-15336:
-
bq. In any case, I think this is worth committing.
Agree.
bq. Actually I would switch the null check by hasKeyProviderUri().
I don't think {{FSserverDefaults}} has {{hasKeyProviderUri}} method.
bq. Note that the reverse convert does actually check fs.hasKeyProviderUri().
Reverse is an object FSserverDefaultsProto, thats why it has
{{hasKeyProviderUri}} method.
We can add {{hasKeyProviderUri}} method in {{FSserverDefaults}} and do null
check in that method.
> NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication
> between 2.7 and 3.2
> -
>
> Key: HADOOP-15336
> URL: https://issues.apache.org/jira/browse/HADOOP-15336
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Sherwood Zheng
>Assignee: Sherwood Zheng
>Priority: Major
> Labels: backward-incompatible, common
> Attachments: HADOOP-15336.000.patch, HADOOP-15336.001.patch
>
>
> KeyProviderUri is not available in 2.7 so when 2.7 clients contact with 3.2
> services, it cannot find the key provider URI and triggers a
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419154#comment-16419154
]
Rushabh S Shah commented on HADOOP-15336:
-
Thanks [~elgoiri] for chiming in.
bq. The reason for the Router to decode the request is because it needs to know
the path the request is going to.
In the above stack trace, router is decoding response from namenode _not
request_. So it doesn't need to decode.
bq. Regarding the backwards compatibility issue, I don't think this is
introducing one but fixing one.
The {{backwards incompatible}} flag on this jira means {{HADOOP-14104}}
introduced this incompatibility and that is not correct. I wasn't clear earlier.
bq. In any case, this is a general issue and you can repro by having a client
and a server (no need for a Router).
You can't reproduce this issue if there are only 2 components.
Assuming client (dfs client) is 3.2 and server(namenode) is 2.7.
2.7 Namenode doesn't support return keyProviderUri in {{FsServerDefaults}} and
there is {{hasKeyProviderUri}} check on client side so it won't throw NPE.
Assuming client (dfs client) is 2.7 and server(namenode) is 3.2.
Namenode supports returning {{keyProviderUri}} but client doesn't understand
{{keyProviderUri}} and since {{keyProviderUri}} is an optional field in
protobuf, client will decode the fields only it understands.
> NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication
> between 2.7 and 3.2
> -
>
> Key: HADOOP-15336
> URL: https://issues.apache.org/jira/browse/HADOOP-15336
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Sherwood Zheng
>Assignee: Sherwood Zheng
>Priority: Major
> Labels: backward-incompatible, common
> Attachments: HADOOP-15336.000.patch, HADOOP-15336.001.patch
>
>
> KeyProviderUri is not available in 2.7 so when 2.7 clients contact with 3.2
> services, it cannot find the key provider URI and triggers a
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418039#comment-16418039 ] Rushabh S Shah commented on HADOOP-15336: - Thanks [~zhengxg3] for the stack trace. The stack trace shows that the router(acting as client) got an NPE from namenode. I am not familiar with RouterRpcServer so I apologize in advance if I sound confused. But why does router decodes the protobuf response and then encodes again ? Why can't router just relays the response from namenode without decoding and re-encoding ? IIUC now I have to think about backwards compatibility w.r.t to dfs client, namenode *and router* even if majority of people don't use router feature ? IMO this shouldn't be flagged as backwards incompatible. > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.2 > - > > Key: HADOOP-15336 > URL: https://issues.apache.org/jira/browse/HADOOP-15336 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Major > Labels: backward-incompatible, common > Attachments: HADOOP-15336.000.patch, HADOOP-15336.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15345) Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient adding/getting/removing nodes
[ https://issues.apache.org/jira/browse/HADOOP-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417384#comment-16417384 ] Rushabh S Shah commented on HADOOP-15345: - bq. And the conflict part is all about package import. Then I am +1 non-binding. > Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient > adding/getting/removing nodes > --- > > Key: HADOOP-15345 > URL: https://issues.apache.org/jira/browse/HADOOP-15345 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.6 >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15345-branch-2.7.001.patch > > > As per discussion in HADOOP-15343 backport HADOOP-12185 to branch-2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15345) Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient adding/getting/removing nodes
[ https://issues.apache.org/jira/browse/HADOOP-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417382#comment-16417382 ] Rushabh S Shah commented on HADOOP-15345: - bq. I have resolved the conflict after cherry-pick Can you paste the conflict ? > Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient > adding/getting/removing nodes > --- > > Key: HADOOP-15345 > URL: https://issues.apache.org/jira/browse/HADOOP-15345 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.6 >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15345-branch-2.7.001.patch > > > As per discussion in HADOOP-15343 backport HADOOP-12185 to branch-2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15345) Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient adding/getting/removing nodes
[ https://issues.apache.org/jira/browse/HADOOP-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417335#comment-16417335 ] Rushabh S Shah commented on HADOOP-15345: - [~hexiaoqiao]: thanks for the patch. Was the backport without any conflict ? If yes, then I am +1 (non-binding). If no, please post the diff and then I can review. > Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient > adding/getting/removing nodes > --- > > Key: HADOOP-15345 > URL: https://issues.apache.org/jira/browse/HADOOP-15345 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.6 >Reporter: He Xiaoqiao >Priority: Major > Attachments: HADOOP-15345-branch-2.7.001.patch > > > As per discussion in HADOOP-15343 backport HADOOP-12185 to branch-2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415651#comment-16415651
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/27/18 2:06 PM:
--
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
But I think its late now since the other jira is already committed.
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
was (Author: shahrs87):
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415651#comment-16415651
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/27/18 2:05 PM:
--
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
was (Author: shahrs87):
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
Let me see if some other test does something smart to get around this problem.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415651#comment-16415651
]
Rushabh S Shah commented on HADOOP-14445:
-
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
Let me see if some other test does something smart to get around this problem.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15344) LoadBalancingKMSClientProvider#close should not swallow exceptions
[
https://issues.apache.org/jira/browse/HADOOP-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415626#comment-16415626
]
Rushabh S Shah commented on HADOOP-15344:
-
Now I am thinking whether we should do this change.
{{keyProvider#close}} is getting called from {{KMSTokenRenewer#close}}.
{code:java}
// Some comments here
public long renew(Token token, Configuration conf) throws IOException {
{
...
...
return ((KeyProviderDelegationTokenExtension.DelegationTokenExtension)
keyProvider).renewDelegationToken(token);
} finally {
if (keyProvider != null) {
keyProvider.close();
}
}
}
{code}
Lets assume the {{keyProvider}} object is a {{LoadBalancingKMSCP}} comprising
of 2 {{KMSCP}} and one {{KMSClientProvider}} is having problems while closing.
Since there are retries inside {{LoadBalancingKMSCP#doOp}},
{{keyProvider#renewDelegationToken(token)}} will retry after encountering bad
{{KMSCP}} and it succeeds on second KMSCP but {{renew}} will throw an Exception
in finally block which will fail the renew operation.
Hope it makes sense.
Let me know if I missed anything or if my analysis is incorrect.
> LoadBalancingKMSClientProvider#close should not swallow exceptions
> --
>
> Key: HADOOP-15344
> URL: https://issues.apache.org/jira/browse/HADOOP-15344
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Xiao Chen
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HADOOP-15344.001.patch
>
>
> As [~shahrs87]'s comment on HADOOP-14445 says:
> {quote}
> LoadBalancingKMSCP never throws IOException back. It just swallows all the
> IOException and just logs it.
> ...
> Maybe we might want to return MultipleIOException from
> LoadBalancingKMSCP#close.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15343) NetworkTopology#getLoc should exit loop earlier rather than traverse all children
[
https://issues.apache.org/jira/browse/HADOOP-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414246#comment-16414246
]
Rushabh S Shah commented on HADOOP-15343:
-
Does it make sense to backport HADOOP-12185 to branch-2.7 ?
HADOOP-12185 is trying to solve the in-efficiencies in adding/getting/removing
nodes from {{NetworkTopology}}.
I understand this jira is subset of HADOOP-12185 but doesn't hurt getting more
performance benefits.
> NetworkTopology#getLoc should exit loop earlier rather than traverse all
> children
> -
>
> Key: HADOOP-15343
> URL: https://issues.apache.org/jira/browse/HADOOP-15343
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.6
>Reporter: He Xiaoqiao
>Priority: Major
> Labels: performance
> Attachments: HADOOP-15343-branch-2.7.001.patch
>
>
> NetworkTopology#getLoc return a proper node after traverse ALL children of
> current {{InnerNode}} even if it has found expected result, based on
> branch-2.7. This issue may lead some performance loss especially for a large
> & busy cluster and many nodes under a rack. I think it should exit loop
> earlier rather than traverse all children of {{InnerNode}}.
> {code:java}
> private Node getLoc(String loc) {
> if (loc == null || loc.length() == 0) return this;
>
> String[] path = loc.split(PATH_SEPARATOR_STR, 2);
> Node childnode = null;
> for(int i=0; i if (children.get(i).getName().equals(path[0])) {
> childnode = children.get(i);
> }
> }
> if (childnode == null) return null; // non-existing node
> if (path.length == 1) return childnode;
> if (childnode instanceof InnerNode) {
> return ((InnerNode)childnode).getLoc(path[1]);
> } else {
> return null;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412915#comment-16412915
]
Rushabh S Shah commented on HADOOP-14445:
-
Thanks [~xiaochen] for the revision.
[~daryn] shared his concerns offline that this patch is dependent on config
key and we will live with the baggage of 2 tokens forever.
I will request him to review asap.
But below are my review comments Mostly all are minor.
+KMSClientProvider.java+
1. {{KMSCP#addTokenToCredentials}}
I don't agree with the method name. The method name suggests that we are just
adding token to credentials object.
But we are doing much more in this method.
I would add {{credentials.addToken}} line to calling method
{{addDelegationTokens}} and rename method {{addTokenToCredentials}} to
something like {{createLegacyKmsToken}}
and add some javadoc to it saying that creating this token is dependent on
conf key {{KMS_CLIENT_COPY_LEGACY_TOKEN_KEY}}.
+DelegationTokenAuthenticatedURL.java+
1. Lets change the {{selectDelegationToken}} to {{getDelegationToken}}.
In the base implementation, we are not implementing any Selector.
We are just getting the token based on the service field.
In {{KMSCP}} we can change the method name {{getKMSToken}} to
{{selectKMSToken}} because there we are implementing {{TokenSelector}}.
+core-default.xml+
{quote}With the default value set to true, the client will locally duplicate
the KMS_DELEGATION_TOKEN token and create a kms-dt token, all other parts of
the token remain the same.
{quote}
Technically this is not true. The service is also changed.
I am sorry I _should have_ mentioned all the above comments in the previous
review.
+TestKMS.java+
1. This is unrelated to patch.
Do we really want to stop kdc after every test ?
2. {{providersCreated}}: Should this be a list or just {{KeyProvider}} ?
It will always create {{LoadBalancingKeyProivder}} which internally is a set
of {{KeyProvider}}.
{{LoadBalancingKMSCP}} never throws IOException back. It just swallows all the
{{IOException}} and just logs it.
Maybe we might want to return MultipleIOException from
{{LoadBalancingKMSCP#close}}. Totally fine if we want to do it as a separate
jira.
But as far as this jira is concerned, we can get rid of
{{MultipleIOException}} related changes and can just change it to
{{IOException}}.
3. {{testKMSHAZKDelegationTokenRenewCancel(final Text tokenKind)}}
Unable to understand why were are calling {{setupConfForToken}} multiple times.
If we filter out all tokens other than passed {{tokenKind}}, then we can just
call {{setupConfForToken}} once at the start.
That way the code will be more clean and _will only_ work on {{token ==
tokenKind}}.
In short just one for loop, filter out the token at the start and test renew,
cancel and again renew(which should fail) in sequence.
4. {{testTokenCompatibilityOldRenewer}}
This test ran for {{34.887}} on my local machine.
I am sure majority of time was spent in sleeping for token to expire.
Can we decrease the {{renewInterval}} period to less than 30 seconds (maybe 5
seconds or so).
Also the test renewed the token 30 times. Is that expected ?
Did you mean to renew after while the code came out of while loop ?
{quote}LOG.info("Rolling key {} via provider {} with tokenUGi.",keyName);
kp.createKey("newkey", new KeyProvider.Options(conf));
{quote}
The log line should be {{creating a new key}} instead of {{Rolling key}}.
Let me know if any part is unclear.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411625#comment-16411625 ] Rushabh S Shah commented on HADOOP-15336: - Also was client 2.7 or 3.2 ? and same for server. > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.2 > - > > Key: HADOOP-15336 > URL: https://issues.apache.org/jira/browse/HADOOP-15336 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Major > Labels: backward-incompatible, common > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-11423: Comment: was deleted (was: If we agree that we will support only LTS releases then why java 10 ? It should be either java 8 or java 11. Did I miss something ?) > [Umbrella] Fix Java 10 incompatibilities in Hadoop > -- > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411394#comment-16411394 ] Rushabh S Shah commented on HADOOP-11423: - If we agree that we will support only LTS releases then why java 10 ? It should be either java 8 or java 11. Did I miss something ? > [Umbrella] Fix Java 10 incompatibilities in Hadoop > -- > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408698#comment-16408698
]
Rushabh S Shah commented on HADOOP-14445:
-
One more high level comment.
Lets say we are running in an environment where the job submitter and task
client are running with the new code but RM is not upgraded yet.
Job submitter will have {{KMS_DELEGATION_TOKEN}} and {{kms-dt}} tokens in its
credentials bag.
Lets say the job ran for more than 24 hours so that RM has to renew the token.
Since the RM is old, it will only renew {{kms-dt}} token and _will fail to
renew KMS_DELEGATION_TOKEN_.
Now when the new task tries to select the token via {{getKMSToken}}, it will
always select the token with kind {{KMS_DELEGATION_TOKEN}}.
If I understand the DelegationToken correctly, even if it fails to renew token
specific to \{{KMS_DELEGATION_TOKEN}}, that token will be valid since
\{{kms-dt}} is a copy of \{{KMS_D_T}}.
Lets just write a test case to verify.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407144#comment-16407144
]
Rushabh S Shah commented on HADOOP-14445:
-
Made a pass through the production code and not test code. Will go through test
code tomorrow.
Overall the patch looks very good.
Below are my review comments.
+KMSClientProvider.java+
1. {{token.setService(dtService)}} should be done in {{addDelegationTokens}}
and not {{addTokenToCredentials}}
2. final Collection tokens =
ugi.getTokens();
{{Ugi#getTokens}} internally creates a new ArrayList of all tokens.
Why there is a need to create a new ArrayList for every call to kms ?
+KMSUtil.java+
1. In {{createKeyProviderFromTokenService}} why do we want to fall back to
conf ?
This method is getting called from {{KMSTokenRenewer#createKeyProvider}} which
has delegation token kind as {{KMS_DELEGATION_TOKEN}}. So we are absolutely
sure that the service in the token will be {{URI}}.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404983#comment-16404983 ] Rushabh S Shah commented on HADOOP-12125: - [~jzhuge]: I don't have enough cycles to work on this jira. Please go ahead and re-assign if you plan to work on this. > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: Rushabh S Shah >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15319) hadoop fs -rm command misbehaves on recent hadoop version 2.8.2
[ https://issues.apache.org/jira/browse/HADOOP-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-15319: Summary: hadoop fs -rm command misbehaves on recent hadoop version 2.8.2 (was: hadoop fs -rm command misbehaves on recent hadoop version 2.5.0) > hadoop fs -rm command misbehaves on recent hadoop version 2.8.2 > --- > > Key: HADOOP-15319 > URL: https://issues.apache.org/jira/browse/HADOOP-15319 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.8.2 >Reporter: Saurabh Padhy >Priority: Major > > This issue is regarding hadoop fs -rm command. > In hadoop version 2.4.0 when we execute "hadoop fs -rm /a/b/c/*", > It removes the files inside the c directory only. > But in case of versions higher to 2.8.2, > When we execute "hadoop fs -rm /a/b/c/**" or "hdfs dfs -rm /a/b/c/**" > It removes the inside files and directory as well. > Please look into the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
