[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640513#comment-16640513
]
Ajay Kumar edited comment on HADOOP-14445 at 10/6/18 2:07 AM:
--
[~xiaochen] thanks for taking it forward. LGTM. Have a question about
{{canonicalService}} field in {{LoadBalancingKMSClientProvider}}. Did some
basic testing and i think whole thing will work without this field as well.
Shall we get rid of it? Also in
{{TestLoadBalancingKMSClientProvider#testTokenServiceCreationWithUriFormat}}
shall we add assertion for dtService as well?
was (Author: ajayydv):
[~xiaochen] thanks or taking it forward. LGTM. Have a question about
{{canonicalService}} field in {{LoadBalancingKMSClientProvider}}. Did some
basic testing and i think whole thing will work without this field as well.
Shall we get rid of it? Also in
{{TestLoadBalancingKMSClientProvider#testTokenServiceCreationWithUriFormat}}
shall we add assertion for dtService and canonicalService as well?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580524#comment-16580524
]
Ajay Kumar edited comment on HADOOP-14445 at 8/14/18 10:59 PM:
---
[~xiaochen] thanks for taking this forward. I might be missing something here
but wanted to put this question in open. L347 in
DelegationTokenAuthenticatedURL#selectDelegationToken will return a service in
ip:port or host:port format (not uri). So creds.getToken(service) in next line
may not return the HA KMS DT as service name will be in URI format. Could you
please share if this is not the case.
was (Author: ajayydv):
[~xiaochen] thanks for taking this forward. I might be missing something here
but wanted to put this question in open. L347 in
DelegationTokenAuthenticatedURL#selectDelegationToken will return a service in
ip:port or host:port format (not uri). So creds.getToken(service) in next line
may not return the HA KMS DT as service name will be in URI format.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566084#comment-16566084
]
Ajay Kumar edited comment on HADOOP-14445 at 8/2/18 6:29 PM:
-
[~xiaochen] thanks for updating the patch. LGTM.
{quote}This patch does not change that code path and is unrelated to client
retry behavior, but on the delegation token side. So I don't think adding the
config would add reasonable value here.
{quote}
Although this patch is mostly about KMS DT i was thinking of failover testing
for KMS HA when auth method will be DT. But i don't feel too strongly about it,
feel free to commit it as this is important improvement over current state.
was (Author: ajayydv):
[~xiaochen] thanks for updating the patch. LGTM.
{quote}This patch does not change that code path and is unrelated to client
retry behavior, but on the delegation token side. So I don't think adding the
config would add reasonable value here.{quote}
Although this patch is mostly about KMS DT i was thinking of failover testing
for KMS HA when auth method will be DT.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558856#comment-16558856
]
Ajay Kumar edited comment on HADOOP-14445 at 7/26/18 8:33 PM:
--
[~xiaochen] thanks for working on this important missing functionality for KMS.
Patch v15 LGTM. Few minor comments:
# KMSClientProvider#renew Shall we consolidate keyProvider retrieval in single
function? We can call {{createKeyProviderFromTokenService}} from
{{KMSUtil.createKeyProvider}}. Corresponding debug statement may be moved there
as well.
# KMSClientProvider#cancel: same as above
# TestLoadBalancingKMSClientProvider: To test failure in KMS HA with uri
enabled setting shall we add another test case to call
testLoadBalancingWithFailure with KMS_CLIENT_TOKEN_USE_URI_FORMAT_KEY set to
true?
was (Author: ajayydv):
[~xiaochen] thanks for working on this important missing functionality for KMS.
Patch v15 LGTM. Few minor comments:
# KMSClientProvider#renew Shall we consolidate kerProvider retrievel in single
function? We can call {{createKeyProviderFromTokenService}} from
{{KMSUtil.createKeyProvider}}.We can move the debug statement there as well.
# KMSClientProvider#cancel: same as above
# TestLoadBalancingKMSClientProvider: To test failure in KMS HA shall we add
another test case to call testLoadBalancingWithFailure with
KMS_CLIENT_TOKEN_USE_URI_FORMAT_KEY set to true?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556371#comment-16556371
]
Xiao Chen edited comment on HADOOP-14445 at 7/25/18 10:51 PM:
--
Thanks for the review [~jojochuang]. Will rebase on the next patch.
1. Good catch, you're right, will update in next rev.
2. Would {{KMSUtiltestInjector}} suffice? Or any other suggestions?
3. Could you clarify which line?
bq. 4
I feel that's the goal of release notes. The last sentence of the release note
seems clear to me that the flag should not be set arbitrarily. If we raise the
incompatible flag, we theoretically cannot backport this to 2.x and 3.x...
Open to any recommendations though.
was (Author: xiaochen):
Thanks for the review [~jojochuang]. Will rebase on the next patch.
bq. 1
Good catch, you're right, will update in next rev.
bq. 2
Would {{KMSUtiltestInjector}} suffice? Or what any other suggestions?
bq. 3
Could you clarify which line?
bq. 4
I feel that's the goal of release notes. The last sentence of the release note
seems clear to me that the flag should not be set arbitrarily. If we raise the
incompatible flag, we theoretically cannot backport this to 2.x and 3.x...
Open to any recommendations though.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556250#comment-16556250
]
Wei-Chiu Chuang edited comment on HADOOP-14445 at 7/25/18 8:43 PM:
---
Had another look a the latest patch (rev14), thanks [~xiaochen] for continuing
the work.
Mostly LGTM but the patch won't apply cleanly now due to HADOOP-15591:
{code:java}
LOG.debug("New token received: ({})", token);
{code}
Nit:
1. why is the setKind() repeated for token?
{code:java|title=TestKMSClientProvider:setup}
token.setKind(TOKEN_KIND);
token.setService(new Text(providerUriString));
token.setKind(TOKEN_KIND);
{code}
Do you meant to set oldToken instead?
2. Question: why is it called “KMSUtilFaultInjector”? It doesn’t emit an
exception or introduce a fault.
{code:java|title=TestLoadBalancingKMSClientProvider#setup()}
KMSUtilFaultInjector.set(new KMSUtilFaultInjector());
{code}
3. it doesn’t look like this line is needed.
4. Finally, how about marking this jira an Incompatible Change? Clearly, this
is only an incompatible change when you flip the switch, however I want to
raise a little more awareness of the potential problem once this switched on.
was (Author: jojochuang):
Had another look a the latest patch (rev14), thanks [~xiaochen] for continuing
the work.
Most LGTM but the patch won't apply due to HADOOP-15591:
{code}
LOG.debug("New token received: ({})", token);
{code}
Nit:
1. why is the setKind() repeated for token?
{code:title=TestKMSClientProvider:setup}
token.setKind(TOKEN_KIND);
token.setService(new Text(providerUriString));
token.setKind(TOKEN_KIND);
{code}
Do you meant to set oldToken instead?
2. Question: why is it called “KMSUtilFaultInjector”? It doesn’t emit an
exception or introduce a fault.
{code:title=TestLoadBalancingKMSClientProvider#setup()}
KMSUtilFaultInjector.set(new KMSUtilFaultInjector());
{code}
3. it doesn’t look like this line is needed.
4. Finally, how about marking this jira an Incompatible Change? Clearly, this
is only an incompatible change when you flip the switch, however I want to
raise a little more awareness of the potential problem once this switched on.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466221#comment-16466221
]
Xiao Chen edited comment on HADOOP-14445 at 5/7/18 7:54 PM:
Reopening Jira as I'm reverting those changes.
Will remove fix versions as I proceed. Some minor conflicts due to
HADOOP-14188, HADOOP-15390 and HADOOP-15313. Ran {{mvn clean test -DskipShade
-Dmaven.javadoc.skip=true -Dtest=TestKMS*,TestDelegationTokenRenewer}} before
pushing. Attached a trunk and a branch-2.8 version for reference - 3.x lines
are similar to trunk, and 2.x lines similar to 2.8.
(HDFS-13430 will also be reverted to accommodate this)
was (Author: xiaochen):
Reopening Jira as I'm reverting those changes.
Will remove fix versions as I proceed. Some minor conflicts due to HADOOP-14188
and HADOOP-15313, so building repo and running touched tests before I push.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420784#comment-16420784
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/31/18 6:19 PM:
--
Thanks [~xiaochen] for revising the patch.
The patch looks really good and almost close.
Just few minor comments.
+TestKMS.java+
{code:java}
public void doKMSHAZKWithDelegationTokenAccess() throws Exception {
...
...
...
doAs("client",
new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// verify both tokens can be used to authenticate
for (Token t : credentials.getAllTokens()) {
assertTokenAccess(lbkp1, keyName, t);
}
return null;
}
});
}
private void assertTokenAccess(final LoadBalancingKMSClientProvider lbkp,
final String keyName, final Token token) throws Exception {
UserGroupInformation tokenUgi =
UserGroupInformation.createUserForTesting("test", new String[] {});
// Verify the tokens can authenticate to any KMS
tokenUgi.addToken(token);
tokenUgi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
{code}
I don't understand much of UGI so I apologize in advance if this question
doesn't make sense.
Why are we doing {{doAs("client")}} when we are again going to do {{doAs
("test")}} user in {{assertTokenAccess}} ?
+KMSClientProvider.java+
{code:java}
assert KMSDelegationToken.TOKEN_KIND.equals(token.getKind());
if (!copyLegacyToken || !KMSDelegationToken.TOKEN_KIND
.equals(token.getKind())) {
LOG.debug("Not creating legacy token because copyLegacyToken={}, "
+ "token={}", copyLegacyToken, token);
return null;
}
{code}
Is it ever possible that {{token.getKind != KMSDelegationToken.TOKEN_KIND}} ?
I would replace {{assert}} with {{PreConditions}} and remove
{{!KMSDelegationToken.TOKEN_KIND.equals(token.getKind())}} check from if
statement.
+core-default.xml+
typo
{noformat}
confronting to kms-dt. All other parts of the token remain the same.
{noformat}
Instead of {{confronting}}, I think you meant {{confirming}} ?
was (Author: shahrs87):
Thanks [~xiaochen] for revising the patch.
The patch looks really good and almost close.
Just few minor comments.
+TestKMS.java+
{code}
public void doKMSHAZKWithDelegationTokenAccess() throws Exception {
...
...
...
doAs("client",
new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
// verify both tokens can be used to authenticate
for (Token t : credentials.getAllTokens()) {
assertTokenAccess(lbkp1, keyName, t);
}
return null;
}
});
}
private void assertTokenAccess(final LoadBalancingKMSClientProvider lbkp,
final String keyName, final Token token) throws Exception {
UserGroupInformation tokenUgi =
UserGroupInformation.createUserForTesting("test", new String[] {});
// Verify the tokens can authenticate to any KMS
tokenUgi.addToken(token);
tokenUgi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
{code}
I don't understand much of UGI so I apologize in advance if this question
doesn't make sense.
Why are we doing {{doAs("client")}} when we are again going to do
{{doAs("test")} user in {{assertTokenAccess}}} ?
+KMSClientProvider.java+
{code}
assert KMSDelegationToken.TOKEN_KIND.equals(token.getKind());
if (!copyLegacyToken || !KMSDelegationToken.TOKEN_KIND
.equals(token.getKind())) {
LOG.debug("Not creating legacy token because copyLegacyToken={}, "
+ "token={}", copyLegacyToken, token);
return null;
}
{code}
Is it ever possible that {{token.getKind != KMSDelegationToken.TOKEN_KIND}} ?
I would replace {{assert}} with {{PreConditions}} and remove
{{!KMSDelegationToken.TOKEN_KIND.equals(token.getKind())}} check from if
statement.
+core-default.xml+
typo
{noformat}
confronting to kms-dt. All other parts of the token remain the same.
{noformat}
Instead of {{confronting}}, I think you meant {{confirming}} ?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments:
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415651#comment-16415651
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/27/18 2:06 PM:
--
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
But I think its late now since the other jira is already committed.
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
was (Author: shahrs87):
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415651#comment-16415651
]
Rushabh S Shah edited comment on HADOOP-14445 at 3/27/18 2:05 PM:
--
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
If we can test that new RM is able to renew both tokens (which is already
present in your test suite in last patch) and the identifier bits are the same
in both tokens then we can remove this test case.
Hope it makes sense.
was (Author: shahrs87):
{quote}
2. providersCreated: Should this be a list or just KeyProvider ?
bq. I disagree because even in tests we should code against interface.
{quote}
I think I was not clear earlier.
What I meant was should it be a list of KeyProvider {{List}} or
just single element {{KeyProvider}} ?
I agree with you completely that we should code against interface. Thats why I
feel it should be just a {{KeyProvider}}.
{noformat}
KeyProvider keyProvider = KeyProviderFactory.get(providerUri, conf);
{noformat}
bq. testTokenCompatibilityOldRenewer
Your comment does makes sense.
Let me see if some other test does something smart to get around this problem.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414729#comment-16414729
]
Xiao Chen edited comment on HADOOP-14445 at 3/26/18 11:01 PM:
--
[^HADOOP-14445.09.patch] should address all comments from Rushabh, exceptions
below.
Regarding TestKMS,
{quote} 2. providersCreated:
{quote}
I disagree because even in tests we should code against interface. It's
implementation detail that {{createProvider}} only returns the KMSCP subclass
of KeyProvider, and the Test code should just handle KeyProvider for cleanness.
This has been split out to HADOOP-15313 to limit the scope, let's move further
discussions there or feel free to file follow-ons.
Agreed {{LoadBalancingKMSCP#close}} should throw instead of swallow - feels
like a bug. Created HADOOP-15344 for that.
{quote}4. testTokenCompatibilityOldRenewer
{quote}
The reason for not choosing a shorter amount of time is after the renewal, we
want to authenticate using that token to all KMS instances. While a small renew
interval would mean less wait, it also poses higher risks of flaky test
failures if the authentication did not run within that time. Jenkins slaves are
usually unreliable. Ideally one should find a way to haul into the secret
manager, and change intervals from the test - but that seems pretty messy to do
so left as-is. Let me know what you think.
Also updated the test to verify it actually works with every KMCSP inside
LBKMSCP.
was (Author: xiaochen):
[^HADOOP-14445.09.patch] should address all comments from Rushabh, exceptions
below.
Regarding TestKMS,
bq. 2. providersCreated:
I disagree because even in tests we should code against interface. It's
implementation detail that {{createProvider}} only returns the KMSCP subclass
of KeyProvider, and the Test code should just handle KeyProvider for cleanness.
Agreed {{LoadBalancingKMSCP#close}} should throw instead of swallow - feels
like a bug. Created HADOOP-15344 for that.
bq. 4. testTokenCompatibilityOldRenewer
The reason for not choosing a shorter amount of time is after the renewal, we
want to authenticate using that token to all KMS instances. While a small renew
interval would mean less wait, it also poses higher risks of flaky test
failures if the authentication did not run within that time. Jenkins slaves are
usually unreliable. Ideally one should find a way to haul into the secret
manager, and change intervals from the test - but that seems pretty messy to do
so left as-is. Let me know what you think.
Also updated the test to verify it actually works with every KMCSP inside
LBKMSCP.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397866#comment-16397866
]
Xiao Chen edited comment on HADOOP-14445 at 3/14/18 10:59 PM:
--
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but created HADOOP-15313 for cleanness..
All other comments are addressed in patch 7, and good catch on the duplicate
test method. Indeed client versions are hard to manage - the config is only a
way to not duplicate tokens once we're sure everything is upgraded. I added
more text into core-default.xml to explain, and will add similar lines to
release notes once this is in. Didn't add to documentation because I fear this
would confuse average users when they see that from documentation...
was (Author: xiaochen):
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but created HADOOP-15313 for cleanness..
All other comments are addressed in, and good catch on the duplicate test
method. Indeed client versions are hard to manage - the config is only a way to
not duplicate tokens once we're sure everything is upgraded. I added more text
into core-default.xml to explain, and will add similar lines to release notes
once this is in. Didn't add to documentation because I fear this would confuse
average users when they see that from documentation...
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397866#comment-16397866
]
Xiao Chen edited comment on HADOOP-14445 at 3/13/18 11:48 PM:
--
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but created HADOOP-15313 for cleanness..
All other comments are addressed in, and good catch on the duplicate test
method. Indeed client versions are hard to manage - the config is only a way to
not duplicate tokens once we're sure everything is upgraded. I added more text
into core-default.xml to explain, and will add similar lines to release notes
once this is in. Didn't add to documentation because I fear this would confuse
average users when they see that from documentation...
was (Author: xiaochen):
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but created HADOOP-13513 for cleanness..
All other comments are addressed in, and good catch on the duplicate test
method. Indeed client versions are hard to manage - the config is only a way to
not duplicate tokens once we're sure everything is upgraded. I added more text
into core-default.xml to explain, and will add similar lines to release notes
once this is in. Didn't add to documentation because I fear this would confuse
average users when they see that from documentation...
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397866#comment-16397866
]
Xiao Chen edited comment on HADOOP-14445 at 3/13/18 11:48 PM:
--
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but created HADOOP-13513 for cleanness..
All other comments are addressed in, and good catch on the duplicate test
method. Indeed client versions are hard to manage - the config is only a way to
not duplicate tokens once we're sure everything is upgraded. I added more text
into core-default.xml to explain, and will add similar lines to release notes
once this is in. Didn't add to documentation because I fear this would confuse
average users when they see that from documentation...
was (Author: xiaochen):
Thanks for the review [~jojochuang], good comments! Also looking forward to
[~daryn]'s review. Appreciate the review cycles.
bq. Why was KerberosConfiguration removed in the patch?
I was confused when adding tests and found that it's not used anywhere. Added
it back, can have the removal done in a separate jira for cleanness.
bq. close the KeyProviders in TestKMS... in the initial test code ...
Good catch. I think this was missed in day 0 tests. Handled in this patch for
review convenience, but will create a separate jira for it.
All other comments are addressed in, and good catch on the duplicate test
method. Indeed client versions are hard to manage - the config is only a way to
not duplicate tokens once we're sure everything is upgraded. I added more text
into core-default.xml to explain, and will add similar lines to release notes
once this is in. Didn't add to documentation because I fear this would confuse
average users when they see that from documentation...
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397069#comment-16397069
]
Wei-Chiu Chuang edited comment on HADOOP-14445 at 3/13/18 2:58 PM:
---
Thanks a lot [~xiaochen] [~shahrs87], [~daryn], [~yzhangal] and [~asuresh] 's
valuable comments and patch coding. This is a tricky case and it impacts a
number of our customers (as well as partner software). I haven't pay attention
to the development of this Jira but just synced up with [~xiaochen] on the
latest patch. The approach totally makes sense to me.
I am reviewing the latest patch and will post review comments later today(-ish).
was (Author: jojochuang):
Thanks a lot [~xiaochen] [~shahrs87], [~daryn], [~yzhangal] and [~asuresh] 's
valuable comments and patch coding. This is a tricky case and it impacts a
number of our customers (as well as partner software). I haven't pay attention
to the development of this Jira but synced up with [~xiaochen] on the latest
patch. Totally makes sense to me.
I am reviewing the latest patch and will post review comments later today(-ish).
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337925#comment-16337925
]
Xiao Chen edited comment on HADOOP-14445 at 1/25/18 5:07 AM:
-
Thanks for the work here Rushabh, latest patch looks really promising!
Appreciate the thorough test coverage.
I also ran some more tests with this patch on a test cluster, and things look
good. The binary upgrade is compatible, and the format switch is done with an
additional config deployment. This is more or less inline with what [~daryn]
suggested above I think.
Some comments in latest patch, mostly nits:
* We need some sort of formal documentation on upgrading -> changing the
config. Either in docs or a detailed release note would do.
* KMSCP: Please rename the variables to be the common hadoop naming convention
(xxx_KEY, xxx_DEFAULT)
* The above config should be added to core-default.
* KMSCP: typo, "then he value is read from" s/he value/the value/g
* KMSCP: There are some extra line breaks after createKMSAuthenticatedURL.
* KMSCP: {{containsKmsDt}}'s existing log isn't accurate. Suggest to move that
to after the {{getTokenFromCredentials}} call, and log out what token it
actually got.
* TestLoadBalancingKMSClientProvider.java: I think we don't need to set the
injector during @BeforeClass
* TestKMS: some unused variables can be removed: line 2606 {{renewed}}, line
2653 {{kp1}}.
My 'key operations' comments meant to say we actually use the token to do
something. You happen to add the same in the new tests. :) So overall I'm +1
pending the above.
was (Author: xiaochen):
Thanks for the work here Rushabh, latest patch looks really promising!
Appreciate the thorough test coverage.
I will run some more tests with this patch on a test cluster, and will comment
back.
Some comments in latest patch, mostly nits:
* We need some sort of formal documentation on upgrading -> changing the
config. Either in docs or a detailed release note would do.
* KMSCP: Please move the new config to KMSConfiguration.java. Also rename the
variables to be the common hadoop naming convention (xxx_KEY, xxx_DEFAULT)
* KMSCP: typo, "then he value is read from" s/he value/the value/g
* KMSCP: There are some extra line breaks after createKMSAuthenticatedURL.
* KMSCP: {{containsKmsDt}}'s existing log isn't accurate. Suggest to move that
to after the {{getTokenFromCredentials}} call, and log out what token it
actually got.
* TestLoadBalancingKMSClientProvider.java: I think we don't need to set the
injector during @BeforeClass
* TestKMS: some unused variables can be removed: line 2606 {{renewed}}, line
2653 {{kp1}}.
My 'key operations' comments meant to say we actually use the token to do
something. You happen to add the same in the new tests. :)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch, HADOOP-14445.003.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269164#comment-16269164
]
Xiao Chen edited comment on HADOOP-14445 at 11/28/17 6:05 PM:
--
Thanks for the offering [~shahrs87], totally fine if you want to update the
patch - I just want momentum on this. :)
For compatibility, I think we need:
- old client works with new format token
- old client renewer works with new token
- new client works with old format token
- new client renewer works with old format token (the RM case you commented)
This is rather exhaustive, but IMO necessary to support rolling upgrade and
clients that talks to multiple clusters.
I'll review HDFS-12355 and related jiras.
was (Author: xiaochen):
Thanks for the offering [~shahrs87], totally fine if you want to update the
patch - I just want momentum on this. :)
For compatibility, I think we need:
- old client works with new format token
- old client renewer works with new token (the RM case you commented)
- new client works with old format token
- new client renewer works with old format token
This is rather exhaustive, but IMO necessary to support rolling upgrade and
clients that talks to multiple clusters.
I'll review HDFS-12355 and related jiras.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033046#comment-16033046
]
Rushabh S Shah edited comment on HADOOP-14445 at 6/1/17 3:32 PM:
-
bq. With either the current patch of the Arun's #2 solution, we can address
this problem, so this seems an orthogonal issue, no?
Let me understand Arun's#2 proposed fix.
For example: if the loadbalancing kms url is
{{kms://ht...@kms01.example.com;kms02.example.com:9600/kms}}
After applying Arun's#2 proposed fix, what will the token map look like ?
1.
||TokenService||Token||
|kms://ht...@kms01.example.com:9600/kms|t1|
|kms://ht...@kms02.example.com:9600/kms|t1|
or
2.
||TokenService||Token||
|{{kms01.example.com}}'s ip:9600|t1|
|{{kms02.example.com}}'s ip:9600|t1|
[~asuresh] [~yzhangal]
was (Author: shahrs87):
bq. With either the current patch of the Arun's #2 solution, we can address
this problem, so this seems an orthogonal issue, no?
Let me understand Arun's#2 fix.
For example: if the loadbalancing kms url is
{{kms://ht...@kms01.example.com;kms02.example.com:9600/kms}}
After applying Arun's#2 fix, what will the token map look like ?
1.
||TokenService||Token||
|kms://ht...@kms01.example.com:9600/kms|t1|
|kms://ht...@kms02.example.com:9600/kms|t1|
or
2.
||TokenService||Token||
|{{kms01.example.com}}'s ip:9600|t1|
|{{kms02.example.com}}'s ip:9600|t1|
[~asuresh] [~yzhangal]
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032144#comment-16032144
]
Yongjun Zhang edited comment on HADOOP-14445 at 5/31/17 10:43 PM:
--
HI [~shahrs87],
The current format of key provider URI is:
{code}
/**
* This provider expects URIs in the following form :
* kms://@/
*
* where :
* - PROTO = http or https
* - AUTHORITY = [:]
* - HOSTS = [;]
* - HOSTNAME = string
* - PORT = integer
*
* If multiple hosts are provider, the Factory will create a
* {@link LoadBalancingKMSClientProvider} that round-robins requests
* across the provided list of hosts.
*/
{code}
Each host corresponds to one KMS, I thought we just need split the hosts part,
and make each one its own URI with the corresponding host, and use that for
tokenService key for each KMS.
So when we collect a delegation token from one KMS, we associate the token with
each KMS and the corresponding URI. It's more robust to me because the info
about each KMS is only about itself, no host concatenation in the key of map.
About
{quote}
currently RM (kms token renewer) is still dependent on local conf to find the
key provider since the current token service format is ip:port. It doesn't know
anything about the protocol (http or https).
{quote}
With either the current patch of the Arun's #2 solution, we can address this
problem, so this seems an orthogonal issue, no?
Hi [~asuresh], thanks for your earlier proposal, would you please comment here?
Thanks.
was (Author: yzhangal):
HI [~shahrs87],
The current format of kmy provider URI is:
{code}
/**
* This provider expects URIs in the following form :
* kms://@/
*
* where :
* - PROTO = http or https
* - AUTHORITY = [:]
* - HOSTS = [;]
* - HOSTNAME = string
* - PORT = integer
*
* If multiple hosts are provider, the Factory will create a
* {@link LoadBalancingKMSClientProvider} that round-robins requests
* across the provided list of hosts.
*/
{code}
Each host corresponds to one KMS, I thought we just need split the hosts part,
and make each one its own URI with the corresponding host, and use that for
tokenService key for each KMS.
So when we collect a delegation token from one KMS, we associate the token with
each KMS and the corresponding URI. It's more robust to me because the info
about each KMS is only about itself, no host concatenation in the key of map.
About
{quote}
currently RM (kms token renewer) is still dependent on local conf to find the
key provider since the current token service format is ip:port. It doesn't know
anything about the protocol (http or https).
{quote}
With either the current patch of the Arun's #2 solution, we can address this
problem, so this seems an orthogonal issue, no?
Hi [~asuresh], thanks for your earlier proposal, would you please comment here?
Thanks.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16031630#comment-16031630
]
Yongjun Zhang edited comment on HADOOP-14445 at 5/31/17 6:23 PM:
-
Thanks [~shahrs87],
As I commented earlier, [~asuresh]'s #2 proposal at
https://issues.apache.org/jira/browse/HADOOP-14441?focusedCommentId=16019893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16019893
seems more robust, and more backward compatible.
We need hadoop.kms.authentication.zk-dt-secret-manager.enable set to true
anyways.
Have you tried this approach? Any thought?
Thanks.
--Yongjun
was (Author: yzhangal):
Thanks [~shahrs87],
As I commented earlier, Arun Suresh's #2 proposal at
https://issues.apache.org/jira/browse/HADOOP-14441?focusedCommentId=16019893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16019893
seems more robust, and more backward compatible.
We need hadoop.kms.authentication.zk-dt-secret-manager.enable set to true
anyways.
Have you tried this approach? Any thought?
Thanks.
--Yongjun
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028710#comment-16028710
]
Yongjun Zhang edited comment on HADOOP-14445 at 5/30/17 5:06 PM:
-
HI [~shahrs87],
Thanks much for the patch and [~jojochuang] and [~asuresh] for the discussion
and review.
I did a review and here are my comments.
1. about new field "clientProvidedUriText"
a. typo "clientProvidedUriText" should be "clientProviderUriText"
b. make it private instead of public, and add accessor methods
c. AND provide a comment on the format of its value.
2. I would like to see a description of the solution as a detailed comment
somewhere in the code, maybe right at the new field of item #1.
3. Suggest to change
{code}
public static KeyProvider createKeyProviderFromToken(final Configuration conf,
final String tokenServiceValue) throws IOException {
{code}
to
{code}
public static KeyProvider createKeyProviderFromTokenService(final
Configuration conf,
final String tokenService) throws IOException {
{code}
4.
{code}
/**
* This method is created for testing of @LoadBalancingKMSClientProvider
* since multiple kms servers in unit tests will only differ in port number
* and multiple kms servers in real world application are designed to
* share the same port number.
* @param conf
* @param tokenServiceValue
* @return
* @throws IOException
*/
public static KeyProvider createKeyProviderFromToken(final Configuration conf,
final String tokenServiceValue) throws IOException {
if (tokenServiceValue == null) {
return null;
}
if (!tokenServiceValue.contains(",")) {
return createKeyProviderFromUri(conf, URI.create(tokenServiceValue));
}
// The following code will be executed in just unit tests.
// The syntax for kms servers will be
// kms://localhost:port1/kms,kms://localhost:port2/kms
String[] keyProviderUrisStr = tokenServiceValue.split(",");
KMSClientProvider[] keyProviderArr =
new KMSClientProvider[keyProviderUrisStr.length];
{code}
a. Comment need to be changed to say that "," only appear in tokenServiceValue
of unit tests because different ports need to be used in unit tests.
b. And indicate what format to expect for production code, and what format to
expect for unit test.
c. We can separate the code used for test only to a separate method and call
this method in the above method, to make it more clear.
5. With this solution, if we add a new kms server, the currently running
application will not be able to use the new kms server.
And if we replace an existing kms server with a different one, currently
running application may fail. So the KMS HA may not be really HA.
This may not be a big concern, but I would like to bring it up here.
Thanks.
was (Author: yzhangal):
HI [~shahrs87],
Thanks much for the patch and [~jojochuang] and [~asuresh] for the discussion
and review.
I did a review and here are my comments.
1. about new field "clientProvidedUriText"
a. typo "clientProvidedUriText" should be "clientProviderUriText"
b. make it private instead of public, and add accessor methods
c. AND provide a comment on the format of its value.
2. I would like to see a description of the solution as a detailed comment
somewhere in the code, maybe right at the new field of item #1.
3. Suggest to change
{code}
public static KeyProvider createKeyProviderFromToken(final Configuration conf,
final String tokenServiceValue) throws IOException {
{code}
to
{code}
public static KeyProvider createKeyProviderFromTokenService(final
Configuration conf,
final String tokenService) throws IOException {
{code}
3.
{code}
/**
* This method is created for testing of @LoadBalancingKMSClientProvider
* since multiple kms servers in unit tests will only differ in port number
* and multiple kms servers in real world application are designed to
* share the same port number.
* @param conf
* @param tokenServiceValue
* @return
* @throws IOException
*/
public static KeyProvider createKeyProviderFromToken(final Configuration conf,
final String tokenServiceValue) throws IOException {
if (tokenServiceValue == null) {
return null;
}
if (!tokenServiceValue.contains(",")) {
return createKeyProviderFromUri(conf, URI.create(tokenServiceValue));
}
// The following code will be executed in just unit tests.
// The syntax for kms servers will be
// kms://localhost:port1/kms,kms://localhost:port2/kms
String[] keyProviderUrisStr = tokenServiceValue.split(",");
KMSClientProvider[] keyProviderArr =
new KMSClientProvider[keyProviderUrisStr.length];
{code}
a. Comment need to be changed to say that "," only appear in tokenServiceValue
of unit tests because different ports need to be used in unit tests.
b. And indicate what
