[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035306#comment-16035306
]
Daryn Sharp commented on HADOOP-14445:
--------------------------------------
I had a feeling "nameservice" alluded to the hdfs HA configuration – which is
horrible for the reasons I've detailed and why we don't use it. I'll politely
stress and repeat: *Updating configs of tens of thousands of nodes, launchers,
oozie, storm, spark, etc and restarting the services is just not logistically
possible*.
bq. Although, IIRC, the tokens are renewed only if they are expired, so if they
are renewed serially, it should not be a problem.
The RM renews immediately to verify token validity and to determine the next
renewal time. If they are expired, it's too late. Any kp token using just a
service authority cannot determine the kp uri and is only renewable via the kp
uri in the config – enforcing one and only 1 kms cluster. If the kp client can
be instantiated via the service, then multi-kms setups are possible.
bq. I do like the idea of using a nameservice though, as Yongjun Zhang
suggested which will ensure that we will still have only 1 single entry.
There must be a disconnect here. 1 single entry is the advantage of just
setting the service to the provider uri. Adding an extra layer of indirection
through the config creates a logistical mess with no added benefits. I'm not
going to bounce all my services and RMs because I added or changed a KMS
cluster.
Here's the big picture we are trying to achieve:
* client requests kp uri from NN
* client creates kp client from kp uri
* client gets tokens and sets service to kp uri
* RM calls kms token renewer which uses kp uri in service to create kp client
* tasks use the NN->kp uri mapping established at job submission to locate
tokens
It's +config-less+ other than a setting on the NN. This is what we are running
internally because the current kms client design is completely broken. We now
have the ability to enable EZs on a NN and/or change kms cluster configuration
without changing configs or restarting services.
We only care about this load balancing provider because we need to ensure the
kp client can be instantiated from the service.
> Delegation tokens are not shared between KMS instances
> ------------------------------------------------------
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
> Affects Versions: 2.8.0, 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
