[
https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419308#comment-15419308
]
Alex Ivanov commented on HADOOP-13487:
--------------------------------------
[~xiaochen], here are some more details about the issue, and thank you for
looking into it!
ZK dtoken config for KMS (included for completeness):
{code}
<!-- Configuration to store delegation tokens in zookeeper -->
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.enable</name>
<value>true</value>
<description>
Enables storage of delegation tokens in Zookeeper.
</description>
</property>
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.znodeWorkingPath</name>
<value>hadoop-kms-dt</value>
<description>
The znode path where KMS will store delegation tokens.
</description>
</property>
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.zkConnectionString</name>
<value>HOST1:2181,HOST2:2181,HOST3:2181</value>
<description>
The Zookeeper connection string: a list of hostnames and quorum port
comma-separated.
</description>
</property>
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType</name>
<value>sasl</value>
<description>
The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
</description>
</property>
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.kerberos.keytab</name>
<value>/etc/hadoop-kms/conf/kms.keytab</value>
<description>
The absolute path for the Kerberos keytab with the credentials to
connect to Zookeeper.
</description>
</property>
<property>
<name>hadoop.kms.authentication.zk-dt-secret-manager.kerberos.principal</name>
<value>kms/HOST@BIGDATA</value>
<description>
The Kerberos service principal used to connect to Zookeeper.
</description>
</property>
<!-- KMS delegation token configuration
Extend the lifetime of delegation tokens to support SPAS -->
<property>
<name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
<value>1209600</value>
<description>
How often the master key is rotated, in seconds. Set to 2 weeks.
</description>
</property>
<property>
<name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
<value>2419200</value>
<description>
Maximum lifetime of a delagation token, in seconds. Set to 4 weeks.
</description>
</property>
<!-- Due to a bug in ZKDelegationTokenSecretManager.java (CDH 5.5.1), this
needs to be in millis -->
<!-- https://issues.apache.org/jira/browse/HADOOP-12659 -->
<property>
<name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
<value>1209600000</value>
<description>
Renewal interval of a delegation token, in seconds. Set to 2 weeks.
</description>
</property>
<property>
<name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
<value>3600</value>
<description>
Scan interval to remove expired delegation tokens.
</description>
</property>
{code}
Since I set *delegation-token.renew-interval.sec* to 2 weeks, I expect the
tokens to be invalid after that time (NOTE: I account for HADOOP-12659
specifying the time in millis). There is no process renewing the tokens right
now, but even if they were renewed, the maximum lifetime would be 4 weeks based
on the setting.
If I use *zkCli* to connect to one of the ZK servers, I see there are many
delegation tokens (NOTE: I ran all commands today, 08/12/2016):
{code}
[zk: HOST:2181(CONNECTED) 0] stat /hadoop-kms-dt/ZKDTSMRoot/ZKDTSMTokensRoot
cZxid = 0x1002395a5
ctime = Mon Jun 13 21:29:02 UTC 2016
mZxid = 0x1002395a5
mtime = Mon Jun 13 21:29:02 UTC 2016
pZxid = 0x100501d21
cversion = 109499
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 11
numChildren = 103229
{code}
As you can see, there are over 100k dtokens in that znode. Here's a sample old
delegation token from June 29th:
{code}
[zk: HOST:2181(CONNECTED) 2] get
/hadoop-kms-dt/ZKDTSMRoot/ZKDTSMTokensRoot/DT_20000
adminyarnoozie�U��V&�V+�f&�N U�^&�DJ�=��}ؒ�R����
cZxid = 0x10029f135
ctime = Wed Jun 29 09:38:40 UTC 2016
mZxid = 0x10029f135
mtime = Wed Jun 29 09:38:40 UTC 2016
pZxid = 0x10029f135
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 75
numChildren = 0
{code}
Note that the renewal time is NOT available to see in the {{zkCli}} console
output. I had to write a small program to extract this datum from the znode.
Here's the output of the custom program:
{code}
>> ReadDelTokenFromZK 20000
DT renew date: 1468402720294
{code}
1468402720294 = GMT: Wed, 13 Jul 2016 09:38:40.294 GMT
As you can see, the renewal date corresponds to the interval I've specified,
i.e. 2 weeks (June 29th - July 13th).
The only problem is, it is August 12th today, and the dtoken is still there,
which leads me to believe KMS is NOT cleaning up old tokens.
> Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
> ---------------------------------------------------------------------
>
> Key: HADOOP-13487
> URL: https://issues.apache.org/jira/browse/HADOOP-13487
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.6.0
> Reporter: Alex Ivanov
>
> Configuration:
> CDH 5.5.1 (Hadoop 2.6+)
> KMS configured to store delegation tokens in Zookeeper
> DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties
> Findings:
> It seems to me delegation tokens never get cleaned up from Zookeeper past
> their renewal date. I can see in the logs that the removal thread is started
> with the expected interval:
> {code}
> 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting
> expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> {code}
> However, I don't see any delegation token removals, indicated by the
> following log message:
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager
> --> removeStoredToken(TokenIdent ident), line 769 [CDH]
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Removing ZKDTSMDelegationToken_"
> + ident.getSequenceNumber());
> }
> {code}
> Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't
> get cleaned up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
