[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Chen updated HADOOP-13487: ------------------------------- Attachment: HADOOP-13487.01.patch After further looking into this, I think we can simply go with option #1 above. On thread startup, {{PathChildrenCache}} need to load the znode anyways, which is the most time consuming operation. Patch 1 to express the idea, I will test it in a test cluster and update here. Benchmarked with 100k existing expired znodes, while kms start up takes minutes, the new node running in memory take about 2 seconds, which I think is fine. I intentionally ignored exceptions for compatibility - if the directory contains some znodes that can't be understood by ZKDTSM, KMS should still be able to start and run as normal. > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > --------------------------------------------------------------------- > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms > Affects Versions: 2.6.0 > Reporter: Alex Ivanov > Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org