[
https://issues.apache.org/jira/browse/HADOOP-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116662#comment-18116662
]
ASF GitHub Bot commented on HADOOP-19966:
-----------------------------------------
Hexiaoqiao commented on PR #8688:
URL: https://github.com/apache/hadoop/pull/8688#issuecomment-5727049803
> @Hexiaoqiao Token verify/renew/cancel are not affected: `getTokenInfo` and
`getDelegationKey` fall back to ZK on a cache miss. The in-memory load exists
for the expired-token remover, which only scans `currentTokens`
([HADOOP-13487](https://issues.apache.org/jira/browse/HADOOP-13487)). That
relied on PathChildrenCache `BUILD_INITIAL_CACHE` being synchronous;
[HADOOP-17835](https://issues.apache.org/jira/browse/HADOOP-17835) (3.4.0)
switched to `CuratorCache.start()`, which is asynchronous. The listener still
fills the map a few seconds later, so the lasting effects are: token owner
stats (Router `getTopTokenRealOwners`) are computed from a partial map and
never corrected for pre-restart tokens, the root container znode is parsed on
every startup and logs an ERROR, and `testNodesLoadedAfterRestart` is flaky in
CI. Router with the token watcher disabled uses its own synchronous
`rebuildTokenCache` and is unaffected. Retitled the PR and rewrote the
description accordingly.
Got it. LGTM. Give my +1 for this PR. Thanks.
> ZKDelegationTokenSecretManager startup cache load races with CuratorCache
> initialization
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-19966
> URL: https://issues.apache.org/jira/browse/HADOOP-19966
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Cheng Pan
> Priority: Major
> Labels: pull-request-available
>
> Since HADOOP-17835 (3.4.0), ZKDelegationTokenSecretManager starts its
> CuratorCache asynchronously and reads it immediately, so loadFromZKCache runs
> against an empty or partial cache. Token verify/renew/cancel are unaffected
> because they fall back to ZooKeeper, and the cache listener fills the
> in-memory map a few seconds later. The lasting effects are:
> * token owner stats (Router getTopTokenRealOwners) are computed from a
> partial map and never corrected for pre-restart tokens;
> * the root container znode, which carries Curator's default data (the local
> address), is parsed as a key/token on every startup and logs an ERROR;
> * testNodesLoadedAfterRestart is flaky in CI.
> Fix: wait for CuratorCache initialization (forInitialized) before loading
> each cache, skip the root container znodes when processing cache events and
> when streaming the cache, create the root znodes with explicit empty data,
> and release the caches, counters and Curator client when startThreads() fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
