jojochuang opened a new pull request, #8889: URL: https://github.com/apache/hadoop/pull/8889
### Description of PR ZooKeeper-backed token renewal, cancellation, and master-key maintenance hold the global API write lock during synchronous storage calls. A slow write or batch of expired-key deletions can therefore stall unrelated cached-token verification in KMS and HDFS Router. This follow-up moves those storage calls outside the global write lock, using bounded per-token lock stripes and a separate rotation coordinator. Versioned ZooKeeper operations preserve peer renewals and prevent cancelled tokens from being recreated. Replacement keys and retirement bounds are durable before publication; non-ZooKeeper storage-hook locking is retained. Router renewal no longer speculatively publishes cache state, and its cancellation flag is reset on failure. **Depends on #8888 (HADOOP-20045), which is still open.** This branch is stacked on its creation fix. The follow-up is commit `bdbcd97c790`; the current trunk comparison also includes the parent change. Rebase after #8888 merges. This PR does not modify #8888. [HADOOP-20046](https://issues.apache.org/jira/browse/HADOOP-20046) contains the design, benchmark report, charts, and exported raw measurements. ### Benchmarks The opt-in KMS HTTP harness now supports renewal, cancellation, and verification during actual manager maintenance. One KMS JVM, one standalone ZooKeeper JVM, 32 concurrent clients, sequence batch size 100, and no injected storage delay. The baseline already includes HADOOP-20045, isolating this follow-up. Single-run HTTP checks at 95% verification / 5% mutation: | Mutation | Parent requests/s | Fixed requests/s | Ratio | Parent verify p99 ms | Fixed verify p99 ms | | --- | ---: | ---: | ---: | ---: | ---: | | Renewal | 3,748 | 22,258 | 5.94× | 18.60 | 5.86 | | Cancellation | 4,261 | 28,056 | 6.58× | 18.05 | 1.71 | Verification-only HTTP traffic during background maintenance, median of three runs per build (24 accepted cases): | Maintenance batch | Parent active verify/s | Fixed active verify/s | Parent / fixed duration, s | | --- | ---: | ---: | ---: | | 500 new-key installations | 13,739 | 29,025 | 3.850 / 6.054 | | 500 rotations (retiring + replacement key) | 11,471 | 26,208 | 6.062 / 6.144 | | Delete 1,000 expired keys | 8 | 26,650 | 3.858 / 3.230 | | Delete 5,000 expired tokens | 29,782 | 31,589 | 23.542 / 18.226 |   These are closed-loop stress workloads, not production maintenance cadence or matched-arrival-rate latency measurements. Active maintenance intervals differ; the attached report also gives whole-window throughput and full latency statistics. New-key installation in the fixed build additionally persists a retirement bound. Expired-token deletion was already outside the global write lock in the parent; that scenario measures changed cleanup coordination and versioned deletion. Tail latency does not improve uniformly: median per-run maximum verification latency rose from 34.79 to 126.25 ms during new-key installation and from 35.12 to 179.47 ms during rotation; rotation p99 rose from 7.15 to 9.64 ms. Expired-key deletion provides the clearest stall removal (overlap p99 3891.30 → 10.98 ms). Three repetitions do not establish significance for small differences. The report records rejected attempts and retries; only validated cases enter the summaries. ### How was this patch tested? Fresh JDK 17 / Maven 3.9.15 runs with `-Dmaven.test.failure.ignore=false`: - Common: `TestDelegationToken`, `TestZKDelegationTokenCreation`, `TestZKDelegationTokenSecretManager`, and `TestZKDelegationTokenMutations`: 51 passed, one opt-in benchmark skipped. - Router: `TestZKDelegationTokenSecretManagerImpl`: 16 passed. - KMS: `TestKMSAuthenticationFilter` and `TestKMSHttpBenchmark`: four passed, one opt-in benchmark skipped. - Deterministic storage-blocking regressions cover renewal, cancellation, new-key persistence, retirement persistence, and expired-key deletion. Additional cases cover peer cancellation, monotonic renewals, local serialization, failed/interrupted persistence, and conditional cache publication. The five blocking cases previously reproduced verification timeouts against the parent implementation. - All 24 accepted maintenance cases passed loaded-class fingerprints, authentication preflight, zero HTTP/validation errors, exact storage-node changes, and maintenance completion within the load window. Both HTTP mutation pairs also passed their validation. - Python syntax checks and `git diff --check` passed. Benchmarks remain opt-in; normal tests do not run the capacity suite. ### For code changes - [x] Title starts with the Jira issue ID. - No new dependencies or license-file changes. - Read-lock cache misses and immediate cross-manager revocation are outside this change's scope. ### AI Tooling Contains content generated by OpenAI Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
