[
https://issues.apache.org/jira/browse/HADOOP-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134745#comment-18134745
]
ASF GitHub Bot commented on HADOOP-20046:
-----------------------------------------
jojochuang opened a new pull request, #8889:
URL: https://github.com/apache/hadoop/pull/8889
### Description of PR
ZooKeeper-backed token renewal, cancellation, and master-key maintenance
hold the global API write lock during synchronous storage calls. A slow write
or batch of expired-key deletions can therefore stall unrelated cached-token
verification in KMS and HDFS Router.
This follow-up moves those storage calls outside the global write lock,
using bounded per-token lock stripes and a separate rotation coordinator.
Versioned ZooKeeper operations preserve peer renewals and prevent cancelled
tokens from being recreated. Replacement keys and retirement bounds are durable
before publication; non-ZooKeeper storage-hook locking is retained. Router
renewal no longer speculatively publishes cache state, and its cancellation
flag is reset on failure.
**Depends on #8888 (HADOOP-20045), which is still open.** This branch is
stacked on its creation fix. The follow-up is commit `bdbcd97c790`; the current
trunk comparison also includes the parent change. Rebase after #8888 merges.
This PR does not modify #8888.
[HADOOP-20046](https://issues.apache.org/jira/browse/HADOOP-20046) contains
the design, benchmark report, charts, and exported raw measurements.
### Benchmarks
The opt-in KMS HTTP harness now supports renewal, cancellation, and
verification during actual manager maintenance. One KMS JVM, one standalone
ZooKeeper JVM, 32 concurrent clients, sequence batch size 100, and no injected
storage delay. The baseline already includes HADOOP-20045, isolating this
follow-up.
Single-run HTTP checks at 95% verification / 5% mutation:
| Mutation | Parent requests/s | Fixed requests/s | Ratio | Parent verify
p99 ms | Fixed verify p99 ms |
| -
> Avoid holding the delegation-token secret manager API write lock during
> ZooKeeper renewal, cancellation, and master-key operations
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20046
> URL: https://issues.apache.org/jira/browse/HADOOP-20046
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments: HADOOP-20046-expired-key-stall.png,
> HADOOP-20046-maintenance-benchmark.md,
> HADOOP-20046-maintenance-comparison.png,
> HADOOP-20046-maintenance-results.json, HADOOP-20046-write-locks.md
>
>
> h3. Problem
> HADOOP-18851 introduced a read-write lock in
> AbstractDelegationTokenSecretManager to allow concurrent token verification.
> HADOOP-20045 addresses token creation holding that lock during ZooKeeper
> operations.
> Other paths still hold {{apiLock.writeLock()}} while performing synchronous
> ZooKeeper operations:
> || Write-lock holder || ZooKeeper operations under the lock ||
> | renewToken() | Read signing key/token information on cache misses; check
> existence and update/create token |
> | cancelToken() | Read token information, check existence, and delete token |
> | updateCurrentKey() | Check existence and persist the new master key |
> | rollMasterKey() | Check existence and persist the retiring key’s updated
> expiry |
> | removeExpiredKeys() | Check existence and delete expired master keys,
> potentially multiple keys per lock hold |
> ZooKeeper latency or retries in these paths can block unrelated requests that
> need the API read lock to verify cached tokens. This affects KMS and other
> consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
> h3. Proposed improvement
> Move ZooKeeper operations outside the global API write-lock sections. Retain
> short critical sections for inspecting or publishing local state, with
> explicit coordination to preserve operation ordering.
> The design must ensure:
> * Renewal cannot restore a token after cancellation or overwrite a newer
> renewal.
> * Cache publication and storage-failure handling remain consistent with the
> operation’s outcome.
> * A new master key is persisted before it becomes available for signing
> tokens.
> * Retiring-key updates and expired-key deletion remain correctly ordered with
> rotation and issuance.
> * Existing storage-hook behavior for non-ZooKeeper implementations is
> preserved.
> Per-token coordination, a separate key-rotation coordinator, and versioned
> ZooKeeper operations are potential mechanisms to evaluate. Simply releasing
> the global lock around existing storage calls is insufficient.
> h3. Validation
> Add deterministic tests that pause each relevant ZooKeeper operation and
> confirm that verification of an unrelated cached token continues. Add
> concurrency and failure tests covering renewal versus cancellation,
> concurrent renewals, key rotation versus issuance, interrupted operations,
> and persistence failures.
> Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
> cancellation contention. Exercise key rotation and expired-key cleanup
> separately. Compare throughput and verification tail latency against the
> original implementation, using real ZooKeeper operations without injected
> delay for performance measurements.
> h3. Scope
> This issue specifically tracks ZooKeeper operations executed while holding
> {{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
> CPU-only cleanup scans are outside its scope.
> Implementation and real-ZooKeeper benchmarks are now available. The attached
> design describes token lock stripes, versioned renewal/deletion, durable
> master-key publication, and compatibility with non-ZooKeeper stores.
> Measurements include renewal/cancellation HTTP workloads and 24 validated
> background-maintenance cases (four scenarios, two builds, three repetitions).
> See the implementation/benchmark comment and attached report, charts, and raw
> results for methods, validation, and limitations.
> h3. Related issues
> * [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
> * [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
