[
https://issues.apache.org/jira/browse/HDFS-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104682#comment-18104682
]
ASF GitHub Bot commented on HDFS-17964:
---------------------------------------
greenwich opened a new pull request, #8685:
URL: https://github.com/apache/hadoop/pull/8685
### Description of PR
`DelegationTokenSecretManager.retriableRetrievePassword` (RPC/SASL) and
`FSNamesystem.verifyToken` (WebHDFS, via `JspHelper.getUGI`) wrap
`InvalidToken`
in a `RetriableException` only when `inTransitionToActive()`. An Observer
NameNode in steady state never satisfies that, so a token the active issued
moments earlier - whose editlog entry the observer has not yet tailed - is
rejected fatally instead of retried. HDFS-5322 added that guard in 2.3.0,
before Observer NameNodes existed (HDFS-12943); it was never widened.
Both sites now also treat an observer as possibly behind on tailing. A
standby
is unaffected: `checkOperation(READ)` rejects it before the lookup.
Two behavioural notes worth stating explicitly:
- A password mismatch is deliberately **not** made retriable - it cannot be a
staleness artifact, since every namenode recomputes the same password from
the same master key. This also makes a mismatch fatal while transitioning
to
active, where it was previously retriable: a small scope increase over
HDFS-5322, and the correct behaviour.
- Expiry **is** retriable, because `OP_RENEW_DELEGATION_TOKEN` is journaled
and
an observer that has not tailed a renewal reports a live token as expired.
The cost is that a genuinely expired token on an observer now returns
`RetriableException`, so a WebHDFS client no longer takes the
`InvalidToken`
path into `replaceExpiredDelegationToken()` and instead exhausts its retry
budget. Bounded, and outweighed by the untailed-renewal case this fixes.
Contains content generated by Claude Code.
### How was this patch tested?
New `TestDelegationTokensWithObserver`: 11 tests on a real 3-NameNode
observer
cluster (`HATestUtil.setUpObserverCluster`), no mocks. Covers both entry
points
across active/standby/observer, both happy paths, a forged password, and two
real-token cases - a token the observer has not tailed, and one whose
journaled
renewal it has not tailed.
Negative checks:
- Reverting both guards fails exactly 2 tests, one per code path, with
`InvalidToken: Token for real user: ..., can't be found in cache`.
- Reverting only the password narrowing fails exactly the forged-password
test.
Existing suites pass: `TestDelegationToken`, `TestDelegationTokensWithHA`,
`TestObserverNode`, `TestConsistentReadsObserver` (61 tests total).
RAT, spotbugs, checkstyle and javadoc clean.
As with HDFS-5322, which tested its `verifyToken` branch at this same
in-process
layer, there is no client-level end-to-end test; the RPC and WebHDFS retry
chains were verified by inspection.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(HDFS-17964?
- [ ] Object storage: N/A - no object store code touched.
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under ASF 2.0? N/A - no new
dependencies.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? N/A - no dependency changes.
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> HDFS delegation token not found in cache errors seen on Observer NameNodes
> --------------------------------------------------------------------------
>
> Key: HDFS-17964
> URL: https://issues.apache.org/jira/browse/HDFS-17964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Reporter: Aleksei Ieshin
> Priority: Major
>
> Observer NameNodes fatally reject delegation tokens they have not yet tailed.
> *DelegationTokenSecretManager.retriableRetrievePassword* and
> *FSNamesystem.verifyToken* wrap *InvalidToken* in a *RetriableException* only
> when *inTransitionToActive()*. An observer in steady state never is, so a
> token the active issued moments ago fails with
> {code:java}
> InvalidToken: Token for real user: <user>, can't be found in cache
> {code}
> HDFS-5322 added that guard in 2.3.0, before Observer NameNodes existed
> (HDFS-12943). Widening it to cover observers fixes both paths.
> h6. Details:
> The common trigger is *WebHDFS* *op=OPEN*, the one operation with a second
> hop: the NameNode redirects to a DataNode, and that DataNode validates the
> token against a NameNode chosen by its own *ConfiguredFailoverProxyProvider*.
> Nothing the client configures influences that choice, so in a three-NameNode
> HA setup with one observer a substantial fraction of reads land on the
> observer and fail with HTTP 403.
>
>
>
>
>
> Fix: treat an observer as possibly behind on tailing at both sites, so a
> failed lookup yields *RetriableException*. Callers already handle it - that
> is what HDFS-5322 relies on. No config, no new mechanism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
