Siyao Meng created HDDS-17176:
---------------------------------
Summary: Delegation token is still accepted after its max date
when the renew interval is longer than the max lifetime
Key: HDDS-17176
URL: https://issues.apache.org/jira/browse/HDDS-17176
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-6-reject-delegation-token-past-max-date.patch,
TestBugMC6TokenAcceptedPastMaxDate.java
h3. Mechanism
When a token is issued, {{OzoneDelegationTokenSecretManager.updateToken}}
stores {{issueDate + renewInterval}} as its renew date without capping it at
the token's {{maxDate}}. {{OzoneDelegationTokenSecretManager.validateToken}},
which {{retrievePassword}} calls for every token login, checks only that renew
date and never {{maxDate}}. {{renewToken}} does both: it refuses a token past
{{maxDate}} with {{TOKEN_EXPIRED}} and caps the new expiry with
{{Math.min(maxDate, ...)}}. {{OzoneManager.createDelegationTokenSecretManager}}
does not check {{ozone.manager.delegation.token.renew-interval}} against
{{ozone.manager.delegation.token.max-lifetime}}, so OM starts with a renew
interval longer than the max lifetime without a warning.
Hadoop's {{AbstractDelegationTokenSecretManager}} and HDFS (3.4.3) have the
same behaviour and no check of the two values either, although
{{hdfs-default.xml}} describes max-lifetime as the lifetime for which a token
is valid. In Ozone, the {{ozone.manager.delegation.token.max-lifetime}}
description and the startup check added by HDDS-13343 size the secret key
lifetime on the assumption that a token is not used after max-lifetime.
h3. Trigger
# Configure {{ozone.manager.delegation.token.max-lifetime}} below
{{ozone.manager.delegation.token.renew-interval}}, for example below the 1d
default renew interval. OM starts normally.
# Get a delegation token.
# After its {{maxDate}}, {{renewDelegationToken}} fails with {{TOKEN_EXPIRED}},
but a client holding the token still logs in with it until {{issueDate +
renewInterval}} (or until signature verification fails because the secret key
is gone, whichever is earlier).
Default configs (1d renew interval, 7d max lifetime) do not trigger it.
Lowering max lifetime is the natural way to get there: the startup check from
HDDS-13343 requires the secret key expiry to cover max lifetime plus rotation
and scan intervals, so shortening max lifetime is the knob that satisfies it
while the renew interval stays at 1d. Several in tree secure integration tests
run with such a setting, for example {{TestSecureOzoneCluster}} (9 s),
{{TestDelegationToken}} (60 s), {{TestSecretKeysApi}} (1500 ms) and
{{TestContainerCommandReconciliation}} (300 s), with the renew interval left at
its default.
h3. Impact
A token authenticates past its documented maximum lifetime, up to the
configured renew interval. The stored renew date is persisted and reloaded on
restart, and the expired token remover keys on it, so the window lasts until
that renew date unless the signing secret key is no longer available (for
example, after an OM restart a token whose key SCM has already dropped as
expired is not loaded). Only clusters with max lifetime below the renew
interval are affected.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC6TokenAcceptedPastMaxDate.java] ({{hadoop-ozone/ozone-manager}})
builds a real secret manager with max lifetime 1 s and renew interval 1 h,
issues a token with {{createToken}} and registers it with {{updateToken}} as
{{OMGetDelegationTokenRequest}} does on apply. After {{maxDate}},
{{renewToken}} fails with {{TOKEN_EXPIRED}} while {{retrievePassword}} still
returns the token's password. A passing test means the defect is present; with
the patch the {{retrievePassword}} step throws {{InvalidToken}}. The full token
login over RPC was not exercised by this test; {{retrievePassword}} is the
method the RPC server's SASL handler calls for it.
h3. Patch
[^MC-6-reject-delegation-token-past-max-date.patch], against
b94df213f20ec0d675f079b57a4566e4e64e4060. It also applies to master at
df42f7f60a2.
{{validateToken}} now rejects a token whose {{maxDate}} has passed, before the
renew date check. This also covers tokens issued before the upgrade, which keep
their uncapped renew date in the store; capping in {{updateToken}} alone would
not. The patch deliberately does not cap the stored renew date: that value is
written when the request is applied, so OMs of different versions would store
different values during a rolling upgrade, and it would not help tokens already
stored. A token past its {{maxDate}} therefore stays in the token table,
unusable, until its stored renew date, when the remover deletes it as today.
With renew interval <= max lifetime the stored renew date never exceeds
{{maxDate}}, so nothing changes for those configs. Rejecting such a config at
startup was not chosen because OMs that start today would then refuse to start;
a startup warning could be added separately. The
{{ozone.manager.delegation.token.max-lifetime}} description in
{{ozone-default.xml}} now says the token is not renewed or accepted after that
interval.
Covered by {{testRetrievePasswordFailureMaxTime}} in the existing
{{TestOzoneDelegationTokenSecretManager}}, which checks both a live token and
one reloaded from the store after a restart. Without the change it fails
because {{retrievePassword}} returns the password. With it the secret manager,
token identifier and get delegation token request and response suites (30
tests) and the secure mini cluster suite {{TestDelegationToken}} pass,
checkstyle is clean.
Found by TLA+ model checking and code review of the OM delegation token secret
manager and request paths under HDDS-15926, on commit
b94df213f20ec0d675f079b57a4566e4e64e4060. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
