Siyao Meng created HDDS-17179:
---------------------------------
Summary: OM skips the secret key expiry check before symmetric
token signing is finalized, so a runtime finalize issues tokens that fail early
Key: HDDS-17179
URL: https://issues.apache.org/jira/browse/HDDS-17179
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch,
TestBugCR2FinalizeSkipsSecretKeyGuard.java
h3. Mechanism
HDDS-13343 added a check to
{{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start unless
{{hdds.secret.key.rotate.duration}} +
{{ozone.manager.delegation.token.max-lifetime}} +
{{ozone.manager.delegation.remover.scan.interval}} is at most
{{hdds.secret.key.expiry.duration}}, so that no SCM secret key expires while a
token it signed can still be renewed. The check only runs when
{{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is already finalized at construction; an OM
that is not finalized only checks the certificate grace period.
{{OzoneDelegationTokenSecretManager.createToken}} checks
{{isAllowed(DELEGATION_TOKEN_SYMMETRIC_SIGN)}} on every call, and finalization
({{finalizeupgrade}}, then {{OMCompleteFinalizeUpgradeRequest}} and
{{OMVersionManager.finalizeUpgrade}}) does not restart OM and does not run the
check. So an OM that started before finalization with a configuration the check
would refuse signs delegation tokens with SCM secret keys from the moment
finalization completes, and the check only fires at the next OM start.
h3. Trigger
# A secure cluster that has not finalized {{DELEGATION_TOKEN_SYMMETRIC_SIGN}}
(layout 8, added in 2.0.0; for example a cluster upgraded from 1.4.x) runs this
version with a configuration that breaks the rule. The default
{{hdds.secret.key.expiry.duration}} became 9d in 2.1.0; before that it was 7d,
so an {{ozone-site.xml}} that sets it explicitly to the old default (7d with a
7d token lifetime and 1d rotation) is enough. With default values the rule
holds and nothing happens.
# An administrator runs {{ozone admin om finalizeupgrade}}. OM finalizes
without a restart.
# A client gets a delegation token, now signed with the current SCM secret key.
# SCM removes that key when it expires, while the token's max date is still in
the future (for a token signed just before the next rotation, rotation period
plus token lifetime minus key expiry, so 1d with the values above).
# Renewing the token fails ("Tampered/Invalid token", caused by "Secret verify
key ... not found"), and the token cannot recover. Authenticating with it goes
through the same signature check in
{{OzoneDelegationTokenSecretManager.retrievePassword}} (code reading, not run).
# The next OM start fails with the check's {{IllegalArgumentException}}. A
checkpoint install from the leader fails the same way, see Impact.
h3. Impact
Tokens issued between finalization and the next OM restart lose up to the last
day of their lifetime with the configuration above (computed from the formula;
the scaled run shows the same effect), which breaks long-running jobs that
renew their tokens up to the max lifetime. The OM then refuses to start until
the configuration is fixed. An OM that has finalized also terminates when it
next installs a checkpoint from the leader: {{reloadOMState}} calls
{{instantiateServices(true)}}, which runs the same check, and
{{installCheckpoint}} exits the process when the reload throws (code reading,
not run). Requires an explicit non-compliant configuration on a cluster that
finalizes this layout feature at runtime; default configurations are not
affected.
Affected versions: 2.1.0 and later (2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1), the
releases that contain the check from HDDS-13343. 2.0.0 has layout 8 but no
check at all.
h3. Reproduction
PASS on the unmodified source. [^TestBugCR2FinalizeSkipsSecretKeyGuard.java]
({{hadoop-ozone/integration-test}}) starts a secure SCM (no datanodes, safe
mode off) and a secure OM with MiniKdc, with the OM storage initialized at
{{HBASE_SUPPORT}} (layout 7) through the existing
{{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}} hook, and scaled timings
(rotate 10s, expiry 20s, token max lifetime 45s, remover interval 1s). It then
calls {{finalizeUpgrade}}, gets a token, waits until SCM has removed the
signing key, renews the token and restarts the OM. Output on the unmodified
source:
{noformat}
OM started pre-finalized, apparent version HBASE_SUPPORT (7), symmetric DT
allowed false
pre-finalize token secretKeyId='null' (RSA signed)
OM finalized at runtime, apparent version GET_KEY_INFO_REJECTS_OBS (103)
symmetric token issued=1792500182109 maxDate=1792500227109
signedBy=f068b740-... keyExpiry=1792500196294 (key expires 30815 ms before
maxDate)
SCM removed key f068b740-... at 1792500196962, token maxDate 1792500227109 is
still 30147 ms away
(OM log) SCMSecurityException: Secret verify key f068b740-... not found for
token (...)
renew of unexpired token rejected at 1792500196983 (30126 ms before maxDate,
...): TOKEN_ERROR_OTHER ... Delegation token renewal failed ...
control token signed by 4ee85a92-... renewed, new expiry 1792500241984
OM restart after runtime finalization refused: Secret key expiry duration
hdds.secret.key.expiry.duration should be greater than value of (...)
{noformat}
A passing test means the defect is present. With the patch the first OM start
already fails with the check's message.
h3. Patch
[^CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch],
against b94df213f20ec0d675f079b57a4566e4e64e4060. It also applies to master at
df42f7f60a2.
The secret key check now runs at every OM start, whether or not
{{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is finalized; the certificate grace period
check still runs only before finalization. This changes behaviour for one setup
only: an OM that is not finalized and has a configuration that breaks the rule
now refuses to start (with the same message as after finalization) instead of
starting and failing later. The {{ozone.manager.delegation.token.max-lifetime}}
description in {{ozone-default.xml}} now says that a secure Ozone Manager fails
to start if the rule does not hold. The same check also runs in the checkpoint
reload, so with the patch such an OM refuses at start instead of exiting during
a later install.
HDDS-16958 (open) changes the same check and the same {{ozone-default.xml}}
description. Each patch applies to b94df213f20ec0d675f079b57a4566e4e64e4060 on
its own; this one does not apply on top of the patch attached to HDDS-16958
({{OzoneManager.java}} and {{ozone-default.xml}} conflict, the test applies),
so whichever lands second needs a small rebase, and the {{ozone-default.xml}}
clause here can be dropped if HDDS-16958 lands first.
Rejected alternatives: checking in {{OMFinalizeUpgradeRequestBase.preExecute}}
covers only the leader's configuration and not an OM that finalizes from a
leader snapshot; an upgrade action for the layout feature runs during Ratis
apply, and if it threw on one OM that OM would stay at the previous layout
while its peers move on.
Covered by an extension of
{{assertSecureOmInitFailsWithInvalidTokenAndSecretKeyConfig}} in the existing
{{TestSecureOzoneCluster}}: OM initialization with the invalid configuration is
refused when the OM is not finalized (new) and when it is (existing), and an OM
that is not finalized still starts with a valid configuration. Without the
change it fails at the case that is not finalized ({{Expected
java.lang.IllegalArgumentException to be thrown, but nothing was thrown.}}).
With it {{TestSecureOzoneCluster}} (9 tests), {{TestDelegationToken}},
{{TestSecureOzoneManager}} and {{TestOMUpgradeFinalization}} (7 tests) on mini
clusters and the OM unit suites for the finalize requests, the finalize
service, the delegation token secret manager and the bucket and key managers
(83 tests) pass, and checkstyle is clean.
h3. Release note
A secure Ozone Manager that has not finalized
{{DELEGATION_TOKEN_SYMMETRIC_SIGN}} now refuses to start if
{{hdds.secret.key.rotate.duration}} +
{{ozone.manager.delegation.token.max-lifetime}} +
{{ozone.manager.delegation.remover.scan.interval}} exceeds
{{hdds.secret.key.expiry.duration}}, as a finalized one already did. This hits
secure clusters upgraded from 1.4.x or older and not yet finalized whose
{{ozone-site.xml}} sets the expiry explicitly (for example to the old 7d
default) or raises the token max lifetime or rotate duration; raise
{{hdds.secret.key.expiry.duration}} on SCM and OM (9d covers the defaults)
before upgrading.
Found by code review of the OM delegation token secret manager and request
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
b94df213f20ec0d675f079b57a4566e4e64e4060. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
