Siyao Meng created HDDS-17408:
---------------------------------
Summary: STS revocation cutoff is overwritten on apply, so a
revoke applied with a lower stamp lowers the stored cutoff
Key: HDDS-17408
URL: https://issues.apache.org/jira/browse/HDDS-17408
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-keep-highest-sts-revocation-cutoff.patch
h3. Mechanism
{{S3RevokeSTSTokenRequest.preExecute}} stamps {{revocationTimeMillis}} with the
current time of the OM that runs it, before the request is submitted to Ratis.
On apply, {{S3RevokeSTSTokenRequest.validateAndUpdateCache}} puts that value
into the {{s3RevokedStsTokenTable}} cache and
{{S3RevokeSTSTokenResponse.addToDBBatch}} writes it to the DB, both without
reading the cutoff already stored for the {{originalAccessKeyId}}.
Nothing orders the stamp with the log position of the request:
{{OMExecutionFlow.submitExecutionToRatis}} runs {{preExecute}} and the Ratis
submit without a lock, and the stamps come from whichever OM runs
{{preExecute}}. So the stamps of two revokes for the same key do not
necessarily increase in log order, and when a revoke whose stamp is lower than
the stored cutoff is applied, every OM replaces the stored cutoff with the
lower value.
h3. Trigger
A revoke for an {{originalAccessKeyId}} is applied while the cutoff stored for
that key is higher than the revoke's stamp. This is the case when the revoke
was stamped before an earlier applied revoke for the same key was stamped
(stamp and apply are not ordered), or when it was stamped by an OM whose clock
is behind the clock that stamped the stored cutoff.
h3. Impact
* After the apply, the stored cutoff for the key is the lower stamp on every
OM, in the cache and in the DB, and it stays there when the table is loaded
from the DB. Only a later revoke with a higher stamp raises it;
{{RevokedSTSTokenCleanupService}} only deletes entries.
* {{S3SecurityUtil.isRevokedStsToken}} compares a token's signed
{{creationTime}} with this stored value only. For tokens created between the
lower stamp and the higher one, its result follows the lower cutoff, although
the revoke that set the higher cutoff had been applied, until they expire.
* No data is lost or changed. STS is not part of a released Ozone version.
h3. Reproduction
The regression test in the attached patch covers it and fails on unmodified
source:
{{TestS3RevokeSTSTokenRequest.testRevokeWithLowerStampKeepsStoredCutoff}}
({{hadoop-ozone/ozone-manager}}) applies revokes stamped 11000, 10000 and 12000
for one key through the real request and response on a real
{{OmMetadataManagerImpl}}, commits each response, and checks the cache and the
DB. Without the change it fails with {{expected: 11000L but was: 10000L}} after
the second revoke.
No standalone reproduction is attached. Not run: a multi OM cluster; the test
sets the apply order directly.
h3. Patch
[^MC-2-keep-highest-sts-revocation-cutoff.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
65320362775.
{{S3RevokeSTSTokenRequest.validateAndUpdateCache}} reads the stored cutoff for
the key ({{getIfExist}}) and keeps the larger of the stored value and the
stamp, and uses that value for both the cache entry and the response, so the
cache and the DB agree and a revoke never lowers the stored cutoff. The table
is fully cached, so the read sees every value applied before. All OMs keep the
same value as long as their caches agree. At this commit a purge removes the
row only from the DB, so an OM that kept the table in memory still holds a
purged cutoff; for a stamp more than 12 hours below a purged cutoff it would
keep a different value than an OM that loaded the table from the DB. The purge
change in HDDS-17406, which also removes the cached entry, closes this. There
is no proto change: the request still carries the stamp from {{preExecute}},
and {{S3RevokeSTSTokenResponse}} is unchanged.
One behaviour changes: a cutoff stamped ahead of real time can no longer be
lowered by a later revoke, so for that key STS tokens created before it are
rejected until real time passes it; this fails closed and is bounded by how far
the clock was ahead.
This patch should land with or after HDDS-17406. Both change the test setup of
{{TestS3RevokeSTSTokenRequest}}, so whichever lands second needs a rebase.
With the patch, {{TestS3RevokeSTSTokenRequest}}, {{TestS3SecurityUtil}},
{{TestRevokedSTSTokenCleanupService}}, {{TestOzoneManagerRatisRequest}},
{{TestS3AssumeRoleRequest}} and {{TestOmMetadataManager}} pass, and checkstyle
on {{hadoop-ozone/ozone-manager}} is clean.
Found by TLA+ model checking and code review of the OM STS token revocation and
cleanup paths under HDDS-15926, on commit
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16110 (introduced the per key revocation cutoff), HDDS-14067
(the revoke request and {{revokeststoken}}), HDDS-14094 (the cleanup service).
The attached patch is a proposal for review. Generated with Specula (Claude
Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
