[
https://issues.apache.org/jira/browse/HDDS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17408:
------------------------------
Affects Version/s: 2.3.0
> STS revocation cutoff is overwritten on apply, so a revoke applied with a
> lower stamp lowers the stored cutoff
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17408
> URL: https://issues.apache.org/jira/browse/HDDS-17408
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-2-keep-highest-sts-revocation-cutoff.patch
>
>
> h3. Mechanism
> {{S3RevokeSTSTokenRequest.preExecute}} stamps {{revocationTimeMillis}} with
> the current time of the OM that runs it, before the request is submitted to
> Ratis. On apply, {{S3RevokeSTSTokenRequest.validateAndUpdateCache}} puts that
> value into the {{s3RevokedStsTokenTable}} cache and
> {{S3RevokeSTSTokenResponse.addToDBBatch}} writes it to the DB, both without
> reading the cutoff already stored for the {{originalAccessKeyId}}.
> Nothing orders the stamp with the log position of the request:
> {{OMExecutionFlow.submitExecutionToRatis}} runs {{preExecute}} and the Ratis
> submit without a lock, and the stamps come from whichever OM runs
> {{preExecute}}. So the stamps of two revokes for the same key do not
> necessarily increase in log order, and when a revoke whose stamp is lower
> than the stored cutoff is applied, every OM replaces the stored cutoff with
> the lower value.
> h3. Trigger
> A revoke for an {{originalAccessKeyId}} is applied while the cutoff stored
> for that key is higher than the revoke's stamp. This is the case when the
> revoke was stamped before an earlier applied revoke for the same key was
> stamped (stamp and apply are not ordered), or when it was stamped by an OM
> whose clock is behind the clock that stamped the stored cutoff.
> h3. Impact
> * After the apply, the stored cutoff for the key is the lower stamp on every
> OM, in the cache and in the DB, and it stays there when the table is loaded
> from the DB. Only a later revoke with a higher stamp raises it;
> {{RevokedSTSTokenCleanupService}} only deletes entries.
> * {{S3SecurityUtil.isRevokedStsToken}} compares a token's signed
> {{creationTime}} with this stored value only. For tokens created between the
> lower stamp and the higher one, its result follows the lower cutoff, although
> the revoke that set the higher cutoff had been applied, until they expire.
> * No data is lost or changed. STS is not part of a released Ozone version.
> h3. Reproduction
> The regression test in the attached patch covers it and fails on unmodified
> source:
> {{TestS3RevokeSTSTokenRequest.testRevokeWithLowerStampKeepsStoredCutoff}}
> ({{hadoop-ozone/ozone-manager}}) applies revokes stamped 11000, 10000 and
> 12000 for one key through the real request and response on a real
> {{OmMetadataManagerImpl}}, commits each response, and checks the cache and
> the DB. Without the change it fails with {{expected: 11000L but was: 10000L}}
> after the second revoke.
> No standalone reproduction is attached. Not run: a multi OM cluster; the test
> sets the apply order directly.
> h3. Patch
> [^MC-2-keep-highest-sts-revocation-cutoff.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
> 65320362775.
> {{S3RevokeSTSTokenRequest.validateAndUpdateCache}} reads the stored cutoff
> for the key ({{getIfExist}}) and keeps the larger of the stored value and the
> stamp, and uses that value for both the cache entry and the response, so the
> cache and the DB agree and a revoke never lowers the stored cutoff. The table
> is fully cached, so the read sees every value applied before. All OMs keep
> the same value as long as their caches agree. At this commit a purge removes
> the row only from the DB, so an OM that kept the table in memory still holds
> a purged cutoff; for a stamp more than 12 hours below a purged cutoff it
> would keep a different value than an OM that loaded the table from the DB.
> The purge change in HDDS-17406, which also removes the cached entry, closes
> this. There is no proto change: the request still carries the stamp from
> {{preExecute}}, and {{S3RevokeSTSTokenResponse}} is unchanged.
> One behaviour changes: a cutoff stamped ahead of real time can no longer be
> lowered by a later revoke, so for that key STS tokens created before it are
> rejected until real time passes it; this fails closed and is bounded by how
> far the clock was ahead.
> This patch should land with or after HDDS-17406. Both change the test setup
> of {{TestS3RevokeSTSTokenRequest}}, so whichever lands second needs a rebase.
> With the patch, {{TestS3RevokeSTSTokenRequest}}, {{TestS3SecurityUtil}},
> {{TestRevokedSTSTokenCleanupService}}, {{TestOzoneManagerRatisRequest}},
> {{TestS3AssumeRoleRequest}} and {{TestOmMetadataManager}} pass, and
> checkstyle on {{hadoop-ozone/ozone-manager}} is clean.
> Found by TLA+ model checking and code review of the OM STS token revocation
> and cleanup paths under HDDS-15926, on commit
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16110 (introduced the per key revocation cutoff), HDDS-14067
> (the revoke request and {{revokeststoken}}), HDDS-14094 (the cleanup
> service). The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
