Siyao Meng created HDDS-17245:
---------------------------------
Summary: SetLifecycleConfiguration validated against one bucket
can be stored on a different bucket created under the same name before it is
applied
Key: HDDS-17245
URL: https://issues.apache.org/jira/browse/HDDS-17245
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-7-bind-lifecycle-configuration-to-validated-bucket.patch
h3. Mechanism
{{OMLifecycleConfigurationSetRequest.preExecute}} runs on the leader before the
request is replicated. It resolves the bucket and checks that the layout tag of
the configuration matches the layout of the bucket. {{validateAndUpdateCache}}
runs when the request is applied. It checks only that a bucket with that name
exists and that the rules are valid for the layout tag in the request, and then
binds the configuration to that bucket's object ID. If the bucket is deleted
and another one is created with the same name between the two steps, the
configuration is stored on a bucket it was never validated against, and
{{KeyLifecycleService}} accepts it because the stored object ID matches.
With a layout change this stores rules that the new bucket would reject. A
configuration tagged OBJECT_STORE with the prefix {{dir1}} (no trailing slash)
is stored on a FILE_SYSTEM_OPTIMIZED bucket, while the same request sent
directly to that bucket is rejected, either for the layout mismatch or because
an FSO prefix must end with {{/}}.
h3. Trigger
# A client sends SetLifecycleConfiguration for bucket {{b}}, and preExecute
passes on the leader.
# Before the request is applied, {{b}} (empty) is deleted and created again
under the same name.
# The request is applied and returns OK.
The window is the time between preExecute on the leader and the apply of the
request. I reached the state through the request methods directly (see
Reproduction) and did not race it over RPC.
h3. Impact
* The request succeeds and the configuration is attached to a bucket it was not
validated for. It stays until it is replaced or deleted.
* For the OBJECT_STORE prefix stored on an FSO bucket above,
{{KeyLifecycleService.getRuleUnion}} skips an FSO rule whose prefix does not
end with {{/}} and logs a warning, so the rule silently does nothing (run with
a scan of the bucket). When the configuration also has a rule without a prefix,
every rule is evaluated per key and the rule acts with OBJECT_STORE prefix
matching, which is what the client asked for (argued, not run).
h3. Reproduction
No standalone reproduction is attached. The regression test in the patch shows
the behaviour: {{testValidateAndUpdateCacheRejectsRecreatedBucket}} runs
{{preExecute}} against a bucket, replaces the bucket in the bucket table with
another one of the same name, and applies the request. On the unfixed code the
apply returns OK and binds the configuration to the new bucket.
For the layout case I ran the real {{preExecute}} and
{{validateAndUpdateCache}} against a real {{OmMetadataManagerImpl}}, with
bucket link resolution stubbed to report the layout of the bucket currently in
the table and the delete and create of the bucket written to the bucket table
between the two steps. Output on the unfixed code:
{noformat}
preExecute passed against OBJECT_STORE bucket objectID=1000
apply on recreated FSO bucket: status=OK
stored configuration: layout=OBJECT_STORE bucketObjectID=2000 prefix=dir1 |
bucket: layout=FILE_SYSTEM_OPTIMIZED objectID=2000
control, OBJECT_STORE tagged request on FSO bucket: INVALID_REQUEST Bucket
layout mismatch: requested lifecycle configuration has bucket layout
OBJECT_STORE but the actual bucket has layout FILE_SYSTEM_OPTIMIZED
control, FSO tagged request on FSO bucket: INVALID_REQUEST
FILE_SYSTEM_OPTIMIZED bucket prefix must end with '/'.
{noformat}
h3. Patch
[^MC-7-bind-lifecycle-configuration-to-validated-bucket.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
64d4a791502.
{{preExecute}} records the object ID of the bucket it validated in the existing
{{bucketObjectID}} field of the request's {{LifecycleConfiguration}}, and
{{validateAndUpdateCache}} rejects the request with BUCKET_NOT_FOUND if the
bucket now has a different object ID. This does not depend on what changed with
the bucket, so I preferred it to re-checking the layout at apply. A request
without the field is applied as before. SetLifecycleConfiguration is behind the
{{S3_LIFECYCLE_SUPPORT}} layout feature, which is not in a release yet, so no
released OM applies these requests differently (only OMs built from master
before this change would accept such a request in the window).
Covered by {{testValidateAndUpdateCacheRejectsRecreatedBucket}} in the existing
{{TestOMLifecycleConfigurationSetRequest}}, which also checks that a request
validated against the new bucket is applied to it. Without the change it fails
with {{expected: <BUCKET_NOT_FOUND> but was: <OK>}}. With it the lifecycle
request and helper suites (32 tests) and {{TestKeyLifecycleService}} (722
tests, 18 skipped) pass and checkstyle is clean. With the patch the layout case
above is rejected at the apply with BUCKET_NOT_FOUND.
Found by TLA+ model checking and code review of the OM volume, bucket and
snapshot lifecycle paths under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
