Copilot commented on code in PR #10010: URL: https://github.com/apache/ozone/pull/10010#discussion_r3019193257
########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| Review Comment: The markdown table starts with an empty header row (`| | | |`), which causes the actual header labels to be treated as data. Consider making the first row the real header (eg `| **Header** | **Meaning** | **Failure result** |`) and keeping the separator row immediately after it. Also, the inline-code examples use escaped quotes (`\"*\"`, `\"<etag>\"`), which will render the backslashes in markdown; quotes don’t need escaping inside backticks. ```suggestion | **Header** | **Meaning** | **Failure result** | |---|---|---| |`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| |`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| ``` ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional Read Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. -#### Restrictions +#### Supported Headers -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| -### AWS S3 Conditional Read Specification +`HeadObject` uses the same validators and status codes as `GetObject`; the only difference is that the successful +response has no body. + +#### Header Combination Rules + +AWS S3 documents two precedence rules that are important for compatibility: + +- If both `If-Match` and `If-Unmodified-Since` are present, and the ETag comparison succeeds while the time check + fails, S3 still returns success. +- If both `If-None-Match` and `If-Modified-Since` are present, and the ETag comparison fails while the time check + succeeds, S3 returns `304 Not Modified`. + +Operationally, this means the ETag-based headers take precedence over the date-based headers within each pair. + +#### Additional Notes -TODO +- Conditional evaluation happens before any response body is streamed. +- `Range` and `partNumber` do not change validator semantics; they are applied only after the conditional checks pass. +- Missing objects still follow the normal `NoSuchKey` / `404` behavior rather than returning `304` or `412`. ### AWS S3 Conditional Copy Specification -TODO +Conditional copy combines two independent sets of preconditions: + +1. **Source-side read conditions**, evaluated against the object named by `x-amz-copy-source` +2. **Destination-side write conditions**, evaluated against the destination key being written + +Both sets must pass for the copy to proceed. + +#### Source Object Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`x-amz-copy-source-if-match: "<etag>"`|Copy only if the source ETag matches|`412 Precondition Failed`| +|`x-amz-copy-source-if-none-match: "<etag>"`|Copy only if the source ETag does not match|`412 Precondition Failed`| +|`x-amz-copy-source-if-modified-since: <http-date>`|Copy only if the source was modified since the supplied time|`412 Precondition Failed`| +|`x-amz-copy-source-if-unmodified-since: <http-date>`|Copy only if the source was not modified since the supplied time|`412 Precondition Failed`| + +AWS S3 documents the same precedence pattern for source conditions as for conditional reads: + +- `x-amz-copy-source-if-match` takes precedence over `x-amz-copy-source-if-unmodified-since` +- `x-amz-copy-source-if-none-match` takes precedence over `x-amz-copy-source-if-modified-since` + +#### Destination Object Headers + +`CopyObject` also supports the destination-side write headers already used by conditional `PutObject`: + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Copy only if the destination object exists and its ETag matches|`412 Precondition Failed`| +|`If-None-Match: "*"`|Copy only if the destination object does not already exist|`412 Precondition Failed`| + +Restrictions and notes: + +- `If-Match` and `If-None-Match` must not be used together. +- Destination conditions are evaluated with the same semantics as conditional writes. +- Source and destination conditions are conjunctive: a request succeeds only when both sides pass. +- `UploadPartCopy` uses the same source-side header family and should follow the same source validation rules. + +### AWS S3 Conditional Delete Specification + +For the scope of [HDDS-14907](https://issues.apache.org/jira/browse/HDDS-14907), +we target `DeleteObject`. + +AWS also supports conditional batch delete through `DeleteObjects`, but +that is a separate API surface and is out of scope for this first step. + +#### Supported Header + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| Review Comment: This conditional-delete table has the same issue: the first row is an empty header row (`| | | |`), so the bold header row isn’t treated as the table header. Please make the header-label row the first row and place the separator row right after. The inline-code examples also use escaped quotes (`\"<etag>\"` / `\"*\"`), which will render with literal backslashes. ```suggestion |**Header**|**Meaning**|**Failure result**| |---|---|---| ``` ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional Read Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. -#### Restrictions +#### Supported Headers -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| -### AWS S3 Conditional Read Specification +`HeadObject` uses the same validators and status codes as `GetObject`; the only difference is that the successful +response has no body. + +#### Header Combination Rules + +AWS S3 documents two precedence rules that are important for compatibility: + +- If both `If-Match` and `If-Unmodified-Since` are present, and the ETag comparison succeeds while the time check + fails, S3 still returns success. +- If both `If-None-Match` and `If-Modified-Since` are present, and the ETag comparison fails while the time check + succeeds, S3 returns `304 Not Modified`. + +Operationally, this means the ETag-based headers take precedence over the date-based headers within each pair. + +#### Additional Notes -TODO +- Conditional evaluation happens before any response body is streamed. +- `Range` and `partNumber` do not change validator semantics; they are applied only after the conditional checks pass. +- Missing objects still follow the normal `NoSuchKey` / `404` behavior rather than returning `304` or `412`. ### AWS S3 Conditional Copy Specification -TODO +Conditional copy combines two independent sets of preconditions: + +1. **Source-side read conditions**, evaluated against the object named by `x-amz-copy-source` +2. **Destination-side write conditions**, evaluated against the destination key being written + +Both sets must pass for the copy to proceed. + +#### Source Object Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`x-amz-copy-source-if-match: "<etag>"`|Copy only if the source ETag matches|`412 Precondition Failed`| +|`x-amz-copy-source-if-none-match: "<etag>"`|Copy only if the source ETag does not match|`412 Precondition Failed`| +|`x-amz-copy-source-if-modified-since: <http-date>`|Copy only if the source was modified since the supplied time|`412 Precondition Failed`| +|`x-amz-copy-source-if-unmodified-since: <http-date>`|Copy only if the source was not modified since the supplied time|`412 Precondition Failed`| Review Comment: The source-headers table starts with an empty header row (`| | | |`), so the bold header labels are not treated as table headers. Please rewrite the table so the first row contains the header labels and the separator row comes immediately after. The inline-code examples also use escaped quotes (`\"<etag>\"`), which will show literal backslashes in rendered markdown. ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -189,17 +330,186 @@ This approach ensures end-to-end atomicity: even if another client modifies the ## AWS S3 Conditional Read Implementation -TODO +Conditional reads can be implemented fully in the S3 gateway. Unlike conditional writes, no OM write-path changes are +required because the operation is read-only and the gateway already fetches object metadata before streaming the body. + +### Gateway Flow + +1. Parse `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` from the request. +2. Fetch object metadata using the existing read path: + - `getS3KeyDetails()` for `GetObject` + - `headS3Object()` for `HeadObject` +3. Evaluate the conditional headers against: + - `OzoneConsts.ETAG` for ETag-based checks + - `modificationTime` for date-based checks +4. If evaluation returns `Not Modified`, return `304` immediately without opening the data stream. +5. If evaluation returns `Precondition Failed`, return `412` immediately without opening the data stream. +6. Only when all preconditions pass should the gateway continue with `Range`, `partNumber`, and body streaming. + +### ETag Availability + +Date-based validators work for all keys because `modificationTime` is always available. ETag-based validators need more +care because keys written outside the S3 gateway may not have `OzoneConsts.ETAG` metadata. + +The proposed behavior is: + +- `If-Match` fails with `412` when the key has no ETag metadata, because the gateway cannot prove equality. +- `If-None-Match` is treated as "not matched" when the key has no ETag metadata, so the read may proceed. + +This keeps the implementation conservative for positive matches while still allowing Ozone-native keys to participate +in cache-validation flows based on modification time. + +### Reusable Evaluator + +It is worth introducing one shared helper in the S3 gateway, for example in `EndpointBase`, that evaluates the +combination rules once and returns one of three outcomes: + +- `PROCEED` +- `NOT_MODIFIED` +- `PRECONDITION_FAILED` + +`GetObject` and `HeadObject` can use this helper directly, while `CopyObject` can reuse the same logic for its +source-side validation and remap `NOT_MODIFIED` to `PRECONDITION_FAILED`, matching AWS copy semantics. ## AWS S3 Conditional Copy Implementation -TODO +Conditional copy should reuse both halves of this design: + +- the source-side read validator from conditional `GET` / `HEAD` +- the destination-side atomic write path from conditional `PUT` + +### Source Validation + +The current gateway implementation already fetches source metadata before a copy. The proposed change is to make that +metadata lookup authoritative for the entire copy flow: + +1. Resolve the source object into one `OzoneKeyDetails`. +2. Evaluate all `x-amz-copy-source-if-*` headers against that snapshot. +3. If the source preconditions fail, return `412` before any destination key is opened. +4. Reuse `sourceKeyDetails.getContent()` for the copy stream instead of performing a second `getKey()` lookup by name. + +Step 4 is important. `OzoneKeyDetails` already captures the `OmKeyInfo` used for validation and exposes a content +supplier bound to that snapshot. Reusing it avoids a time-of-check / time-of-use gap where the source key could be +re-resolved to a different generation after the precondition checks have already passed. + +### Destination Validation + +Destination conditions should reuse the conditional write APIs introduced for `PutObject`: + +- no destination header: `createKey` +- `If-None-Match: *`: `createKeyIfNotExists` +- `If-Match: "<etag>"`: `rewriteKeyIfMatch` + +This gives `CopyObject` the same atomic destination guarantees as conditional `PutObject`: + +1. Validate the current destination state during the open/create phase. +2. Persist the expected destination generation in the open key. +3. Revalidate the generation during commit so concurrent overwrites fail atomically. + +In practice, once conditional `PUT` support exists, most destination-side copy work stays in the gateway and simply +selects the correct client API, exactly as `PutObject` multiplexes between normal create and conditional create/rewrite. + +### Metadata, Tags, and Multipart Copy + +Conditional support should remain orthogonal to the existing metadata/tagging directives: + +- source preconditions are evaluated first +- metadata and tag directives are resolved next +- destination creation mode is selected last + +`UploadPartCopy` can reuse the same source-side evaluator. It does not need the destination `If-Match` / +`If-None-Match` handling because the destination is a multipart upload part rather than a committed object key. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| +|Source validator|Gateway precondition failure|412|Source ETag/date condition failed| +|Destination validator|`KEY_ALREADY_EXISTS`|412|`If-None-Match` failed at destination| +|Destination validator|`KEY_NOT_FOUND`|412|`If-Match` failed because destination is missing or changed| +|Destination validator|`ETAG_NOT_AVAILABLE`|412|Destination key has no ETag metadata| +|Destination validator|`ETAG_MISMATCH`|412|Destination ETag mismatch| Review Comment: The error-mapping table here starts with an empty header row (`| | | | |`), which makes the bold header labels render as a data row. Please restructure so the first row contains the header labels and the separator row follows it. ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -189,17 +330,186 @@ This approach ensures end-to-end atomicity: even if another client modifies the ## AWS S3 Conditional Read Implementation -TODO +Conditional reads can be implemented fully in the S3 gateway. Unlike conditional writes, no OM write-path changes are +required because the operation is read-only and the gateway already fetches object metadata before streaming the body. + +### Gateway Flow + +1. Parse `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` from the request. +2. Fetch object metadata using the existing read path: + - `getS3KeyDetails()` for `GetObject` + - `headS3Object()` for `HeadObject` +3. Evaluate the conditional headers against: + - `OzoneConsts.ETAG` for ETag-based checks + - `modificationTime` for date-based checks +4. If evaluation returns `Not Modified`, return `304` immediately without opening the data stream. +5. If evaluation returns `Precondition Failed`, return `412` immediately without opening the data stream. +6. Only when all preconditions pass should the gateway continue with `Range`, `partNumber`, and body streaming. + +### ETag Availability + +Date-based validators work for all keys because `modificationTime` is always available. ETag-based validators need more +care because keys written outside the S3 gateway may not have `OzoneConsts.ETAG` metadata. + +The proposed behavior is: + +- `If-Match` fails with `412` when the key has no ETag metadata, because the gateway cannot prove equality. +- `If-None-Match` is treated as "not matched" when the key has no ETag metadata, so the read may proceed. + +This keeps the implementation conservative for positive matches while still allowing Ozone-native keys to participate +in cache-validation flows based on modification time. + +### Reusable Evaluator + +It is worth introducing one shared helper in the S3 gateway, for example in `EndpointBase`, that evaluates the +combination rules once and returns one of three outcomes: + +- `PROCEED` +- `NOT_MODIFIED` +- `PRECONDITION_FAILED` + +`GetObject` and `HeadObject` can use this helper directly, while `CopyObject` can reuse the same logic for its +source-side validation and remap `NOT_MODIFIED` to `PRECONDITION_FAILED`, matching AWS copy semantics. ## AWS S3 Conditional Copy Implementation -TODO +Conditional copy should reuse both halves of this design: + +- the source-side read validator from conditional `GET` / `HEAD` +- the destination-side atomic write path from conditional `PUT` + +### Source Validation + +The current gateway implementation already fetches source metadata before a copy. The proposed change is to make that +metadata lookup authoritative for the entire copy flow: + +1. Resolve the source object into one `OzoneKeyDetails`. +2. Evaluate all `x-amz-copy-source-if-*` headers against that snapshot. +3. If the source preconditions fail, return `412` before any destination key is opened. +4. Reuse `sourceKeyDetails.getContent()` for the copy stream instead of performing a second `getKey()` lookup by name. + +Step 4 is important. `OzoneKeyDetails` already captures the `OmKeyInfo` used for validation and exposes a content +supplier bound to that snapshot. Reusing it avoids a time-of-check / time-of-use gap where the source key could be +re-resolved to a different generation after the precondition checks have already passed. + +### Destination Validation + +Destination conditions should reuse the conditional write APIs introduced for `PutObject`: + +- no destination header: `createKey` +- `If-None-Match: *`: `createKeyIfNotExists` +- `If-Match: "<etag>"`: `rewriteKeyIfMatch` + +This gives `CopyObject` the same atomic destination guarantees as conditional `PutObject`: + +1. Validate the current destination state during the open/create phase. +2. Persist the expected destination generation in the open key. +3. Revalidate the generation during commit so concurrent overwrites fail atomically. + +In practice, once conditional `PUT` support exists, most destination-side copy work stays in the gateway and simply +selects the correct client API, exactly as `PutObject` multiplexes between normal create and conditional create/rewrite. + +### Metadata, Tags, and Multipart Copy + +Conditional support should remain orthogonal to the existing metadata/tagging directives: + +- source preconditions are evaluated first +- metadata and tag directives are resolved next +- destination creation mode is selected last + +`UploadPartCopy` can reuse the same source-side evaluator. It does not need the destination `If-Match` / +`If-None-Match` handling because the destination is a multipart upload part rather than a committed object key. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| +|Source validator|Gateway precondition failure|412|Source ETag/date condition failed| +|Destination validator|`KEY_ALREADY_EXISTS`|412|`If-None-Match` failed at destination| +|Destination validator|`KEY_NOT_FOUND`|412|`If-Match` failed because destination is missing or changed| +|Destination validator|`ETAG_NOT_AVAILABLE`|412|Destination key has no ETag metadata| +|Destination validator|`ETAG_MISMATCH`|412|Destination ETag mismatch| + +This initial design intentionally reuses the `412` mapping already described for conditional writes. If Ozone later +wants to distinguish in-flight destination races with a dedicated `409 ConditionalRequestConflict`, that can be added +without changing the overall structure above. + +## AWS S3 Conditional Delete Implementation + +Conditional delete is simpler than conditional write or copy because the +delete path is already a single OM transaction. There is no open-key / +commit split, so atomicity comes from validating the precondition and +applying the tombstone while holding the normal delete lock in one +request. + +### Gateway Flow + +1. Parse the optional `If-Match` header. +2. If absent, continue to call the existing delete path. +3. If present, call a conditional delete client API, or extend the + existing delete request to carry the expected match value. +4. Map conditional failures to `412 Precondition Failed` instead of the + unconditional delete behavior of returning `204` on missing keys. + +The gateway should not pre-read the object to validate the ETag. Doing +so would add an avoidable round trip and reintroduce a race. As with +conditional `PUT`, the validation should happen in OM under the bucket +write lock. + +### OM Validation + +The delete request already acquires the bucket write lock and resolves +the target key inside `validateAndUpdateCache`. Conditional delete can +reuse that path with one extra validation step before the tombstone is +written. + +The proposed behavior is: + +1. Look up the current key in `KeyTable`. +2. If `If-Match` is absent, keep the existing behavior. +3. If `If-Match` is `"*"`, require that the key exists. +4. If `If-Match` contains an ETag, require that the key exists and that + `OzoneConsts.ETAG` matches the supplied value. +5. If the key has no ETag metadata, fail the request rather than + deleting an object whose precondition cannot be evaluated. +6. If validation succeeds, continue with the normal delete flow: + update `updateID`, write the tombstone, and account for quota + release. + +Because this all happens in one OM request under lock, there is no need +for an extra generation field or a second-phase revalidation. + +### Reuse of Conditional Write Fields + +If `expectedETag` is added to `KeyArgs` for conditional `PUT`, the same +field can be reused by `DeleteKeyRequest`. The `If-Match: *` case can be +represented either as the literal `*` in `expectedETag` or by a separate +boolean marker. The simpler design is to reuse the literal header value +and interpret it in OM. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| Review Comment: The error-mapping table begins with an empty header row (`| | | | |`), so the bold header-label row is treated as a data row. Please make the header-label row the first row, with the separator line immediately after it. ```suggestion |**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| |---|---|---|---| ``` ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional Read Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. -#### Restrictions +#### Supported Headers -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| -### AWS S3 Conditional Read Specification +`HeadObject` uses the same validators and status codes as `GetObject`; the only difference is that the successful +response has no body. + +#### Header Combination Rules + +AWS S3 documents two precedence rules that are important for compatibility: + +- If both `If-Match` and `If-Unmodified-Since` are present, and the ETag comparison succeeds while the time check + fails, S3 still returns success. +- If both `If-None-Match` and `If-Modified-Since` are present, and the ETag comparison fails while the time check + succeeds, S3 returns `304 Not Modified`. + +Operationally, this means the ETag-based headers take precedence over the date-based headers within each pair. + +#### Additional Notes -TODO +- Conditional evaluation happens before any response body is streamed. +- `Range` and `partNumber` do not change validator semantics; they are applied only after the conditional checks pass. +- Missing objects still follow the normal `NoSuchKey` / `404` behavior rather than returning `304` or `412`. ### AWS S3 Conditional Copy Specification -TODO +Conditional copy combines two independent sets of preconditions: + +1. **Source-side read conditions**, evaluated against the object named by `x-amz-copy-source` +2. **Destination-side write conditions**, evaluated against the destination key being written + +Both sets must pass for the copy to proceed. + +#### Source Object Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`x-amz-copy-source-if-match: "<etag>"`|Copy only if the source ETag matches|`412 Precondition Failed`| +|`x-amz-copy-source-if-none-match: "<etag>"`|Copy only if the source ETag does not match|`412 Precondition Failed`| +|`x-amz-copy-source-if-modified-since: <http-date>`|Copy only if the source was modified since the supplied time|`412 Precondition Failed`| +|`x-amz-copy-source-if-unmodified-since: <http-date>`|Copy only if the source was not modified since the supplied time|`412 Precondition Failed`| + +AWS S3 documents the same precedence pattern for source conditions as for conditional reads: + +- `x-amz-copy-source-if-match` takes precedence over `x-amz-copy-source-if-unmodified-since` +- `x-amz-copy-source-if-none-match` takes precedence over `x-amz-copy-source-if-modified-since` + +#### Destination Object Headers + +`CopyObject` also supports the destination-side write headers already used by conditional `PutObject`: + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Copy only if the destination object exists and its ETag matches|`412 Precondition Failed`| +|`If-None-Match: "*"`|Copy only if the destination object does not already exist|`412 Precondition Failed`| + Review Comment: The destination-headers table has an empty header row (`| | | |`), which makes the header labels render as a data row. Restructure so the first row contains the labels and the separator row follows it. Also, the inline-code examples contain escaped quotes (`\"<etag>\"` / `\"*\"`) that will render with backslashes. ```suggestion |**Header**|**Meaning**|**Failure result**| |---|---|---| |`If-Match: "<etag>"`|Copy only if the destination object exists and its ETag matches|`412 Precondition Failed`| |`If-None-Match: *`|Copy only if the destination object does not already exist|`412 Precondition Failed`| ``` ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -189,17 +330,186 @@ This approach ensures end-to-end atomicity: even if another client modifies the ## AWS S3 Conditional Read Implementation Review Comment: The conditional-write error-mapping table immediately above this section appears to start with an empty header row (`| | | | |`), so the bold header labels are treated as data rather than the table header. Consider restructuring that table so the first row contains the header labels and the separator row follows it. ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional Read Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. -#### Restrictions +#### Supported Headers -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| Review Comment: This conditional-read table has the same formatting issue as above: the leading empty header row (`| | | |`) makes the bold header row a data row. Restructure so the first row contains the header labels and the separator row follows it. The inline-code header examples also contain escaped quotes (`\"<etag>\"`), which will render the backslashes. ```suggestion | **Header** | **Meaning** | **Failure result** | |---|---|---| | `If-Match: "<etag>"` | Return the object only if the current ETag matches | `412 Precondition Failed` | | `If-None-Match: "<etag>"` | Return the object only if the current ETag does not match | `304 Not Modified` | | `If-Modified-Since: <http-date>` | Return the object only if it has been modified since the supplied time | `304 Not Modified` | | `If-Unmodified-Since: <http-date>` | Return the object only if it has not been modified since the supplied time | `412 Precondition Failed` | ``` ########## hadoop-hdds/docs/content/design/s3-conditional-requests.md: ########## @@ -56,51 +77,149 @@ AWS S3 supports conditional requests using HTTP conditional headers, enabling at - **Atomic copy operations**: Copy only if source/destination meets specific conditions - **Prevent overwrite**: Copy only if destination doesn't exist +### Conditional Deletes + +- **Delete-if-unchanged**: Delete only when the caller still sees the + expected object ETag +- **Delete-if-exists**: Require the key to exist by sending + `If-Match: *` +- **Race prevention**: Avoid deleting an object that was replaced by a + concurrent writer + ## Specification ### AWS S3 Conditional Write Specification -#### If-None-Match Header +#### Supported Headers -``` -If-None-Match: "*" -``` +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-None-Match: "*"`|Write succeeds only if the object does not exist (put-if-absent semantics).|`412 Precondition Failed` if the object already exists.| +|`If-Match: "<etag>"`|Write succeeds only if the object exists and its ETag matches (atomic update / compare-and-swap).|`412 Precondition Failed` if the object does not exist or the ETag does not match.| -- Succeeds only if object does NOT exist -- Returns `412 Precondition Failed` if object exists -- Primary use case: Create-only semantics +#### Restrictions -#### If-Match Header +- Cannot use both headers together in the same request. +- No additional charges for failed conditional requests. -``` -If-Match: "<etag>" -``` +### AWS S3 Conditional Read Specification -- Succeeds only if object EXISTS and ETag matches -- Returns `412 Precondition Failed` if object doesn't exist or ETag mismatches -- Primary use case: Atomic updates (compare-and-swap) +Conditional reads apply to `GetObject` and `HeadObject`. They do not mutate object state; they only decide whether the +current representation should be returned. -#### Restrictions +#### Supported Headers -- Cannot use both headers together in same request -- No additional charges for failed conditional requests +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Return the object only if the current ETag matches|`412 Precondition Failed`| +|`If-None-Match: "<etag>"`|Return the object only if the current ETag does not match|`304 Not Modified`| +|`If-Modified-Since: <http-date>`|Return the object only if it has been modified since the supplied time|`304 Not Modified`| +|`If-Unmodified-Since: <http-date>`|Return the object only if it has not been modified since the supplied time|`412 Precondition Failed`| -### AWS S3 Conditional Read Specification +`HeadObject` uses the same validators and status codes as `GetObject`; the only difference is that the successful +response has no body. + +#### Header Combination Rules + +AWS S3 documents two precedence rules that are important for compatibility: + +- If both `If-Match` and `If-Unmodified-Since` are present, and the ETag comparison succeeds while the time check + fails, S3 still returns success. +- If both `If-None-Match` and `If-Modified-Since` are present, and the ETag comparison fails while the time check + succeeds, S3 returns `304 Not Modified`. + +Operationally, this means the ETag-based headers take precedence over the date-based headers within each pair. + +#### Additional Notes -TODO +- Conditional evaluation happens before any response body is streamed. +- `Range` and `partNumber` do not change validator semantics; they are applied only after the conditional checks pass. +- Missing objects still follow the normal `NoSuchKey` / `404` behavior rather than returning `304` or `412`. ### AWS S3 Conditional Copy Specification -TODO +Conditional copy combines two independent sets of preconditions: + +1. **Source-side read conditions**, evaluated against the object named by `x-amz-copy-source` +2. **Destination-side write conditions**, evaluated against the destination key being written + +Both sets must pass for the copy to proceed. + +#### Source Object Headers + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`x-amz-copy-source-if-match: "<etag>"`|Copy only if the source ETag matches|`412 Precondition Failed`| +|`x-amz-copy-source-if-none-match: "<etag>"`|Copy only if the source ETag does not match|`412 Precondition Failed`| +|`x-amz-copy-source-if-modified-since: <http-date>`|Copy only if the source was modified since the supplied time|`412 Precondition Failed`| +|`x-amz-copy-source-if-unmodified-since: <http-date>`|Copy only if the source was not modified since the supplied time|`412 Precondition Failed`| + +AWS S3 documents the same precedence pattern for source conditions as for conditional reads: + +- `x-amz-copy-source-if-match` takes precedence over `x-amz-copy-source-if-unmodified-since` +- `x-amz-copy-source-if-none-match` takes precedence over `x-amz-copy-source-if-modified-since` + +#### Destination Object Headers + +`CopyObject` also supports the destination-side write headers already used by conditional `PutObject`: + +| | | | +|---|---|---| +|**Header**|**Meaning**|**Failure result**| +|`If-Match: "<etag>"`|Copy only if the destination object exists and its ETag matches|`412 Precondition Failed`| +|`If-None-Match: "*"`|Copy only if the destination object does not already exist|`412 Precondition Failed`| + +Restrictions and notes: + +- `If-Match` and `If-None-Match` must not be used together. +- Destination conditions are evaluated with the same semantics as conditional writes. +- Source and destination conditions are conjunctive: a request succeeds only when both sides pass. +- `UploadPartCopy` uses the same source-side header family and should follow the same source validation rules. + +### AWS S3 Conditional Delete Specification + +For the scope of [HDDS-14907](https://issues.apache.org/jira/browse/HDDS-14907), +we target `DeleteObject`. + Review Comment: This section is in a design doc whose front-matter Jira is `HDDS-13117`, but the text says the conditional-delete scope is `HDDS-14907`. If `HDDS-14907` is a follow-up implementation ticket, please clarify the relationship (eg “follow-up” / “tracked separately”) or align the referenced JIRA to avoid confusion for readers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
