Gargi-jais11 commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4345196339
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,597 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design adds S3 Object Lock to OBS buckets, so keys can be protected from deletion and overwrite. +Authorization uses Ozone native ACLs or Apache Ranger. + +## Problem statement + +Data stored for security or compliance must be protected from accidental or malicious deletion and overwrite. Object Lock provides this protection. + +The design supports both native ACLs and Ranger. Native ACLs serve deployments without Ranger, and let the core logic be built and tested without the Ranger integration. +To ship the core feature sooner, the first release excludes versioning, Legacy buckets and FSO buckets. + +**Goal:** + +* Object Lock on OBS buckets, with authorization through native ACLs or Ranger. +* Single-version keys only. + +## Non-Goal + +* Versioning: locks across multiple object versions (versioning is still under development). +* Legacy and FSO buckets. FSO needs its own decisions on: + * Lock granularity: files and directories are in different tables (FileTable / DirectoryTable), so locks could apply to files, directories or both. + * Rename and move: they change the namespace without changing the data, so they must either be blocked on locked paths or keep the lock with the same objectID. +* Value-level Ranger authorization, for example allowing a legal hold to be set `ON` but not `OFF`. + +## Differences from AWS + +* Single version: locks apply to the current key, not to object versions. +* Overwrites are rejected: in AWS S3, Put, Copy or MPU on a locked object writes a new version. In Ozone it would replace the locked key, + so Ozone rejects Put, Copy, MPU and rename (native clients only) on locked keys with `403 Access Denied` (`WORMProtectionException`). +* Governance bypass on overwrite and rename: a Governance-locked key can only be overwritten or renamed with the `bypassGovernanceRetention` flag and the bypass permission. + AWS S3 does not need this because it writes a new version (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). +* Authorization: Ozone has no root account. + * Native ACLs: changing locks and bypassing Governance need the volume or bucket owner; setting or changing a Compliance retention needs an Ozone administrator (see [Native ACLs](#1-native-acls)). + * Ranger: action-level permissions need `enableActionMatcherInPoliciesCondition`, which is `false` by default (see [Ranger](#2-ranger)). + Object Lock uses the same access types as data (`READ` / `WRITE`), so without `action-matches`, anyone who can write a key can also change its locks. + * No API can remove a Compliance retention, in either model. Only the offline `ozone repair` command can (see [Offline Compliance Retention Repair](#offline-compliance-retention-repair)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* Locks a key with no end date, until someone allowed to change the legal hold removes it + (the owner with native ACLs; `WRITE` with Ranger, limited to `PutObjectLegalHold` when `action-matches` is used). +* Used for litigation, investigations or audits, when it is not known how long the data must be kept. +* Has no mode and no date, and is independent of retention: a key can have a legal hold, a retention, or both, and stays locked until both are gone. + A legal hold can be set on a key without a retention. +* Blocks: Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients). +* Allows: Get / Head, Get / Put Object Tagging, Get Legal Hold, Put Legal Hold (with permission). + +**Retention** + +* Protects a key until a `RetainUntilDate`. The date is either fixed (set directly, or from days / years) or comes from an event hold. +* Modes: + * Compliance: no one, including administrators, can remove it, shorten it or overwrite the key through the Ozone API before it expires. Only the offline `ozone repair` command can change it. + * Governance: blocks normal users, but users with the bypass permission can bypass it. +* Event hold (AWS S3 *variable retention*): + * Used when the retention depends on an external event, such as a contract ending, an employee leaving or a loan closing. + * While the event has not happened (`EventHold=ON`), the key is locked with no end date. When it happens, the hold is released (`EventHold=OFF`) + and the duration (`EventHoldDuration`, in days or years) becomes a fixed `RetainUntilDate`. + * See [Event Hold Trigger Workflow](#event-hold-trigger-workflow) for the rules. +* Blocks (Governance allows them with `bypassGovernanceRetention`): Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients), + shortening or removing the retention (never allowed for Compliance). +* Allows: Get / Head, Get / Put Object Tagging, extending the retention. + +#### Event Hold Trigger Workflow + +```text +object created + | +event hold applied (locked with no end date) + | +business event occurs + | +hold released + | +RetainUntilDate is calculated + | +locked until RetainUntilDate +``` + +Event holds use `PutObjectRetention`, as in AWS S3, so no new API or Ranger action is needed (native clients use `OzoneBucket#setRetention`). +OM handles the request under the key write lock and uses the leader-stamped request time as now. + +The key stores `retentionDate` and the event hold (status and duration): +* The duration comes from the request, or from the bucket default when the key is created. Later changes to the bucket default do not affect existing keys. +* If the bucket default has both `rule` and `eventHold`, a new key gets `retentionDate` = creation time + `rule` and an active event hold. + With only `eventHold`, `retentionDate` is unset until the hold is released. +* While the hold is `ON`, the effective `RetainUntilDate` is the later of `retentionDate` and now + duration, so it moves forward over time. + `GetObjectRetention`, `HeadObject` and `GetObject` return this date. + +Below, "earlier" and "later" compare the requested `RetainUntilDate` with the effective date. Only `GOVERNANCE` with the bypass can make it earlier; `COMPLIANCE` never can. + +1. **Request format**: checked before the lock and the bypass; a request that breaks a rule gets `400 Bad Request`. + * The request has a mode, and `RetainUntilDate`, `EventHold` or both. + The only exception is an empty retention, which removes the retention: it needs the bypass for `GOVERNANCE` (`403` without it) and is never allowed for `COMPLIANCE`. + * `RetainUntilDate` is in the future. + * `EventHold=ON` has exactly one `EventHoldDuration` (positive, at most 36,500 days or 100 years). A `RetainUntilDate` in the same request is at least now + duration. + * `EventHold=OFF` has no `EventHoldDuration`, and `EventHoldDuration` needs `EventHold=ON`. + * The same applies to the `x-amz-object-lock-*` headers at creation: the mode header needs the retain-until-date header or the event hold headers, and they need the mode header. +2. **`EventHold=ON`** sets or updates the hold. + * Without the bypass, `RetainUntilDate` can only be later (`403` otherwise). The duration can change; + when it gets shorter, OM first saves the current effective date in `retentionDate`, so the date never moves earlier, even with the bypass. + * With the bypass (`GOVERNANCE` only), `RetainUntilDate` can be earlier, but still at least now + the new duration (rule 1). +3. **`EventHold=OFF`** releases the hold. It is the only way to release it. + * Without `RetainUntilDate`: `retentionDate` = the later of `retentionDate` and now + duration. + * With `RetainUntilDate`: that date is used. An earlier date needs the bypass (`GOVERNANCE` only), otherwise `403`. + * On a key without an active hold: no change if the key has a future `retentionDate`; `400` if neither the key nor the request has a future date. +4. **No `EventHold`** keeps the hold as it is and only changes `RetainUntilDate`. + * A later date is accepted, also while the hold is `ON`. + * An earlier date needs the bypass (`GOVERNANCE` only), otherwise `403`. While the hold is `ON`, it gets `400` even with the bypass: the request must use rule 2 or 3. + +Releasing the hold keeps the mode. A different mode in the request is a mode change (see [Retention Updates](#retention-updates)). + +> _**Note**_: AWS S3 requires versioning for Object Lock, so a Put on a locked object only adds a new version and Object Lock mainly blocks deletes. +> Ozone has no versioning yet, so it rejects every overwrite of a locked key. + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + optional bool objectLockEnabled = 24 [default = false]; + optional BucketRetention defaultRetention = 25; + } + + message BucketRetention { + required RetentionMode retentionMode = 1; + optional Rule rule = 2; + optional EventHold eventHold = 3; + } + + message EventHold { + required bool enabled = 1; + required Rule rule = 2; + } + + message Rule { + required TimeUnit timeUnit = 1; + required uint64 duration = 2; + } + + enum TimeUnit { + DAYS = 1; + YEARS = 2; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +Two new fields: retention & legalHold. + +```protobuf +message KeyInfo { + // ... existing fields + optional KeyRetention retention = 23; + optional bool legalHold = 24 [default = false]; +} + +message KeyRetention { + required RetentionMode retentionMode = 1; + optional uint64 retentionDate = 2; // epoch milliseconds; the minimum date while the event hold is ON + optional EventHold eventHold = 3; +} +``` + + + +### Access Control + +A request on a locked key must pass two checks: authorization (native ACLs or Ranger) and the WORM check in OM. +Even with `CREATE`, `WRITE` or `DELETE`, overwriting or deleting a locked key is rejected with `403 Access Denied` (`WORMProtectionException`), +unless the Governance bypass applies (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). + +The cluster uses one of two authorization models: +* [Native ACLs](#1-native-acls) have no per-action permissions, so changing locks needs the owner, and setting or changing a Compliance retention needs an Ozone administrator. +* [Ranger](#2-ranger) checks access types, and `action-matches` can limit a policy item to specific S3 actions. Use Ranger when each Object Lock operation needs its own permission. + +In both models, keys that only get the bucket default retention need no extra check, because the default was authorized when it was set. +Native clients pass the same settings and the `bypassGovernanceRetention` flag as request arguments, and the same rules apply. + +--- + +#### 1. Native ACLs + +| Operation | Resource | Required | +|:----------------------------------------------------------------------------------|:-------------|:--------------------| +| `GetBucketObjectLockConfiguration` | Bucket | `READ` | +| `GetObjectRetention`, `GetObjectLegalHold` | Key | `READ` | +| `PutBucketObjectLockConfiguration` (no default, or `GOVERNANCE`) | Bucket | Owner | +| `PutObjectRetention` on a `GOVERNANCE` key or a key without retention (setting `GOVERNANCE`), `PutObjectLegalHold` | Key | Owner | +| Governance bypass (including shortening or removing a `GOVERNANCE` retention) | Key | Owner | +| Release the event hold of a `COMPLIANCE` key (`EventHold=OFF` without `RetainUntilDate`) | Key | Owner | +| Set or change `COMPLIANCE`: on a key (including extending it), `GOVERNANCE` → `COMPLIANCE`, or as the bucket default (also changing a `COMPLIANCE` default) | Bucket / Key | Ozone administrator | + +* **Owner**: the owner of the volume or bucket, the same owner that `OzoneNativeAuthorizer` already lets skip ACL checks. Ozone administrators can do everything an owner can. + Locks use the owner instead of an ACL type like `WRITE_ACL`, because any ACL type can be granted to other users. +* **Ozone administrator**: a user in `ozone.administrators` or a group in `ozone.administrators.groups` (if neither is set, the user that starts OM). + Read-only and S3 administrators are not included. The role is cluster wide; use Ranger for finer control. +* Reading lock information only needs `READ`, because `HeadObject` / `GetObject` already return it in the `x-amz-object-lock-*` headers. +* Every change to a Compliance retention needs an administrator, because no API can undo it: setting it, and also extending it (a later date or a longer event hold duration), + which could otherwise lock a key for up to 100 years. This is stricter than AWS S3, where `PutObjectRetention` is enough. +* The one exception is releasing the event hold of a Compliance key without a `RetainUntilDate`: it only turns the stored duration, which an administrator already approved, into a date, + so the owner can do it when the business event happens. Releasing with a `RetainUntilDate` needs an administrator. +* Keys that get a Compliance bucket default need no administrator check. To store Compliance data, an administrator sets `COMPLIANCE` as the bucket default once, + and applications upload as normal users. + +##### Extra Checks for Object Lock Headers + +Requests that carry Object Lock headers also need the owner or an administrator, checked in the same request after the normal ACL check: + +| Request | Header or condition | Extra check | +|:--------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------|:---------------------------------------------------------| +| `CreateBucket` | `x-amz-bucket-object-lock-enabled: true` | Owner | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `x-amz-object-lock-mode`, `-retain-until-date`, `-event-hold`, `-event-hold-duration-days` / `-years` | Owner (`GOVERNANCE`), Ozone administrator (`COMPLIANCE`) | +| `PutObject`, `CopyObject` (destination), `CreateMultipartUpload` | `x-amz-object-lock-legal-hold` | Owner | +| `DeleteObject`, `DeleteObjects`, overwrite, rename (native clients) | `bypassGovernanceRetention` on a key under Governance retention | Owner | + +If the check fails, the request fails (`DeleteObjects` reports a per-key `AccessDenied`). `GetObject` / `HeadObject` need no extra check and always return the lock headers. +So a user who can only write data cannot set a lock by adding a header. + +--- + +#### 2. Ranger + +> _**Important**_: action-level permissions need the Ranger action matcher, which is `false` by default. +> Without it, every principal with `WRITE` on a key can also change its locks, including setting a Compliance retention. +> ```properties +> ranger.servicedef.ozone.enableActionMatcherInPoliciesCondition=true +> ``` + +* Object Lock uses the same access types as data: `READ` to read lock information, `WRITE` to change it, and `WRITE` and `DELETE` for the Governance bypass. + They apply to every client: S3 Gateway, ofs / o3fs and the Ozone CLI. +* Ranger has no administrator role, so setting Compliance needs no special permission: `PutObjectRetention` (`WRITE`) is enough, unlike native ACLs, where it needs an Ozone administrator. Review Comment: This line says "Ranger has no administrator role", but OM's Ozone administrator check (`ozone.administrators`) doesn't depend on the authorizer. OM already uses it on Ranger clusters, for example `OMSnapshotCreateRequest` only allows an Ozone admin or the bucket owner to create snapshots. This matters because with default Ranger settings (`action-matches` off), any user with `WRITE` on `bucket1/*` can set `COMPLIANCE` with a date 100 years out on every key in the bucket. Nobody can undo that through the API; the only fix is stopping all OMs and running the offline repair. Could Compliance changes need an Ozone administrator in both models, the same as with native ACLs? Just this clarification I need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
