ChenSammi commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4356657966
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,598 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design adds S3 Object Lock to OBS buckets, so keys can be protected from deletion and overwrite. +Authorization uses Ozone native ACLs or Apache Ranger. + +## Problem statement + +Data stored for security or compliance must be protected from accidental or malicious deletion and overwrite. Object Lock provides this protection. + +The design supports both native ACLs and Ranger. Native ACLs serve deployments without Ranger, and let the core logic be built and tested without the Ranger integration. +To ship the core feature sooner, the first release excludes versioning, Legacy buckets and FSO buckets. + +**Goal:** + +* Object Lock on OBS buckets, with authorization through native ACLs or Ranger. +* Single-version keys only. + +## Non-Goal + +* Versioning: locks across multiple object versions (versioning is still under development). +* Legacy and FSO buckets. FSO needs its own decisions on: + * Lock granularity: files and directories are in different tables (FileTable / DirectoryTable), so locks could apply to files, directories or both. + * Rename and move: they change the namespace without changing the data, so they must either be blocked on locked paths or keep the lock with the same objectID. +* Value-level Ranger authorization, for example allowing a legal hold to be set `ON` but not `OFF`. + +## Differences from AWS + +* Single version: locks apply to the current key, not to object versions. +* Overwrites are rejected: in AWS S3, Put, Copy or MPU on a locked object writes a new version. In Ozone it would replace the locked key, + so Ozone rejects Put, Copy, MPU and rename (native clients only) on locked keys with `403 Access Denied` (`WORMProtectionException`). +* Governance bypass on overwrite and rename: a Governance-locked key can only be overwritten or renamed with the `bypassGovernanceRetention` flag and the bypass permission. + AWS S3 does not need this because it writes a new version (see [Retention Modes and Governance Bypass](#3-retention-modes-and-governance-bypass)). +* Authorization: Ozone has no root account. + * Native ACLs: changing locks and bypassing Governance need the volume or bucket owner; setting or changing a Compliance retention needs an Ozone administrator (see [Native ACLs](#1-native-acls)). + * Ranger: action-level permissions need `enableActionMatcherInPoliciesCondition`, which is `false` by default (see [Ranger](#2-ranger)). + Object Lock uses the same access types as data (`READ` / `WRITE`), so without `action-matches`, anyone who can write a key can also change its locks. + * No API can remove a Compliance retention, in either model. Only the offline `ozone repair` command can (see [Offline Compliance Retention Repair](#offline-compliance-retention-repair)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* Locks a key with no end date, until someone allowed to change the legal hold removes it + (the owner with native ACLs; `WRITE` with Ranger, limited to `PutObjectLegalHold` when `action-matches` is used). +* Used for litigation, investigations or audits, when it is not known how long the data must be kept. +* Has no mode and no date, and is independent of retention: a key can have a legal hold, a retention, or both, and stays locked until both are gone. + A legal hold can be set on a key without a retention. +* Blocks: Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients). +* Allows: Get / Head, Get / Put Object Tagging, Get Legal Hold, Put Legal Hold (with permission). + +**Retention** + +* Protects a key until a `RetainUntilDate`. The date is either fixed (set directly, or from days / years) or comes from an event hold. +* Modes: + * Compliance: no one, including administrators, can remove it, shorten it or overwrite the key through the Ozone API before it expires. Only the offline `ozone repair` command can change it. + * Governance: blocks normal users, but users with the bypass permission can bypass it. +* Event hold (AWS S3 *variable retention*): + * Used when the retention depends on an external event, such as a contract ending, an employee leaving or a loan closing. + * While the event has not happened (`EventHold=ON`), the key is locked with no end date. When it happens, the hold is released (`EventHold=OFF`) + and the duration (`EventHoldDuration`, in days or years) becomes a fixed `RetainUntilDate`. + * See [Event Hold Trigger Workflow](#event-hold-trigger-workflow) for the rules. +* Blocks (Governance allows them with `bypassGovernanceRetention`): Put / Copy (overwrite), Delete, Multipart Upload (initiate / complete), Rename (native clients), + shortening or removing the retention (never allowed for Compliance). +* Allows: Get / Head, Get / Put Object Tagging, extending the retention. + +#### Event Hold Trigger Workflow + +```text +object created + | +event hold applied (locked with no end date) + | +business event occurs + | +hold released + | +RetainUntilDate is calculated + | +locked until RetainUntilDate +``` + +Event holds use `PutObjectRetention`, as in AWS S3, so no new API or Ranger action is needed (native clients use `OzoneBucket#setRetention`). +OM handles the request under the key write lock and uses the leader-stamped request time as now. + +The key stores `retentionDate` and the event hold (status and duration): +* The duration comes from the request, or from the bucket default when the key is created. Later changes to the bucket default do not affect existing keys. +* If the bucket default has both `rule` and `eventHold`, a new key gets `retentionDate` = creation time + `rule` and an active event hold. + With only `eventHold`, `retentionDate` is unset until the hold is released. +* While the hold is `ON`, the effective `RetainUntilDate` is the later of `retentionDate` and now + duration, so it moves forward over time. + `GetObjectRetention`, `HeadObject` and `GetObject` return this date. + +Below, "earlier" and "later" compare the requested `RetainUntilDate` with the effective date. Only `GOVERNANCE` with the bypass can make it earlier; `COMPLIANCE` never can. + +1. **Request format**: checked before the lock and the bypass; a request that breaks a rule gets `400 Bad Request`. + * The request has a mode, and `RetainUntilDate`, `EventHold` or both. + The only exception is an empty retention, which removes the retention: it needs the bypass for `GOVERNANCE` (`403` without it) and is never allowed for `COMPLIANCE`. + * `RetainUntilDate` is in the future. + * `EventHold=ON` has exactly one `EventHoldDuration` (positive, at most 36,500 days or 100 years). A `RetainUntilDate` in the same request is at least now + duration. + * `EventHold=OFF` has no `EventHoldDuration`, and `EventHoldDuration` needs `EventHold=ON`. + * The same applies to the `x-amz-object-lock-*` headers at creation: the mode header needs the retain-until-date header or the event hold headers, and they need the mode header. +2. **`EventHold=ON`** sets or updates the hold. + * Without the bypass, `RetainUntilDate` can only be later (`403` otherwise). The duration can change; + when it gets shorter, OM first saves the current effective date in `retentionDate`, so the date never moves earlier, even with the bypass. + * With the bypass (`GOVERNANCE` only), `RetainUntilDate` can be earlier, but still at least now + the new duration (rule 1). +3. **`EventHold=OFF`** releases the hold. It is the only way to release it. + * Without `RetainUntilDate`: `retentionDate` = the later of `retentionDate` and now + duration. + * With `RetainUntilDate`: that date is used. An earlier date needs the bypass (`GOVERNANCE` only), otherwise `403`. + * On a key without an active hold: no change if the key has a future `retentionDate`; `400` if neither the key nor the request has a future date. +4. **No `EventHold`** keeps the hold as it is and only changes `RetainUntilDate`. + * A later date is accepted, also while the hold is `ON`. + * An earlier date needs the bypass (`GOVERNANCE` only), otherwise `403`. While the hold is `ON`, it gets `400` even with the bypass: the request must use rule 2 or 3. + +Releasing the hold keeps the mode. A different mode in the request is a mode change (see [Retention Updates](#retention-updates)). + +> _**Note**_: AWS S3 requires versioning for Object Lock, so a Put on a locked object only adds a new version and Object Lock mainly blocks deletes. +> Ozone has no versioning yet, so it rejects every overwrite of a locked key. + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + optional bool objectLockEnabled = 24 [default = false]; + optional BucketRetention defaultRetention = 25; + } + + message BucketRetention { + required RetentionMode retentionMode = 1; + optional Rule rule = 2; + optional EventHold eventHold = 3; + } + + message EventHold { + required bool enabled = 1; + required Rule rule = 2; + } + + message Rule { Review Comment: Can we rename this to EventHoldDuration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
