chungen0126 commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4313481283
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,589 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: draft +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design document aims to plan and implement the Object Lock mechanism for OBS buckets, with authorization through Ozone native ACLs or Apache Ranger. +The primary objective is to provide data immutability and tamper-proof protection through the object locking feature. + +## Problem statement + +With growing demands for data security and compliance, ensuring that critical data stored in Ozone is +protected from accidental or malicious deletion and overwriting has become an essential system protection requirement. +To establish a more rigorous data protection mechanism, we plan to introduce the Object Lock feature. + +Considering the current system architecture and access control strategies, +this design supports both Ozone native ACLs and Apache Ranger to manage Object Lock permissions on OBS buckets. +Native ACLs are used by deployments without Ranger and allow the core logic to be implemented and tested independently of the Ranger integration. +Meanwhile, to accelerate core feature delivery, we have decided to exclude complex multi-version locking (Versioning Lock), +Legacy buckets, and FSO buckets from this initial release. Narrowing the scope allows us to focus on the core functionality and ensure a rapid, +stable rollout of baseline tamper-proof protection. + +**Goal:** + +* Implement the Object Lock feature on standard OBS buckets, with permission and access control through native ACLs or Ranger. +* Support single-version objects only. + +## Non-Goal + +* Versioning Lock: Support for locking across multiple object versions is deferred (multi-version core features are currently under development). +* Legacy and FSO Buckets: Object Lock support for legacy and FSO buckets is excluded. The following considerations are deferred to a future phase: + * Lock Granularity: FSO separates files and directories across FileTable and DirectoryTable, requiring explicit policy decisions on whether lock semantics apply at the file level, directory level, or both. + * Rename & Move Semantics: Renaming or moving paths alters namespace hierarchy without mutating object content; rules must be defined to either block operations on locked paths or preserve lock metadata across the same objectID +* Value-Level Ranger Authorization: a payload-aware Ranger condition evaluator (for example, allowing a legal hold to be set `ON` but not `OFF`) is deferred to a future phase. + +## Differences from AWS + +* Single-Version Model: Locks apply to the current key rather than individual object versions. +* Direct Overwrite Blocking: Unlike AWS S3 (which writes a new version), Ozone strictly rejects Put, Copy, and MPU requests on locked keys with 403 Access Denied (WORMProtectionException). +* Authorization: Ozone has no root account. + * With native ACLs, changing Object Lock settings requires `WRITE_ACL` and bypassing Governance retention is limited to Ozone administrators (see [Native ACL Mapping](#1-native-acl-mapping)). + * With Ranger, AWS-style action-level authorization for S3 requests requires both `ozone.s3g.sts.http.enabled` and `enableActionMatcherInPoliciesCondition` to be enabled; both are `false` by default (see [Prerequisite for S3 Fine-Grained Authorization](#3-prerequisite-for-s3-fine-grained-authorization)). + * Otherwise, and for native clients, Ranger falls back to `READ_ACL` / `WRITE_ACL`, and bypassing Governance retention requires `WRITE_ACL`, `WRITE` and `DELETE` (see [Fallback: Base Permission Mode](#4-fallback-base-permission-mode)). + * A Compliance retention cannot be removed through any API; it can only be changed offline with `ozone repair` (see [Offline Compliance Retention Repair](#offline-compliance-retention-repair)). + +## Technical Description + +### Terminology + +**Legal Hold** + +* **Definition**: Applies an indefinite lock status to an object. The object remains protected until a principal allowed to change the legal hold (`WRITE_ACL` on the key, plus the `PutObjectLegalHold` action in Ranger Fine-Grained Mode) explicitly removes the lock (Remove Legal Hold). +* **Restricted Operations**: + * Put Object + * Delete Object + * Multipart Initial / Complete +* **Allowed Operations**: + * Get Object / Head Object + * Get / Put Object Tagging + * Get Legal Hold + * Put Legal Hold (Depends on permission) +* **Usage**: + * Used to preserve data for litigation, investigations or audits, when it is not known in advance how long the data must be kept. + * A legal hold has no mode and no expiration date. It is independent of retention: a key can have a legal hold, a retention period, or both, + and it stays locked until the legal hold is removed and the retention period has expired. + * A legal hold can be placed on a key in an Object Lock enabled bucket even if the key has no retention. + +**Retention** + +* Definition: Configures a retention policy for an object to prevent deletion or modification. Retention can be duration-based (configured in days or years, establishing a fixed `RetainUntilDate`) or event-driven (**Event Hold / Event-based Retention**, where an object remains protected indefinitely until an external business or legal event triggers the final retention countdown). +* Retention Modes: + * Compliance Mode: The strictest protection tier. Once applied, no user (including root/admin) can remove the lock, shorten the duration, or overwrite the object before the retention period expires. + * Governance Mode: A flexible protection tier. Standard users are restricted by locking rules, but users with `BypassGovernanceRetention` permissions can bypass restrictions to modify or delete the object. +* Event Hold / Event-based Retention: + * Used for records management where the retention lifecycle depends on external events (e.g., contract termination, employee departure, loan closure). + * Keeps the object immutable while awaiting event notification; once the event occurs, the definitive expiration date (`RetainUntilDate`) is calculated and applied. + * While the event has not occurred, `eventHold.enabled = true` keeps the key locked indefinitely. `eventHold.rule` is copied from the bucket default when the key is created, so later changes to the bucket default do not affect existing keys. + * The bucket default can have both `rule` and `eventHold`. In that case a new key gets `retentionDate` = creation time + `rule` and an active event hold; + the key stays locked until the event hold is released and `retentionDate` has passed. With only `eventHold`, `retentionDate` is unset until the hold is released. + * While an event hold is active, a `GOVERNANCE` key can be deleted or overwritten with `bypassGovernanceRetention`; a `COMPLIANCE` key cannot. +* Restricted Operations: + * Put Object / Copy Object (Overwrites) + * Delete Object + * Multipart Upload (Initial / Complete) + * Shorten Retention Period (in Compliance Mode) +* Allowed Operations: + * Get Object / Head Object + * Get / Put Object Tagging + * Extend Retention Period + +#### Event Hold Trigger Workflow +For enterprise governance workflows, the retention lifecycle often depends on specific business triggers rather than a static configuration at creation. The Event Hold mechanism handles this dynamic lifecycle through the following state transitions: + +```text +object created + | +event hold applied (indefinite WORM protection) + | +business event occurs + | +hold released + | +retention timer starts (RetainUntilDate is calculated) + | +WORM until timer expires +``` + +The "business event occurs" step is triggered with `PutObjectRetention`, so no new API or Ranger action is needed +and S3 clients can release the hold through S3 Gateway (native clients use `OzoneBucket#setRetention`). +When OM handles `PutObjectRetention` on a key with an active event hold, it clears `eventHold` under the key write lock Review Comment: Agreed, the previous design could release an event hold too early. The doc now follows the AWS S3 variable retention API: - `PutObjectRetention` carries `EventHold` (`ON` / `OFF`) and `EventHoldDuration`; `PutObject` / `CreateMultipartUpload` accept the `x-amz-object-lock-event-hold*` headers, and the bucket default can include `DefaultEventHold`. - Only an explicit `EventHold=OFF` on a key whose hold is `ON` releases it. Any other `PutObjectRetention` update keeps the event hold active. - The duration becomes a fixed date only on release: OM sets `retentionDate` to the later of the existing date and release time + duration. While the hold is `ON`, `GetObjectRetention` / `HeadObject` return the computed date (the later of now + duration and the stored date). - Changing the duration never shortens the protection: before a decrease, OM saves the current effective date as the minimum, with or without the bypass. - Shortening is done only with an earlier `RetainUntilDate` and the Governance bypass (never for Compliance). While the hold is `ON`, that request must also carry `EventHold`, and with `EventHold=ON` the date cannot be earlier than now + duration. - Invalid requests are rejected with `400`, as in AWS S3: no mode, neither `RetainUntilDate` nor `EventHold`, a duration without `EventHold=ON`, `ON` without exactly one duration, `OFF` with a duration, or `OFF` when there is no future retain-until date. - A new "Retention Updates" table lists the rules: a longer retention, a shorter event hold duration or releasing the event hold only needs `PutObjectRetention` in both modes; an earlier `RetainUntilDate` needs the Governance bypass and is not allowed in Compliance. I checked these rules against AWS S3 (Governance mode) with the AWS CLI, since the AWS docs do not cover all of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
