symious commented on code in PR #10822: URL: https://github.com/apache/ozone/pull/10822#discussion_r3763144329
########## hadoop-hdds/docs/content/design/s3-versioning.md: ########## @@ -0,0 +1,433 @@ +--- +title: S3-compatible Object Versioning +summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation +date: 2026-07-21 +jira: HDDS-15728 +status: accepted +author: Symious +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Summary + +Add S3-compatible object versioning to Ozone: the full three-state bucket state +machine (Unversioned / Enabled / Suspended), per-key version chains with delete +markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in +version reclamation (`maxVersions`, background expiration) — with O(1) metadata +cost per version operation and zero regression on non-versioned paths. + +# Status + +Defined in the markdown header. + +# Problem statement (Motivation / Abstract) + +Amazon S3 provides bucket-level object versioning: a single key can retain multiple +versions, so users can recover objects that were accidentally overwritten or +deleted. A large part of the S3 ecosystem (backup software, data lake components, +DR tooling) depends on the versioning APIs (`PutBucketVersioning`, +`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an +S3-compatible API through the S3 Gateway but does not support object versioning +today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended +state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record +rather than object versions, and the gateway has no versioning endpoints. + +This proposal implements versioning with S3-compatible semantics, usable by +standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a +version operation is decoupled from the number of versions (one extra small KV +write per operation), and reclamation controls are built into the feature itself +to avoid the unbounded version accumulation problems commonly seen on S3 (the S3 +troubleshooting guide documents list degradation and throttling on keys with +millions of versions, and leaves the fix to user-configured Lifecycle rules that +are often forgotten). + +# Non-goals + +- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no + counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning` + returns NotImplemented. +- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities + that versioning itself requires are included. +- **Version-aware cross-cluster replication.** +- **Versioning for FSO / LEGACY bucket layouts** — the first version supports + OBJECT_STORE buckets only; enabling versioning on other layouts returns + NotImplemented. Combining FSO's directory/rename semantics with per-key version + chains is disproportionately complex, and S3 tooling scenarios essentially use + the OBS layout. FSO support can be evaluated as an independent follow-up. +- **Coexistence with Ozone snapshots on the same bucket** — the version-aware + reclamation the combination needs is deferred past the first version, so OM + rejects the combination outright rather than leaving it to convention. The + snapshot section below states the enforcement and what lifts it. + +# Technical Description (Architecture and implementation details) + +## Bucket state machine + +``` +Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended + │ ▲ + └──────suspend────────┘ + (Once Enabled, a bucket can never return to Unversioned) +``` + +A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` / +`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and +`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in +two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are +interpreted via the boolean), so old and new clients/OMs coexist during rolling +upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back +to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's +data-protection promise that no single state change can silently destroy +historical versions. + +## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent) + +A new column family, **versionedKeyTable**, splits responsibilities with keyTable: + +- **keyTable** (existing, semantics unchanged) always holds each key's **current + version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects + read paths are unchanged. +- **versionedKeyTable** (new) holds all **noncurrent** versions (including + noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is + + ``` + /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId} + ``` + + (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key + names contain `/` verbatim, which would interleave a key's versions with those of + keys nested under it), so all versions of a key are physically adjacent and + ordered newest to oldest: `ListObjectVersions` and version promotion are a + single seek plus a sequential read. The table is registered in + `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so + bucket-prefixed iteration and SST filtering resolve its prefix. + +`OmKeyInfo` gains three optional proto fields (old records deserialize +compatibly): `versionId` (int64, assigned once at version creation, then frozen), +`isDeleteMarker` (a marker is a record with this flag and no data blocks — no +datanode storage), and `isNullVersion` (the single overwritable "null version" +slot per key). Keys written before versioning was enabled are interpreted as null +versions on read — **zero migration**, matching S3's "enabling versioning does +not change existing objects". + +Every keyTable ↔ versionedKeyTable update rides OM's existing atomic +multi-table WriteBatch commit (the same double-buffer pattern used today for +keyTable + deletedTable on overwrite): no new transaction mechanism and no +cross-table consistency problem. The new column family is introduced under the OM +layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`): +before finalization, requests carrying a versioning status are rejected. + +## VersionId: a pluggable generator + +`versionId` generation is abstracted behind a `VersionIdGenerator` interface, +chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so +a deployment can plug in its own. The generator is cluster-wide and may be changed +on a running cluster; it is not recorded in bucket metadata. Every generator must Review Comment: It will be guaranted on commiting, versionId less then currentVersion will be rejected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
