cbb330 opened a new issue, #18237: URL: https://github.com/apache/iceberg/issues/18237
### Proposed Change Add an optional, immutable `created-at-ms` field to references (branches and tags) in the table and REST specs. It records when a reference was created and is preserved as the reference moves to new snapshots. **Motivation.** Reference expiration via `max-ref-age-ms` measures the age of the *referenced snapshot*. That intentionally prunes unused branches and tags, but it cannot express a maximum lifetime measured from reference creation: - Each write to a branch resets its age. - A new branch created from an older snapshot is considered old immediately. Many branch workflows need a fixed lifetime: write-audit-publish and audit branches, automated experiment and test branches, and retention or compliance cases where a long-lived branch keeps data reachable after `main` has expired it. Iceberg metadata currently has no reliable source for creation time. Branch creation is usually ref-only, so no snapshot records it. Branch heads move, and older metadata files are not reliably retained. **Proposal.** - Add optional `created-at-ms` to the reference object in table metadata and to the REST `SnapshotReference` schema. - Set it when a reference is created, including implicit creation by a write to a missing branch. - Preserve it through writes, target replacement, retention changes, and rename. - For `set-snapshot-ref`, keep an existing value when an update omits the field, so clients unaware of it cannot clear it. - Treat absence as unknown; never infer it for existing references. - Expose it through the `refs` metadata table. This does **not** change `max-ref-age-ms`, snapshot retention behavior, or any defaults. **Compatibility.** Readers are unaffected. Writers that do not understand the field drop it when rewriting metadata. To keep that from erasing it, the value can't change once set: a REST update that omits it keeps the existing value, and an update that tries to change it is rejected. For catalogs where clients write metadata files directly, the proposal would require format v4 writers to preserve it. **Context.** At LinkedIn, we run [OpenHouse](https://github.com/linkedin/openhouse), an open source control plane for lakehouse tables. Our compliance workflows—member data purges and retention cleanup—run on the `main` branch. Write-audit-publish and experiment branches are created from `main`, but those workflows and other maintenance don't run on them, and we don't want to run that maintenance on short-lived branches. A branch that outlives its intended window therefore keeps data reachable after `main` has purged it. Because we can't tell when a branch was created, we can't bound its lifetime; we have to infer the creation time from snapshot lineage, and once snapshot expiration removes that lineage, the creation time becomes unknown. I plan to contribute the implementation. ### Proposal document _No response_ ### Specifications - [X] Table - [ ] View - [X] REST - [ ] Puffin - [ ] Encryption - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
