aboueleyes opened a new issue, #17773: URL: https://github.com/apache/iceberg/issues/17773
### Feature Request / Improvement ### Problem Dropping a column in Iceberg is metadata-only: data files keep their bytes and their footer field IDs, and the old schema stays in the table's `schemas` array. But there is no supported way to bring the column back under its **original** field ID. Re-adding it through any public API assigns a fresh ID (`last-column-id + 1`), which permanently orphans the historical values: old files' bytes can never again be projected, because projection is by field ID. Users hit this in practice: migrations into Iceberg where a source column was dropped by mistake, pipelines that drop and recreate columns during backfills, and at least one real corruption case where the Flink dynamic sink matched input against a historical schema and silently wrote nulls after a delete-plus-re-add cycle. Time travel does not solve it: reading `AS OF` a pre-drop snapshot works until snapshot expiry, but rollback moves only the branch pointer and provably never restores `current-schema-id`, so nothing puts the column back for future writes except a full-table copy. ### Prior work (this has been proposed and reviewed before) This request was filed twice, produced two implementations, and stalled on inactivity rather than review disagreement. To avoid restarting from zero: - [#14426](https://github.com/apache/iceberg/issues/14426) and [#14488](https://github.com/apache/iceberg/issues/14488): the original requests. @Fokko endorsed the mechanics ("reverting current-schema-id should restore the data") and the open question was the engine SQL surface, not feasibility. - [#15084](https://github.com/apache/iceberg/pull/15084): a complete implementation (`UpdateSchema.undeleteColumn(String)` resolving the original field ID from `table.schemas()`, plus a Spark `undelete_column` procedure and tests). It received approvals, then was closed by the stale bot in April 2026. - [#16089](https://github.com/apache/iceberg/pull/16089): successor revision, closed without reviews. - The spec already permits narrow restoration: "Field deletion cannot be rolled back unless the field was nullable or if the current snapshot has not changed" (spec, Schema Evolution). A follow-up audit of #15084 against current main found one correctness bug and several gaps, listed below. Everything else in its design still applies unchanged. ### Proposed semantics (narrow, spec-compliant first pass) `table.updateSchema().undeleteColumn(name)` restores the most recently deleted field with that name under its **original field ID**, so pre-drop data becomes readable again immediately. 1. **Resolution**: search `schemas()` newest-first; the first schema containing `name` wins (latest-wins). If none contains it, fail with a deterministic not-found error. 2. **Nullability**: the restored field is always added as optional, preserving its historical `initial-default` and `write-default` byte-for-byte. Rows written while the column was dropped read the default/null, exactly like rows written before a later `add column`. 3. **Required-column guard** (the spec-compliance core): walk the branch's snapshot ancestry; find the last ancestor whose schema contains the restored field ID; if any newer snapshot exists, writes happened while the field was absent. Restoring a field that is REQUIRED in history must then throw, telling the user to either restore it as optional explicitly or accept the inconsistency. Schema-only commits create no snapshots, so this walk is cheap and exact. 4. **Parent resolution from the CURRENT schema**, not the historical one. (#15084 resolved the parent from the historical schema, which silently no-ops when a nested parent was itself dropped and re-added.) 5. **Case collisions rejected up front**: restoring `Foo` while live `foo` exists would break case-insensitive resolution later; reject with a clear message instead. 6. **Name mapping hygiene**: route the restore through the normal add path so `schema.name-mapping.default` maintenance runs; dedupe by field ID so a stale post-drop entry cannot produce a duplicate-ID mapping (which fails eagerly today). 7. **Identifier fields**: never auto-restored as identifiers. If the field was an identifier, say so in the result and require an explicit `setIdentifierFields` afterwards. 8. **Concurrency**: standard requirements apply (`AssertLastAssignedFieldId`, `AssertCurrentSchemaID`). Note the dedup case: if the resurrected schema equals a historical schema entry byte-for-byte, the builder dedupes to the existing schema-id and emits no AddSchema change; the requirement set must handle that path. ### Engine surface Spark procedure mirroring #15084/#16089: ```sql CALL catalog.system.undelete_column('db.t', 'x'); ``` Returning the restored field ID, the schema-id applied, and whether writes occurred in the drop window (so callers can audit null/default regions). Other engines can adopt later; the core API is the contract. ### Why not the alternatives - Plain `ADD COLUMN`: fresh ID, data stays orphaned forever. This is the trap users fall into unknowingly. - Rollback / RESTORE-style whole-table state revert: loses all post-drop work; Delta needs exactly this compromise within its vacuum window. Iceberg's immutable field IDs make something better possible: precise, retention-independent restoration. - Hand-edited metadata or raw REST commits (`add-schema` + `set-current-schema` referencing a historical ID): verified working end to end on a live catalog during a recent audit, including correct reads and post-restoration writes, which proves the storage machinery needs nothing new. But raw REST bypasses name-mapping maintenance, has no guards, and is expert-only. The feature is that flow is made safe and first-class. ### Scope and effort estimate Core change is small: roughly 100 lines in `SchemaUpdate` (history search plus injecting the historical ID into the pending updates map), the API method, the Spark procedure, and tests. The heavy lifting is the test matrix: delete/re-add/delete cycles, nested parents, identifier fields, required-with-writes, case variants, name-mapping states, dedup-vs-append outcomes, concurrent commits. ### Open questions for reviewers 1. Is latest-wins acceptable for v1, with undelete-by-ID deferred behind a future historical-fields API (as #15084's author noted)? 2. Should restoring a REQUIRED field with post-drop writes always throw, or take an explicit opt-in flag returning optional? 3. Does the project want the SQL surface in the same PR, or core-only first? Happy to drive this, including reviving the #16089 revision with the fixes above. Given that two implementations previously died to inactivity, we would especially appreciate a committer willing to shepherd review early. ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
