aboueleyes opened a new pull request, #17785:
URL: https://github.com/apache/iceberg/pull/17785
This implements the restoration design discussed in #17773 (and previously
attempted in
#15084/#16089): a dropped column can be brought back under its **original
field ID**, making data files written before the drop readable again without
rewriting anything.
## What it does
- `UpdateSchema.undeleteColumn(name)` resolves the most recently deleted
incarnation of that name from the table's schema history and restores it under
its original field ID. Type, doc, and default values are preserved.
- Requiredness is kept when no rows were written while the column was
absent; restoring a required column whose absence window contains writes fails
with the reason, mirroring the spec sentence that field deletion "cannot be
rolled back unless the field was nullable or if the current snapshot has not
changed".
- Guards reject: names that still exist (including case-insensitive
collisions against live and
pending-added columns), double undeletes in one update, historical IDs
still live under another name (a rename hole), undeleting into a parent deleted
in the same update, and dotted names that would resolve across struct
generations.
- `MappingUtil.update` now merges reused field IDs into the name mapping
instead of emitting duplicate-ID entries (which fail NameMapping construction).
- New `system.undelete_column` procedure on Spark 3.5, 4.0 and 4.1 returns
`restored_field_id`, `applied_schema_id`, and a conservative
`wrote_during_window` flag for auditing.
## Semantics notes
- History lookup is case-sensitive; an additional always-on case-insensitive
collision check is stricter than `addColumn` by design.
- The lineage walk follows the current branch only; `wrote_during_window`
reports `true` unless the current branch's lineage proves no writes happened
during the window.
- Same-batch compositions follow existing `addColumn` conventions:
delete+undelete replaces in place, undelete+rename needs two commits.
## Testing
- Core: new `TestSchemaUndelete`, 24 behaviors x format versions 1-4 (96
executions), including
latest-wins resolution, all four required-guard failure modes, same-batch
interaction matrix,
name-mapping merge, defaults preservation, schema-id dedup reuse, and a
regression test for the parent-resolution bug found in the earlier PR (#15084).
- Spark: full procedure suite on 3.5 (24 tests over
hive/hadoop/rest/spark_catalog catalogs) and registry smoke tests on 4.0/4.1.
- Verified end to end against a live Nessie catalog: drop -> restore via
RESTCatalog -> pre drop values readable -> post-restore writes land under the
restored ID.
Fixes #17773
---
**AI Disclosure**
- Model: ox-alpha
- Platform/Tool: opencode CLI
- Human Oversight: fully reviewed
- Prompt Summary: Implemented the restoration design proposed in issue
#17773 following review
guidance recorded there: core undeleteColumn with original-field-ID
resolution and safety
guards, Spark procedures, docs, and tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
