mattfaltyn opened a new issue, #1951:
URL: https://github.com/apache/iceberg-go/issues/1951
### Description
Scanning a format-v2 table fails after an equality-delete key column is
dropped from the current schema, even when the scan selects only surviving
columns.
Equality delete files store stable table field IDs, and Iceberg retains
historical schemas after a field is dropped. The read path currently resolves
equality field IDs only through `metadata.CurrentSchema()`, so a live delete
file becomes unreadable as soon as its key is absent from that schema.
### Expected behavior
The scan should resolve the equality field ID from schema history, apply the
delete, and return only the requested current-schema columns.
This matches the existing rewrite invariant:
`validateRewriteEqualityFieldIDs` accepts equality fields retained only in
schema history, and `TestReplaceFilesWithDeleteFilesAllowsDroppedEqualityField`
permits rewriting such a delete file.
### Actual behavior
Consuming the Arrow record iterator fails with:
```text
equality delete field ID 1 not found in table schema for
.../data/delete.parquet
```
The failure occurs before equality-delete filtering can run.
### Reproduction
1. Create a format-v2 table with `id: long` (field ID 1) and `data: string`.
2. Append rows `(1, "keep")` and `(2, "delete")`.
3. Commit an equality-delete file keyed by field ID 1 that deletes `id = 2`.
4. Confirm the scan returns only `(1, "keep")`.
5. Drop `id` with `UpdateSchema(...).DeleteColumn([]string{"id"})` and
commit.
6. Scan only `data` and consume the record iterator.
The iterator returns the error above instead of one row containing `"keep"`.
I reproduced this twice on current `main` at
`0bf8092c8586baa279a5378afddb678baafed4a8` with Go 1.26.0 on macOS arm64. The
same current-schema lookup is present in v0.6.0.
### Root cause
`readEqualityDeleteFile` calls `tableSchema.FindColumnName(fid)`, while
`arrowScanner` supplies only `metadata.CurrentSchema()`. Unlike rewrite
validation, this path cannot fall back to the table's historical schemas.
### Suggested fix
Resolve each equality field ID from current or historical table schemas for
physical delete evaluation while preserving the current schema and requested
projection for returned records. Add an end-to-end regression test that drops
the key and scans a surviving column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
