kevinjqliu opened a new issue, #18646: URL: https://github.com/apache/iceberg/issues/18646
### Apache Iceberg version main (development) ### Query engine Other ### Please describe the bug 🐞 The [spec](https://iceberg.apache.org/spec/#snapshots) defines which metadata fields are allowed or required in each format version. `TableMetadata.Builder` only enforces some of these, and the parsers write whatever is present. Java's own writers build version-correct metadata, but REST servers built on `CatalogHandlers` apply updates from any client. **Cases** Each commits with 200 on the REST fixture and is written to metadata: | Update | Table | Spec | Impact | |---|---|---|---| | `add-encryption-key` | v1, v2 | v3-only | #18632 | | `add-snapshot` with `key-id` | v1, v2 | v3-only | | | `add-snapshot` with `first-row-id`, `added-rows` | v1, v2 | v3-only | Row ID ranges overlap after upgrading to v3 | | `add-snapshot` with `manifests`, no `manifest-list` | v2 | `manifest-list` required in v2+ | PyIceberg 0.12 can't load the table | | `add-snapshot` without `summary` | v2, v3 | `summary` required in v2+ | | **Repro** (REST fixture), `manifest-list` case ```bash docker run -d --rm -p 8181:8181 apache/iceberg-rest-fixture H='Content-Type: application/json'; U=localhost:8181/v1 curl -s -XPOST $U/namespaces -H "$H" -d '{"namespace":["ns"]}' curl -s -XPOST $U/namespaces/ns/tables -H "$H" -d '{"name":"t","schema":{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false}]},"properties":{"format-version":"2"}}' curl -s -XPOST $U/namespaces/ns/tables/t -H "$H" -d '{"requirements":[],"updates":[ {"action":"add-snapshot","snapshot":{"snapshot-id":1,"sequence-number":1,"timestamp-ms":1700000000000,"manifests":["file:/tmp/m1.avro"],"summary":{"operation":"append"}}}, {"action":"set-snapshot-ref","ref-name":"main","type":"branch","snapshot-id":1}]}' # 200. Java loads the table, PyIceberg fails with "manifest-list Field required" ``` **Proposed fix** Add the missing checks to `TableMetadata.Builder`: - `addSnapshot`: reject v3-only fields below v3, require `manifest-list` and `summary` from v2 - `addEncryptionKey`: reject below v3 (#18632) The builder already has similar checks: - [`addSnapshot`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1282-L1292): requires `first-row-id` on v3, [rejects non-monotonic timestamps on v4](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1265-L1275) - [`addSchemaInternal`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1625): rejects v3 types and defaults below v3 - [`addPartitionSpecInternal`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1690-L1693): requires sequential IDs on v1 - [`TableMetadata`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L327-L330): requires sequence number 0 on v1 Only new additions are checked, so existing tables still load and commit. RCK tests would give other REST servers the same coverage. **Out of scope** - `write-default` on v2: forward-compatible per spec - `refs` on v1: long-standing Java behavior **Related** - #18632: encryption keys on v1/v2 - #17728: client-side format version checks on create ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
