kevinjqliu opened a new issue, #4156: URL: https://github.com/apache/iceberg-python/issues/4156
### Feature Request / Improvement Several features are only valid in format v3 ([spec, Appendix E](https://iceberg.apache.org/spec/#version-3)), e.g. the new types (`variant`, `geometry`, `geography`, `unknown`, `timestamp_ns`, `timestamptz_ns`), non-null `initial-default`, encryption (`encryption.key-id`, `encryption-keys`, snapshot `key-id`), deletion vectors, and row lineage. PyIceberg handles these inconsistently on pre-v3 tables today. Some checks only run at `create_table`, some features are silently dropped, and others are accepted. We should follow Java here, but Java doesn't cover every case either, so we need to go further. **Rules** 1. **Reject v3 features on pre-v3 tables.** Creating a pre-v3 table, or changing its metadata (schema, properties, partition specs, sort orders), must raise if the change introduces a v3-only feature. Never silently accept, drop, or ignore it. 2. **Allow reads and repairs.** Loading and scanning a pre-v3 table that already contains v3-only features must work. Metadata changes that only remove v3-only features (e.g. `remove_properties("encryption.key-id")`, dropping a column) must succeed, so the table can be repaired from invalid to valid. 3. **Check the version on the write side.** Writers must check the format version before writing v3-only structures (e.g. deletion vectors, encrypted manifest lists, row lineage fields), and raise instead of falling back to a pre-v3 equivalent. 4. **Reject everything else on an invalid table.** If a pre-v3 table contains v3-only features, reject all other writes until it is repaired: data writes (append, overwrite, delete), metadata updates other than repairs, and format version upgrades. Upgrading must not turn an invalid table into a valid one. **Java coverage** | Rule | Java | |---|---| | 1. Reject v3 features | Types and `initial-default` are checked on every schema change ([`Schema.checkCompatibility`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/api/src/main/java/org/apache/iceberg/Schema.java#L604-L637)). Encryption properties are checked only at table creation ([`EncryptionUtil.checkCompatibility`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java#L207-L219)), not on property updates | | 2. Reads and repairs | Allowed, since metadata isn't validated on load | | 3. Write-side checks | Deletion vectors are checked on commit ([`validateDeleteFileForVersion`](https://github.com/apache/iceberg/blob/fb58513787756dedab6550d6695df36a27acb64e/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L295-L315)). Manifest list encryption is not gated by version | | 4. Reject on invalid tables | Not covered: upgrades and other writes are not blocked | Note: Java doesn't fully handle this yet either. Where PyIceberg goes further than Java (rules 1, 3 and 4), we should raise the gaps on the dev list so implementations stay consistent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
