mfellner commented on issue #2411: URL: https://github.com/apache/iceberg-rust/issues/2411#issuecomment-6112673705
Thanks again for the pointers @dannycjones. I ran an agent-assisted audit of the v3 spec (1.12.0) against `main` and `v0.11.0-rc.2`. Findings I could reproduce are now separate issues, so each can be triaged on its own. Everything else below is from reading code and issue state on 2026-10-11, so please treat it as a lead list and correct me where I am wrong. **New issues (each has a test that fails on current main)** - #3399 v2 manifest Avro schema includes v3-only data file fields 142, 144 and 145 (the "V3 fields in V1+V2 manifests" case; schema-level only, v1 and manifest lists are clean) - #3404 Date-to-timestamp promoted column reads as NULL when another column is projected - #3401 COW rewrite does not carry row lineage into replacement files on v3 tables - #3402 Commits to v1/v2 tables with `encryption.key-id` write plaintext files - #3403 Table metadata accepts field ids above 2147483447 - #3400 `DeleteVectorIterator::advance_to` mishandles positions at the 2^32 boundary (only reachable on data files with at least 2^32 rows) I also left comments on #3373 (a required `unknown` field is accepted), #2418 (the current-schema case looks fixed on main; non-current schemas are the remaining gap) and #3041 (concrete cases for the version gate). I have two more reproduced findings that I did not file, to avoid adding noise: manifest read fails on a variant `lower_bounds`/`upper_bounds` entry (`spec/manifest/_serde.rs`), and `timestamptz`/`timestamptz_ns` Avro schemas omit `adjust-to-utc: true` (limited by apache-avro, avro-rs#368). Happy to file either if you want them tracked. **Epic body items with no child issue or PR today.** These are big enough that I did not want to open issues for work other people may already be doing; I can open child issues if the owners prefer. - Variant: non-shredded write, shredded read, shredded write, statistics (#2880 is the unshredded read; #2546 is the writer annotation) - Geometry/Geography: manifest bounds and bbox pruning (#2933 and #3342 defer them) - Multi-arg transforms: `SortField` `source-ids` (#2802 mentions it as a follow-up) and eval - write-default: applying it on write (#3250 covers validation; #2803 refers to the DataFusion integration that has moved to apache/datafusion-iceberg) **Possible updates to the epic body**, if useful: #2188 is merged; #1652 and #2879 look implemented by merged work (#1682, #3058); "Gate v3-only metadata fields when reading v1/v2" could point at #3399, #2418 and #3041; zstd (#2894) is not a v3 spec item; the "Stop writing distinct_counts" bullet is gone. Below is a checklist that links every v3 item I found, as of 2026-10-11. It is meant to be copied into the body if it helps. <details> <summary>Proposed checklist</summary> **Types** - [x] Variant type and Arrow recognition: #2188, #2840 - [ ] Variant unshredded read: #2880 - [ ] Variant writer annotation: #2546 - [ ] Variant non-shredded write, shredded read/write, statistics: no issue yet - [x] Unknown type: #2773. [ ] Parquet omit/reconstruct: #3072, #3073. [ ] Umbrella: #3373 - [ ] Geometry/Geography: #1884, #2933, #3341, #3342. Bounds and pruning: no issue yet - [ ] Promoted date to timestamp(_ns): manifest bounds #3387; data read #3404 **Defaults** - [x] initial-default on read for top-level fields (#737, #2832) - [ ] Nested initial-default: #3261 - [ ] Filters on columns missing from the file: #3298, #3301, #3302 - [ ] JSON literal fixes: #3330, #3331, #3332, #3333 (#3334 merged) - [ ] write-default application and validation: #3250, #2803, #1357 **Row lineage** - [x] `first_row_id` inheritance and metadata columns: #2922, #2952, #3058 - [ ] Open housekeeping: #1652, #2879, #2991 - [ ] Lineage on COW rewrite: #3401 **Deletion vectors and deletes** - [x] DV read: #2866, #2868, #3035, #3145 - [ ] DV write: #3232, #2203, #2367, #2580 - [ ] Equality-delete column projection: #3369 - [ ] Delete I/O metrics and sizes: #3283, #3344, #3345 - [ ] `advance_to` at 2^32: #3400 - [ ] File-scoped position deletes: #3291, #3329 **Transforms** - [x] Unbound `source-ids`: #3267. [ ] Bound `PartitionField`: #2802, #2801. [ ] `SortField`: no issue yet - [ ] Unknown transform names: #2789, #2790 **Metadata and version gating** - [ ] v2 manifest schema carries v3 fields: #3399 - [ ] Non-current schemas not checked per format version: #2418 - [ ] Per-action version gating: #3041 - [ ] REST create-table format version: #2923, #3055 - [ ] Field ids above 2147483447: #3403 - [ ] zstd metadata (not in the spec): #2894 - [ ] v1 manifest entry with null `snapshot_id`: #3371 **Encryption** - [ ] Umbrella: #2034. [ ] KMS: #3173. [ ] `StaticTable`: #2915. [ ] v1/v2 with `encryption.key-id`: #3402 </details> Found during an agent-assisted audit of v3 support. I am not a Rust expert, so I would welcome review of the issues I filed. Happy to help where it is useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
