raghuvanshraj opened a new issue, #3685: URL: https://github.com/apache/iceberg-rust/issues/3685
Iceberg V3 added the Variant type. iceberg-rust has the type in the schema (#2188) and recognizes `arrow.parquet.variant` on Arrow import (#2840), but a table with a variant column cannot be read or written yet. #2880 adds the unshredded read path; the rest of the track has no issues. This umbrella collects it, the same way #1884 does for geo and #3373 for unknown. #2411 lists the same rows under v3 tracking; happy to keep that in sync. I work on a query engine that reads Variant from Parquet through DataFusion and wants to do the same from Iceberg tables written by Spark. I tried the #2880 branch on a V3 table whose files Spark 4.1 / Iceberg 1.11 wrote with `write.parquet.shred-variants=true`: unshredded files come back tagged and correct, every shredded file stops at the #3606 error, and `unshred_variant` from arrow-rs 60.1 rebuilt all of those rows exactly. So I would like to pick up the shredded read path once #2880 lands. **Read** - [x] Variant type in the schema, Avro and Arrow mapping: #2188 - [x] Recognize `arrow.parquet.variant` on Arrow schema import: #2840 - [ ] Read unshredded variant columns: #2880 - [ ] Read shredded variant columns (`typed_value`): #3606 - [ ] Project id-less variant children by Parquet path: #3609 **Write** - [ ] `VARIANT` logical type annotation on write: #2546 - [ ] Write unshredded variant columns (`metadata` / `value` group): TBD - [ ] Write shredded variant columns behind `write.parquet.shred-variants` (Java default off): TBD **Values and statistics** - [ ] Convert a variant Arrow array to `Literal`; JSON single-value serialization: TBD - [ ] Column statistics: no bounds on the variant column itself, bounds on shredded typed children: TBD - [ ] Predicate pruning using shredded child statistics: TBD **Integrations** - [ ] Carry the `arrow.parquet.variant` extension through the datafusion-iceberg scan: TBD (apache/datafusion-iceberg) Reference implementation in iceberg-go: apache/iceberg-go#589. Willingness to contribute: I can contribute to this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
