raghuvanshraj opened a new issue, #3685:
URL: https://github.com/apache/iceberg-rust/issues/3685

   Iceberg V3 added the Variant type. iceberg-rust has the type in the schema 
(#2188) and recognizes `arrow.parquet.variant` on Arrow import (#2840), but a 
table with a variant column cannot be read or written yet. #2880 adds the 
unshredded read path; the rest of the track has no issues. This umbrella 
collects it, the same way #1884 does for geo and #3373 for unknown. #2411 lists 
the same rows under v3 tracking; happy to keep that in sync.
   
   I work on a query engine that reads Variant from Parquet through DataFusion 
and wants to do the same from Iceberg tables written by Spark. I tried the 
#2880 branch on a V3 table whose files Spark 4.1 / Iceberg 1.11 wrote with 
`write.parquet.shred-variants=true`: unshredded files come back tagged and 
correct, every shredded file stops at the #3606 error, and `unshred_variant` 
from arrow-rs 60.1 rebuilt all of those rows exactly. So I would like to pick 
up the shredded read path once #2880 lands.
   
   **Read**
   - [x] Variant type in the schema, Avro and Arrow mapping: #2188
   - [x] Recognize `arrow.parquet.variant` on Arrow schema import: #2840
   - [ ] Read unshredded variant columns: #2880
   - [ ] Read shredded variant columns (`typed_value`): #3606
   - [ ] Project id-less variant children by Parquet path: #3609
   
   **Write**
   - [ ] `VARIANT` logical type annotation on write: #2546
   - [ ] Write unshredded variant columns (`metadata` / `value` group): TBD
   - [ ] Write shredded variant columns behind `write.parquet.shred-variants` 
(Java default off): TBD
   
   **Values and statistics**
   - [ ] Convert a variant Arrow array to `Literal`; JSON single-value 
serialization: TBD
   - [ ] Column statistics: no bounds on the variant column itself, bounds on 
shredded typed children: TBD
   - [ ] Predicate pruning using shredded child statistics: TBD
   
   **Integrations**
   - [ ] Carry the `arrow.parquet.variant` extension through the 
datafusion-iceberg scan: TBD (apache/datafusion-iceberg)
   
   Reference implementation in iceberg-go: apache/iceberg-go#589.
   
   Willingness to contribute: I can contribute to this feature independently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to