swanandx opened a new pull request, #3683: URL: https://github.com/apache/iceberg-rust/pull/3683
Merge `typed_value` back into `value` with arrow-rs `unshred_variant`. Also skip variant leaves in the INT96 coercion, and skip null struct rows when a required variant is unshredded. ## Which issue does this PR close? - Closes #3606. ## What changes are included in this PR? The reader can now read shredded variant columns. It uses arrow-rs `unshred_variant` to merge `typed_value` back into `value`. The caller gets the same rows as from an unshredded column. Before this PR, two more cases stopped the read of a shredded file with an error. This PR also corrects these cases: - A shredded timestamp with nanosecond precision. The INT96 coercion now ignores the leaves of a variant. - A required variant in an optional struct, when a row of the struct is null. This PR adds the dependency `parquet-variant-compute`. ## Are these changes tested? Yes: - Unit tests shred variants with arrow-rs, write a Parquet file, and read the file again. - On my computer, I read a file from iceberg-java 1.12 that has all shredded types. All rows are correct. - On my computer, all 29 tests in `TestVariantShredding` pass with the Comet Iceberg 1.11 / Spark 4.1 suite. Without this PR, 7 of these tests stop with `FeatureUnsupported`. ## AI Disclosure I used Claude Code to write the code, the tests. I read the full diff and did the tests that this description gives. Examine these areas: - The correct result depends on `unshred_variant`. arrow-rs marks its variant support as experimental. - `unshred_variant` ignores unknown fields in nested shredded groups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
