mbutrovich opened a new pull request, #7575: URL: https://github.com/apache/datafusion-comet/pull/7575
## Which issue does this PR close? Part of #4295. ## Rationale for this change This validates apache/iceberg-rust#2880, which adds unshredded Variant reads to iceberg-rust, against Comet's Iceberg suites before it merges. ## What changes are included in this PR? The native dep is temporarily pinned to the apache/iceberg-rust#2880 head on the author's fork. - `CometScanRule`: allow Variant at any depth in projected Iceberg columns. Empty or unresolved projections still fall back. - `CometExecRule`: let the native Iceberg scan output Variant, as `CometScanExec` already can. - No native change. The scan's schema adapter already converts iceberg-rust's `struct<metadata, value>` to Spark's `struct<value, metadata>`. ## How are these changes tested? - `CometIcebergNativeSuite`: the Variant tests now check native reads against Spark for top-level, struct, array, and map Variants, including Variant null and SQL null, and a historical snapshot after a rename. - The Iceberg 1.11 / Spark 4.1 suite reads Variants through the native scan, including `TestSparkVariantRead`. `TestVariantShredding` should fail where it reads a shredded table, because iceberg-rust rejects shredded Variants. - Three `CometIcebergNativeSuite` tests on files without field ids fail because of apache/iceberg-rust#3437, which the new pin picks up, not because of Variant support: `nested schema pruning on data files without field ids`, `migration - INT96 timestamp`, and `filter with nested types in migrated table`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
