andygrove opened a new pull request, #7028: URL: https://github.com/apache/datafusion-comet/pull/7028
## Which issue does this PR close? Closes #6934. Closes #6790. **Stacked on #6851.** The first five commits are #6851's and will drop out when it merges and this branch is rebased. The change for this PR is the last commit, `5fe8f8bafd`. Please review only that one. ## Rationale for this change Since #6543, the native Iceberg scan falls back to Spark whenever a projected struct, list or map column has a nested field that some schema in the table's history lacks or names differently. That was needed because iceberg-rust matched a data file's nested fields to the table's by position (apache/iceberg-rust#2617). apache/iceberg-rust#3255 matches them by field id and fills absent optional fields with NULLs, so most of those fallbacks are no longer needed. The same bump picks up apache/iceberg-rust#3384, which applies a name mapping to nested fields and fixes #6790. ## What changes are included in this PR? - Moves the `iceberg` and `iceberg-storage-opendal` pin to iceberg-rust `b154603a`. `Cargo.lock` changes only those three crates, and no Comet source change was needed to compile. - Replaces `IcebergReflection.nestedFieldsAddedOrRenamed` with `structsWithNoFieldsInCommon`. The scan now falls back only when a projected struct, at any depth, shares no field id with the same struct in some schema in the table's history. That is the one shape where the native read still differs from Spark: it returns a struct of NULLs or NULL depending on what else the scan reads, where Spark returns NULL for a top-level struct and a struct of NULLs for a nested one. A column the table schema has is checked against the full table schema, since the scan schema holds only the fields Spark prunes to. - Narrows the limitation bullet in `docs/source/user-guide/latest/iceberg.md`. ## How are these changes tested? In `CometIcebergNativeSuite`: - The four "falls back" nested-evolution tests (added to a struct, added inside an array or map, dropped and re-added, renamed) now assert native reads that match Spark. The self-join query allows two native scans. - A new test covers the fallback shapes: top-level struct, list element, nested struct, and a pruned `s.b` read. A struct that kept one of its fields stays native. - A new test reads the struct, list and map fields of a `SparkActions.snapshotTable` table through its name mapping (#6790). On Spark 4.1 with Iceberg 1.11, `CometIcebergNativeSuite` gives 126 passed, 0 failed, 1 canceled, and the 165 iceberg native lib tests pass. I have not run the Spark matrix or the other Iceberg shards. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
