peterxcli opened a new pull request, #5751: URL: https://github.com/apache/datafusion-comet/pull/5751
## Which issue does this PR close? Closes #5707. ## Rationale for this change Comet can retain DataFusion's generic `CastExpr` for pure structural narrowing of nested Parquet columns, allowing DataFusion's nested leaf pruning to read only the requested fields. Retaining that cast is safe only when DataFusion's exact-name lookup agrees with Spark's configured field-name resolver. An exact match alone is insufficient: with case-insensitive resolution, a file containing `s: struct<ID: bigint, id: bigint>` and a requested schema of `s: struct<id: bigint>` is ambiguous. DataFusion's generic cast can select the exact `id` field, whereas Spark and Comet's Parquet converter reject the duplicate match. The same problem applies to non-ASCII names such as `CAFÉ` and `café`, and to structs nested inside other structs or lists. The implementation merged in #5262 already requires exactly one source match under Spark's configured resolver at each nested struct level, in addition to an exact-name match. Ambiguous narrowing therefore falls back to `CometCastColumnExpr`, preserving the existing duplicate-field error. This PR adds the missing regression coverage for that behavior; it does not change production code. ## What changes are included in this PR? - Add a native regression covering ASCII and non-ASCII sibling names with an exact requested match, directly in a struct and inside nested structs and lists. - Check both the narrowing predicate and the actual expression-adapter rewrite. Case-insensitive ambiguity must select `CometCastColumnExpr`; case-sensitive exact matches must retain DataFusion's `CastExpr` so valid pruning remains enabled. - Parameterize the existing native-reader regression to request both mixed-case `Café` and exact-case `café` from `CAFÉ`/`café` siblings. Both reads must raise a duplicate-field error, and the test asserts that the plan uses `CometNativeScanExec`. ## How are these changes tested? - 14 native structural-narrowing tests passed. - 2 focused `CometNativeReaderSuite` tests passed on Spark 4.1. - Rust formatting, Maven Spotless, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
