sam-1112 opened a new pull request, #7196: URL: https://github.com/apache/datafusion-comet/pull/7196
## Which issue does this PR close? Closes #6994. ## Rationale for this change When one Parquet field is requested by both ID and name, Comet returns NULL for the losing name match at the root but duplicates the values inside a struct. Apply shared claim arbitration at both levels so the ID match receives the data and the name match reads NULL, following Spark parquet-mr’s behavior for this case. ## What changes are included in this PR? - Move field claim precedence into `FieldResolver`, shared by root remapping, nested conversion, and plan-time conversion checks. - Resolve repeated requested ID claims with last-wins semantics, matching Spark parquet-mr. Retain the existing JVM fallback for these schemas. - Preserve shadowing, placeholder naming, ambiguity checks, and error ordering. - Prevent generic structural casts from bypassing claim arbitration. - Add regressions covering nested containers, projections, pruning, filter pushdown, case sensitivity, ID configuration, off-heap reads, and repeated requested IDs. ## How are these changes tested? Local validation: - Native library tests: 982 passed, 5 ignored. - Final focused regression tests on Spark 4.1.3: 69 passed. - Focused regression matrix passed on Spark 3.4.3, 3.5.9, 4.0.4, and 4.2.0. - Full `ParquetReadV1Suite` and `CometNativeReaderSuite` passed on Spark 4.1.3. - Formatting checks passed. The cross-profile and full-suite runs preceded the final test cleanup, which removed three redundant root pruning cases and reused the parquet-mr helper. The resulting 69 focused cases were rerun successfully on Spark 4.1.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
