dwsmith1983 commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5603306050
Rebased onto main and the three points are in the head (b438dce52). Bounds: `convert_struct` now looks the source index up with `columns().get` and returns an error naming the requested field, the index, and the child count instead of indexing; a test feeds a struct with fewer children than the mapping expects and asserts the error. A struct column arriving with a non-struct mapping errors the same way rather than passing through. I did not add a type-equality check there, since `convert_array` dispatches on the runtime child type and strict equality would reject coercions it handles. Root last-wins: not deliberate, the top level had simply fallen to the default adapter. With duplicate exact names at the root in case-sensitive mode the remap path now runs, the shadowed earlier fields get a placeholder name so the default adapter's `index_of` lands on the last one, and the nested resolver does the same; Spark 3.5's `clipParquetGroupFields` uses one `.toMap` for root and nested groups. Tests at the adapter level and through `DataSourceExec`, plus one that a field id match still wins over the duplicate. One thing worth knowing: parquet-mr writes two root columns named `d` into a single column chunk keyed by path, so a Spark-written file with that shape reads six interleaved values in Spark and arrow-rs alike, and no Spark-comparable end-to-end assertion exists for it; the Rust scan test uses arrow-rs to write the file. The case-insensitive ambiguity now has its companion test in `struct_field_matching`: `A` and `a` against a requested `a` errors naming both in case-insensitive mode and reads `a` in case-sensitive mode. The opener short circuit is tracked in #5801. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
