patrickswedish opened a new pull request, #26395:
URL: https://github.com/apache/datafusion/pull/26395
## Which issue does this PR close?
Closes #26393.
## Rationale for this change
When reading Parquet files with `datafusion.execution.parquet.coerce_int96`
enabled, `Int96Coercer` failed to coerce INT96 columns embedded inside legacy
Parquet list layouts (such as Spark legacy `repeated group bag { optional int96
array; }`, Hive legacy `repeated group bag { optional int96 array_element; }`,
and two-level lists `repeated int96 array;`).
This occurred because `coerce_int96_to_resolution_impl` constructed list
child paths by hardcoding a standard 3-level intermediate group `".list."` and
appending `unprocessed_child.name()`. In legacy layouts, the intermediate group
is `bag` (or absent in 2-level lists), and the repeated field is named `array`
or `array_element`. As a result, the reconstructed path (e.g. `c0.list.array`)
failed to match the Parquet schema's INT96 column path (`c0.bag.array`),
leaving the columns uncoerced as `Timestamp(Nanosecond)` and causing timestamp
wrapping for values outside the nanosecond range.
## What changes are included in this PR?
1. **Parquet Schema-Aware Path Resolution**: Replaced the hard-coded
`".list."` path assembly with dynamic path resolution against the Parquet
schema tree via `resolve_list_child_path` and `find_parquet_type`.
- Handles standard 3-level lists (`list.element`).
- Handles legacy 3-level lists (Spark `bag.array`, Hive
`bag.array_element`).
- Handles 2-level lists (`repeated array` / `repeated element`).
- Handles 1-level repeated primitives.
- Handles structs containing INT96 fields nested inside legacy lists.
- Falls back gracefully to the standard 3-level convention if the Parquet
node is not found.
2. **Extended List Coverage**: Extended coercion traversal to
`DataType::LargeList` and `DataType::FixedSizeList`.
3. **Comprehensive Unit Tests**: Added unit tests in `schema_coercion.rs`
covering Spark legacy, Hive legacy, 2-level lists, and legacy list-of-struct
with timezone configuration.
## Are these changes tested?
Yes, tested locally with:
```bash
cargo test -p datafusion-datasource-parquet --lib schema_coercion
cargo clippy -p datafusion-datasource-parquet --lib -- -D warnings
```
All 20 schema coercion tests pass cleanly without errors or warnings.
---
*Diff minimization and pre-flight validation performed with [ZeroSlop™
CI](https://github.com/marketplace/actions/zeroslop-by-ribbsaeter-systems) —
[Ribbsaeter Systems](https://ribbsaetersystems.com/).*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
