peterxcli opened a new pull request, #5740: URL: https://github.com/apache/datafusion-comet/pull/5740
## Which issue does this PR close? Partially addresses #5553. The remaining filtered-scan mismatch is tracked in #5739, including a verified reproducer and the dependency on apache/datafusion#20871. This PR does not close either issue. ## Rationale for this change An unfiltered read of an overflowing Parquet `TIMESTAMP_MILLIS` inside a struct, list, or map currently returns NULL, while Spark throws an overflow error. The filtered-scan safeguard now applies to the entire scan, so unfiltered nested reads can use checked conversion independently of the remaining pruning work. ## What changes are included in this PR? - Apply the existing checked millisecond-to-microsecond conversion recursively to nested fields in unfiltered scans. - Carry parent validity and list/map offset visibility through recursion so values hidden by null ancestors or array slices cannot trigger false overflow errors. Preserve required-child validity. - Extend native and Parquet scan regression coverage for nested timestamps, including dictionary/plain encoding, LTZ/NTZ, positive/negative overflow, and null/sliced containers. Filtered scans retain their existing safe-cast fallback. Removing it requires additional work: DataFusion 55 supports primitive struct-field row filters, but nested statistics pruning and conversion-aware predicate handling remain necessary for Spark-compatible error behavior. See #5739. ## How are these changes tested? Validated after rebasing onto upstream main: - `cargo test -p datafusion-comet test_millis_to_micros --lib --offline`: 2 passed (with the local JDK library directory on `DYLD_LIBRARY_PATH`). - `cargo build -p datafusion-comet --all-targets --offline`: passed. - `cargo fmt --all --check`: passed. - `SPARK_LOCAL_IP=127.0.0.1 ./mvnw -o test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite TIMESTAMP_MILLIS"`: 3 passed on Spark 4.1; Maven compilation and formatting/style checks passed. The separate #5739 reproducer was also verified with Comet row-filter pushdown both disabled and enabled: Spark throws `ArithmeticException: long overflow`, while Comet returns a NULL timestamp. Its source is included in the issue rather than adding an assertion of the known bug to this PR's regression suite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
