peterxcli opened a new pull request, #5161: URL: https://github.com/apache/datafusion-comet/pull/5161
## Which issue does this PR close? Closes #5060. ## Rationale for this change Comet already maps Spark's ANSI interval types to Arrow's native representations, but the native file-scan type checker rejected both types before planning. This caused a Parquet table containing an interval column to fall back to Spark for the entire scan. The types are admitted in `CometScanTypeChecker` instead of the shared `DataTypeSupport` predicate because the latter also gates the Arrow CSV reader, which cannot read interval or duration columns. ## What changes are included in this PR? - Allow `YearMonthIntervalType` and `DayTimeIntervalType` in native file scans. - Add an end-to-end Parquet regression test covering all 13 valid ANSI interval ranges, null and negative values, dictionary encoding for every interval column, multiple row groups, Spark-compatible results, and native Comet execution. ## How are these changes tested? - `make core` - `./mvnw test -Dtest=none '-Dsuites=org.apache.comet.parquet.ParquetReadV1Suite ANSI interval types' -Dscalastyle.skip=true` - The same focused suite with `-Pspark-3.4` - The same focused suite with `-Pspark-4.2` - `./mvnw scalastyle:check spotless:check -DskipTests` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
