andygrove opened a new pull request, #4541:
URL: https://github.com/apache/datafusion-comet/pull/4541
## Which issue does this PR close?
Part of #4540.
## Rationale for this change
Comet had no support for Spark's interval data types, so every expression
producing or consuming an interval forced a fallback to Spark, and the JVM
codegen dispatcher (`CometCodegenDispatch`) could not cover them either because
`CometBatchKernelCodegen.isSupportedDataType` rejected interval output. This PR
adds the foundational type support and uses the existing codegen-dispatch
mechanism to make the interval constructors run natively while matching Spark
exactly.
## What changes are included in this PR?
Type support for `YearMonthIntervalType` and `DayTimeIntervalType`:
- `native/proto/src/proto/types.proto`: add `YEAR_MONTH_INTERVAL` (18) and
`DAY_TIME_INTERVAL` (19) to `DataTypeId`.
- `native/core/src/execution/serde.rs`: map them to Arrow
`Interval(YearMonth)` and `Duration(Microsecond)`.
- `Utils.toArrowType` / `fromArrowType`: same mapping on the JVM side.
- `QueryPlanSerde.serializeDataType`: emit the new type ids.
- `CometBatchKernelCodegen`: accept both interval types in
`isSupportedDataType`, resolve `IntervalYearVector` / `DurationVector`, and
emit primitive `set()` writes.
Representation note: Spark stores `DayTimeIntervalType` as microseconds in
an `int64`, which round-trips faithfully through Arrow `Duration(Microsecond)`.
Arrow `Interval(DayTime)` uses a `{days, millis}` layout that would lose
microsecond precision, so it is intentionally not used.
Expressions (via `CometCodegenDispatch`, registered in
`temporalExpressions`):
- `make_ym_interval` -> `CometMakeYMInterval`
- `make_dt_interval` -> `CometMakeDTInterval`
Out of scope (follow-ups tracked in #4540): `CalendarIntervalType`, and
interval arithmetic / `multiply_*_interval` / `extract` of interval fields.
## How are these changes tested?
SQL file tests under
`spark/src/test/resources/sql-tests/expressions/datetime/`
(`make_ym_interval.sql`, `make_dt_interval.sql`) run by
`CometSqlFileTestSuite`. The `query` mode uses `checkSparkAnswerAndOperator`,
asserting both answer parity with Spark and native execution (a fallback fails
the test). Cases cover column and literal inputs, default arguments, negatives,
and nulls. The full SQL suite shows no new regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
