andygrove opened a new pull request, #5030: URL: https://github.com/apache/datafusion-comet/pull/5030
## Which issue does this PR close? There is no dedicated issue. This is part of the ongoing effort to provide native support for expressions that currently force a full fallback to Spark. Related to the expression coverage epic #240. ## Rationale for this change `timestampadd` and `timestampdiff` have no Comet handler today, so any query using them falls the entire operator back to Spark. Both are regular expressions (present since Spark 3.3), not `RuntimeReplaceable`, so they reach serde directly. Rather than a native implementation, this PR routes them through the JVM codegen dispatcher. The dispatcher runs Spark's own generated code inside the native Comet pipeline, which keeps the operator native while guaranteeing bit-for-bit Spark compatibility across all supported versions. This avoids the datetime edge-case divergences (timezone and calendar handling) that a `chrono`-based native implementation would be prone to. ## What changes are included in this PR? - `CometTimestampAdd` and `CometTimestampDiff` codegen-dispatch serdes in `datetime.scala`, registered in `QueryPlanSerde`'s `temporalExpressions` map. - Comet SQL file tests `timestampadd.sql` and `timestampdiff.sql` covering all time units, negative quantities, month-end and leap-day rollover, whole-unit truncation, and NULL inputs. - Support status for both functions added to the Supported Spark Expressions guide. `make_interval` was considered but left for a follow-up: its output type is `CalendarIntervalType`, which Comet's columnar layer does not yet support (the dispatcher reports `unsupported output type CalendarIntervalType` and falls back). It can be enabled once CalendarIntervalType support lands (#4898). ## How are these changes tested? New Comet SQL file tests run each query through both Spark and Comet and verify the results match and that Comet executes the expression natively (through the dispatcher) rather than falling back. Coverage includes every supported time unit, positive and negative quantities, month-end and leap-day boundaries, whole-unit truncation toward zero, and NULL inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
