andygrove opened a new issue, #3087:
URL: https://github.com/apache/datafusion-comet/issues/3087
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `date_diff` function, causing
queries using this function to fall back to Spark's JVM execution instead of
running natively on DataFusion.
The DateDiff expression calculates the number of days between two date
values. It computes the difference by subtracting the start date from the end
date, returning a positive value when the end date is later than the start
date, and a negative value when the end date is earlier.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
DATEDIFF(endDate, startDate)
```
```scala
// DataFrame API
col("endDate") - col("startDate") // Using DateDiff internally
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| endDate | DateType | The end date for the difference calculation |
| startDate | DateType | The start date for the difference calculation |
**Return Type:** IntegerType - Returns the number of days as an integer
value.
**Supported Data Types:**
- Input types: DateType only (other types are implicitly cast to DateType if
possible)
- Both arguments must be convertible to DateType through implicit casting
**Edge Cases:**
- **Null handling**: Returns null if either input is null (nullIntolerant =
true)
- **Negative results**: Returns negative integers when endDate is earlier
than startDate
- **Same date**: Returns 0 when both dates are identical
- **Integer overflow**: Theoretically possible with extreme date ranges, but
unlikely in practical usage
**Examples:**
```sql
-- Basic date difference
SELECT DATEDIFF('2009-07-31', '2009-07-30');
-- Returns: 1
-- Negative difference (end date earlier than start date)
SELECT DATEDIFF('2009-07-30', '2009-07-31');
-- Returns: -1
-- Same dates
SELECT DATEDIFF('2009-07-30', '2009-07-30');
-- Returns: 0
-- With null values
SELECT DATEDIFF(NULL, '2009-07-30');
-- Returns: NULL
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(datediff(col("end_date"), col("start_date")).as("days_diff"))
// Using with literal dates
df.select(datediff(lit("2009-07-31"), lit("2009-07-30")).as("days_diff"))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.DateDiff`
**Related:**
- DateAdd - Adds days to a date
- DateSub - Subtracts days from a date
- MonthsBetween - Calculates months between two dates
- TimestampDiff - Calculates differences between timestamps
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
