andygrove opened a new issue, #3108:
URL: https://github.com/apache/datafusion-comet/issues/3108
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `make_timestamp_ntz` function,
causing queries using this function to fall back to Spark's JVM execution
instead of running natively on DataFusion.
`MakeTimestampNTZ` creates a timestamp without timezone (NTZ) by combining a
date with a time value. This expression is a runtime replaceable binary
expression that delegates its implementation to the
`DateTimeUtils.makeTimestampNTZ` method.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
make_timestamp_ntz(date_expr, time_expr)
```
```scala
// DataFrame API
col("date_col").make_timestamp_ntz(col("time_col"))
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| left | DateType | The date component for the timestamp |
| right | AnyTimeType | The time component (can be time or timestamp type) |
**Return Type:** `TimestampNTZType` - A timestamp without timezone
information.
**Supported Data Types:**
- **Left operand**: `DateType` only
- **Right operand**: Any time-related type (`AnyTimeType` which includes
time and timestamp types)
**Edge Cases:**
- **Null handling**: If either input is null, the result will be null
- **Invalid dates**: Behavior depends on the underlying
`DateTimeUtils.makeTimestampNTZ` implementation
- **Time overflow**: Time values that exceed 24-hour boundaries may wrap or
cause errors depending on implementation
- **Timezone considerations**: Output explicitly excludes timezone
information (NTZ = No Time Zone)
**Examples:**
```sql
-- Create timestamp NTZ from date and time
SELECT make_timestamp_ntz(DATE '2023-12-25', TIME '14:30:00') as
christmas_afternoon;
-- Using with table columns
SELECT make_timestamp_ntz(date_col, time_col) as full_timestamp
FROM events_table;
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(expr("make_timestamp_ntz(event_date, event_time)"))
// With column references
df.withColumn("full_timestamp", expr("make_timestamp_ntz(date_col,
time_col)"))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.MakeTimestampNTZ`
**Related:**
- `MakeDate` - Creates date values from year, month, day components
- `MakeTimestamp` - Creates timestamp with timezone
- `DateTimeUtils` - Underlying utility class containing the implementation
- `TimestampNTZType` - The return data type for timezone-naive timestamps
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
