andygrove opened a new issue, #3107:
URL: https://github.com/apache/datafusion-comet/issues/3107
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `make_timestamp_from_date_time`
function, causing queries using this function to fall back to Spark's JVM
execution instead of running natively on DataFusion.
The `MakeTimestampFromDateTime` expression combines a date value with an
optional time value to create a timestamp. It serves as a runtime-replaceable
expression that delegates to the `DateTimeUtils.makeTimestamp` method for the
actual timestamp construction.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
make_timestamp(date_expr [, time_expr] [, timezone_expr])
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| date | DateType | The date component for the timestamp |
| time | AnyTimeType (optional) | The time component; defaults to midnight
(00:00:00) if not provided |
| timezone | StringType (optional) | The timezone identifier; defaults to
session timezone if not provided |
**Return Type:** TimestampType - A timestamp value combining the date and
time components in the specified timezone.
**Supported Data Types:**
- **Date argument**: DateType only
- **Time argument**: AnyTimeType (when provided)
- **Timezone argument**: StringType with collation support (when provided)
**Edge Cases:**
- **Null handling**: Returns null if any non-optional argument is null,
based on child expression nullability
- **Missing time**: Automatically defaults to `Literal(0L, TimeType(0))`
representing midnight
- **Missing timezone**: Falls back to the configured session timezone via
`timeZoneId.get`
- **Invalid combinations**: Relies on underlying
`DateTimeUtils.makeTimestamp` for validation of date/time combinations
**Examples:**
```sql
-- Create timestamp from date only (uses midnight and session timezone)
SELECT make_timestamp(DATE '2023-12-25');
-- Create timestamp with specific time
SELECT make_timestamp(DATE '2023-12-25', TIME '14:30:15');
-- Create timestamp with specific timezone
SELECT make_timestamp(DATE '2023-12-25', TIME '14:30:15',
'America/New_York');
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
// With date only
df.withColumn("timestamp_col", expr("make_timestamp(date_col)"))
// With date and time
df.withColumn("timestamp_col", expr("make_timestamp(date_col, time_col)"))
// With all parameters
df.withColumn("timestamp_col", expr("make_timestamp(date_col, time_col,
'UTC')"))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.MakeTimestampFromDateTime`
**Related:**
- `MakeTimestamp` - Creates timestamps from individual numeric components
- `DateTimeUtils.makeTimestamp` - The underlying utility method that
performs the actual conversion
- `TimeZoneAwareExpression` - Base trait for timezone-sensitive expressions
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
