andygrove opened a new issue, #3118:
URL: https://github.com/apache/datafusion-comet/issues/3118
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark
`try_make_timestamp_from_date_time` function, causing queries using this
function to fall back to Spark's JVM execution instead of running natively on
DataFusion.
`TryMakeTimestampFromDateTime` is a runtime-replaceable Catalyst expression
that attempts to construct a timestamp from date and optional time components.
Unlike its non-try counterpart, this expression returns null instead of
throwing an exception when the input values cannot be converted to a valid
timestamp.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
try_make_timestamp(date [, time] [, timezone])
```
```scala
// DataFrame API usage would depend on the function registry
col("date_col").expr("try_make_timestamp(date_col)")
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| date | Expression | The date component for the timestamp |
| time | Expression (optional) | The time component to combine with the date
|
| timezone | Expression (optional) | The timezone for the resulting
timestamp |
| replacement | Expression | Internal replacement expression used during
runtime |
**Return Type:** Returns `TimestampType` on successful conversion, or `null`
if the conversion fails.
**Supported Data Types:**
- **date**: Date types, string representations of dates
- **time**: Time types, string representations of time
- **timezone**: String representations of valid timezone identifiers
**Edge Cases:**
- **Null date input**: Returns null without attempting conversion
- **Invalid date formats**: Returns null instead of throwing parse
exceptions
- **Invalid timezone**: Returns null for unrecognized timezone identifiers
- **Leap second handling**: Behavior depends on the underlying datetime
implementation
- **Daylight saving transitions**: May return null for ambiguous times
during DST changes
**Examples:**
```sql
-- Basic usage with date only
SELECT try_make_timestamp('2023-12-25') as christmas_timestamp;
-- With date and time components
SELECT try_make_timestamp('2023-12-25', '15:30:00') as afternoon_timestamp;
-- With timezone specification
SELECT try_make_timestamp('2023-12-25', '15:30:00', 'UTC') as utc_timestamp;
-- Handling invalid input (returns null)
SELECT try_make_timestamp('invalid-date') as null_result;
```
```scala
// DataFrame API usage (conceptual)
import org.apache.spark.sql.functions._
df.select(expr("try_make_timestamp(date_col)").as("timestamp_col"))
df.select(expr("try_make_timestamp(date_col, time_col,
'PST')").as("pst_timestamp"))
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.TryMakeTimestampFromDateTime`
**Related:**
- `MakeTimestampFromDateTime` - The non-try version that throws exceptions
on invalid input
- `MakeTimestamp` - Creates timestamps from individual
year/month/day/hour/minute/second components
- `ToTimestamp` - Converts string representations to timestamps with format
patterns
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
