andygrove opened a new issue, #3132:
URL: https://github.com/apache/datafusion-comet/issues/3132
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `convert_timezone` function,
causing queries using this function to fall back to Spark's JVM execution
instead of running natively on DataFusion.
The `ConvertTimezone` expression converts a timestamp from one timezone to
another timezone. It takes a timestamp without timezone (TimestampNTZ) and
interprets it in the source timezone, then converts it to the equivalent
timestamp in the target timezone.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
convert_timezone(source_tz, target_tz, source_timestamp)
convert_timezone(target_tz, source_timestamp) -- uses current session
timezone as source
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| source_tz | String | Source timezone identifier (optional, defaults to
current session timezone) |
| target_tz | String | Target timezone identifier |
| source_timestamp | TimestampNTZ | Timestamp without timezone to convert |
**Return Type:** Returns `TimestampNTZType` - a timestamp without timezone
information representing the equivalent time in the target timezone.
**Supported Data Types:**
- Source and target timezone parameters must be string types with collation
support
- Source timestamp must be TimestampNTZ type
- Supports string trimming collations for timezone parameters
**Edge Cases:**
- Null handling: Returns null if any input parameter is null (null
intolerant behavior)
- Invalid timezone identifiers will cause runtime errors
- Ambiguous times during DST transitions follow Java timezone handling rules
- Non-existent times during DST transitions are handled according to the
target timezone's rules
**Examples:**
```sql
-- Convert from UTC to Pacific Time
SELECT convert_timezone('UTC', 'America/Los_Angeles',
timestamp_ntz'2021-12-06 08:00:00');
-- Result: 2021-12-06 00:00:00
-- Convert using current session timezone as source
SELECT convert_timezone('Europe/London', timestamp_ntz'2021-12-06 12:00:00');
```
```scala
// Example DataFrame API usage
import org.apache.spark.sql.functions._
df.select(
expr("convert_timezone('UTC', 'Asia/Tokyo',
timestamp_col)").as("tokyo_time")
)
// Using two-parameter version
df.select(
expr("convert_timezone('Europe/Berlin', timestamp_col)").as("berlin_time")
)
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Large
**Spark Expression Class:**
`org.apache.spark.sql.catalyst.expressions.ConvertTimezone`
**Related:**
- `CurrentTimeZone` - gets the current session timezone
- `ToUTCTimestamp` - converts to UTC timezone
- `FromUTCTimestamp` - converts from UTC timezone
- Other datetime functions in the `datetime_funcs` group
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
