andygrove opened a new issue, #3124:
URL: https://github.com/apache/datafusion-comet/issues/3124
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification
details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `days` function, causing queries
using this function to fall back to Spark's JVM execution instead of running
natively on DataFusion.
The `Days` expression is a v2 partition transform that converts timestamp
values to the number of days since a reference epoch. This transform is used
for partitioning data by day buckets, allowing efficient querying of
time-series data partitioned at the daily level.
Supporting this expression would allow more Spark workloads to benefit from
Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
-- SQL syntax (when used in partition transforms)
PARTITIONED BY (days(timestamp_column))
```
```scala
// DataFrame API usage
import org.apache.spark.sql.catalyst.expressions.Days
Days(child = timestampColumn)
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| child | Expression | The input expression that should evaluate to a
timestamp or date value |
**Return Type:** `IntegerType` - Returns an integer representing the number
of days since the epoch.
**Supported Data Types:**
- TimestampType
- DateType
- Any expression that can be implicitly cast to timestamp or date
**Edge Cases:**
- Null input values result in null output (standard null propagation)
- Timestamps before Unix epoch (1970-01-01) result in negative day numbers
- Leap years and daylight saving time transitions are handled according to
the configured timezone
- Date boundary calculations respect the session timezone configuration
**Examples:**
```sql
-- Creating a table partitioned by days
CREATE TABLE events (
id BIGINT,
event_time TIMESTAMP,
data STRING
) PARTITIONED BY (days(event_time))
```
```scala
// DataFrame API usage in partition transforms
import org.apache.spark.sql.catalyst.expressions.Days
// Used internally when defining partition transforms
val dayTransform = Days(col("event_timestamp").expr)
```
### Implementation Approach
See the [Comet guide on adding new
expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html)
for detailed instructions.
1. **Scala Serde**: Add expression handler in
`spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if
needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has
built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.Days`
**Related:**
- `Hours` - Partition transform for hourly buckets
- `Months` - Partition transform for monthly buckets
- `Years` - Partition transform for yearly buckets
- `Bucket` - Hash-based partition transform
- `PartitionTransformExpression` - Base class for partition transforms
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
