andygrove opened a new issue, #6333:
URL: https://github.com/apache/datafusion-comet/issues/6333
### Describe the bug
Spark's `Days` and `Hours` are partition transforms. They are `Unevaluable`,
and evaluating one in a projection throws
`PARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY`. Comet evaluates both
natively, but not the same way:
- `CometDays`
(`spark/src/main/scala/org/apache/comet/serde/datetime.scala:826`) casts the
timestamp to a date in the session timezone.
- `CometHours` divides the raw UTC micros.
Iceberg's own `days` and `hours` transforms are UTC-based, and so are
Comet's native versions of them in
`native/spark-expr/src/iceberg_funcs/temporal.rs`. In a non-UTC session,
Comet's `days` gives a different day than Iceberg would for the same timestamp.
### Steps to reproduce
On `main` at `764936187`:
```scala
import org.apache.spark.sql.functions.{col, days, hours}
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
spark.sql("SELECT TIMESTAMP'2024-07-01T03:00:00Z' AS ts").write.parquet(path)
spark.read.parquet(path).select(days(col("ts")), hours(col("ts"))).collect()
```
Spark throws `PARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY`. Comet
returns `19904, 477723`. Day 19904 is 2024-06-30, the Los Angeles date, but the
hour is counted in UTC, and the UTC day would be 19905.
### Expected behavior
There is no Spark answer to match. The two transforms should at least agree
with each other and with Iceberg, which means `days` should use UTC.
Alternatively, Comet could decline both, since Spark never evaluates them.
### Additional context
Low priority. It only matters if something evaluates the transforms, and
Spark itself won't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
