viirya opened a new issue, #6843:
URL: https://github.com/apache/datafusion-comet/issues/6843
### Describe the bug
Spark `DateType` covers any epoch day that fits in `Int` (years from about
-5877641 to 5881580). Several native Comet date kernels convert the value
through chrono's `NaiveDate`, which only covers years -262143 to 262142. For
dates outside that range, these kernels return `NULL` where Spark returns a
value, or fail the query.
Such dates reach native execution from a Parquet `date` column and from
native `make_date` (since #5443).
### Steps to reproduce
```sql
SET spark.sql.session.timeZone=UTC;
CREATE TABLE t(d date) USING parquet;
INSERT INTO t SELECT date_from_unix_date(x)
FROM VALUES (95026237), (-96465293), (108853388), (18262) AS v(x);
-- 262143-01-01, -262144-12-31, 300000-06-15, 2020-01-01
SELECT unix_date(d), year(d), month(d), dayofmonth(d) FROM t;
SELECT unix_date(d), cast(d AS string) FROM t;
SELECT unix_date(d), last_day(d) FROM t;
```
The `2020-01-01` row matches Spark in every case. For the other three rows:
| Expression | Spark | Comet |
|---|---|---|
| `year`, `month`, `dayofmonth`, `dayofyear`, `quarter`, `weekofyear` | the
field value (e.g. `year` = 262143, -262144, 300000) | `NULL` |
| `trunc(d, 'MM')`, `next_day(d, 'Mon')` | a date | `NULL` |
| `cast(d AS string)` | e.g. `+262143-01-01` | query fails: `Cast error:
Failed to convert 95026237 to temporal for Date32` |
| `last_day(d)` | a date | query fails: `Spark last_day: Unable to convert
days value 108853388 to date` |
`dayofweek`, `add_months`, `date_add`, `unix_date` and date comparisons
already match Spark for these values.
### Expected behavior
These expressions should return the same values as Spark for every date that
Spark `DateType` can hold, or fall back to Spark when the native kernel cannot
represent the value.
### Additional context
Casting such dates to `TIMESTAMP` panics, which is tracked separately in
#5456. `date_format` is not listed because Spark itself throws
`ArithmeticException: long overflow` for year 300000.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
