mbutrovich opened a new issue, #26685:
URL: https://github.com/apache/datafusion/issues/26685
### Describe the bug
`date_part` fails when a non-null input gives a NULL result:
```sql
SELECT extract(second from ts)
FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)')))
AS t(ts);
-- Arrow error: Invalid argument error: Column
'date_part(Utf8("SECOND"),t.ts)' is declared as non-nullable but contains null
values
SELECT extract(hour from d)
FROM (VALUES (arrow_cast(9223372036854775807, 'Duration(Second)'))) AS t(d);
-- Arrow error: Invalid argument error: Column 'date_part(Utf8("HOUR"),t.d)'
is declared as non-nullable but contains null values
```
### To Reproduce
Run the queries above on `main` (`dad9063be`).
### Expected behavior
Both return NULL.
### Additional context
[`return_field_from_args`](https://github.com/apache/datafusion/blob/702803d0220cf444da5cf8c8d4ada8393ec89d54/datafusion/functions/src/datetime/date_part.rs#L184-L209)
takes the result's nullability from the input. Arrow's `date_part` returns
NULL for some non-null inputs, such as timestamps it can't convert and values
that overflow `Int32` ([`Duration(s)`
hour](https://github.com/apache/arrow-rs/blob/ef1fa157977633f0ba21aa921f9ef3d5c669235b/arrow-arith/src/temporal.rs#L599)).
The result was always nullable until #19823 (53.0.0). That PR added the
Spark `date_part`, which simplifies to this function, and changed the
nullability so the two fields would match ([review
thread](https://github.com/apache/datafusion/pull/19823#discussion_r2694220228)).
The Spark wrapper still [takes its nullability from the
inputs](https://github.com/apache/datafusion/blob/702803d0220cf444da5cf8c8d4ada8393ec89d54/datafusion/spark/src/function/datetime/date_part.rs#L84-L88),
so a fix here should change both to keep them matching. The optimizer's schema
check
([`logically_equivalent_names_and_types`](https://github.com/apache/datafusion/blob/702803d0220cf444da5cf8c8d4ada8393ec89d54/datafusion/common/src/dfschema.rs#L610-L621))
compares names and types but not nullability, so a mismatch between them would
go unnoticed.
Part of #26672.
### Willingness to contribute
I plan to contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
