Kimahriman commented on issue #11923:
URL: https://github.com/apache/datafusion/issues/11923#issuecomment-2284199250
After digging a little more it seems like there's some other issues that
might make this significantly more difficult:
- When `invoke` is called, there's no way to know the logical nullability of
the ColumnarValue's/ArrayRef's being passed in. The only thing available is
`Array::is_nullable`, but it looks like all that does it check if the current
array has any nulls or not, so that could vary from batch to batch. So inside
`invoke` there's no way to know if field inside
```rust
Ok(Arc::new(GenericListArray::<O>::try_new(
Arc::new(Field::new("item", data_type, true)),
OffsetBuffer::new(offsets.into()),
arrow_array::make_array(data),
None,
)?))
```
should be logically nullable or not. And arrow will verify this when
creating a record batch, which is the error I ran into initially with the
nullability not matching:
https://github.com/apache/arrow-rs/blob/5c5a94a11f01a286dd03b18af0f11c327a9accc6/arrow-array/src/record_batch.rs#L205
- ScalarUDF's in general don't have a concept of nullability, they are
always considered nullable. Only tangentially related but just a larger issue
of nullability and ScalarUDFs:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/scalar_function.rs#L120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
