linzch3 commented on issue #27012:
URL: https://github.com/apache/datafusion/issues/27012#issuecomment-6631176066
also hits this from plain SQL, not just calling the UDF directly 👀
datafusion-python 55.0.0, pyarrow map with `value` non-nullable:
```python
import pyarrow as pa, datafusion
t = pa.map_(pa.field("key", pa.string(), False), pa.field("value",
pa.string(), False))
ctx = datafusion.SessionContext()
ctx.register_record_batches("m", [[pa.record_batch([pa.array([[("k", "v")]],
type=t)], names=["m"])]])
ctx.sql("select map_entries(m) from m").collect()
# PanicException: ... InvalidArgumentError("ListArray expected data type
Struct("key": non-null Utf8, "value": Utf8)
# got Struct("key": non-null Utf8, "value": non-null Utf8) for "item"")
```
looks like the value field's nullability is hardcoded to `true` in two
places in `datafusion/functions-nested/src/map_entries.rs` (main @ c120619):
- `return_type`, L86-90
- `map_entries_inner`, L117, then `ListArray::new` panics on the mismatch
(L113)
taking the nullability from the map's actual entry fields in both spots (or
just reusing the entries struct's `Fields`) should fix it, and
`ListArray::try_new` would at least turn this into an error instead of a panic
(｡•ᴗ•｡)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
