neilconway opened a new issue, #24983:
URL: https://github.com/apache/datafusion/issues/24983
### Describe the bug
`map_extract` can fail to find a non-null struct key when a nullable
dictionary-encoded field uses a different representation of null in the query
key.
Both keys have type `Struct<d: Dictionary<Int8, Utf8>>` and display as `{d:
NULL}`:
- In the stored key, a valid dictionary index points to a null dictionary
entry.
- In the query key, the dictionary index itself is null.
DataFusion reports that the structs are equal using both `=` and `IS NOT
DISTINCT FROM`. It also reports that both `d` fields are null. Nevertheless,
lookup in a single-entry map returns `\
[42]` for the stored key and `[NULL]` for the equivalent query key.
### To Reproduce
```sql
CREATE TABLE dictionary_structs AS
SELECT arrow_cast(
named_struct('d', CAST(NULL AS VARCHAR)),
'Dictionary(Int8, Struct("d": Utf8))'
) AS s;
WITH keys AS (
SELECT named_struct('d', s['d']) AS stored_key,
named_struct('d', arrow_cast(NULL, 'Dictionary(Int8, Utf8)')) AS
query_key
FROM dictionary_structs
)
SELECT stored_key = query_key AS keys_equal,
stored_key IS NOT DISTINCT FROM query_key AS keys_not_distinct,
map_extract(MAP {stored_key: 42}, stored_key) AS self_lookup,
map_extract(MAP {stored_key: 42}, query_key) AS equivalent_key_lookup
FROM keys;
```
Actual result:
| keys_equal | keys_not_distinct | self_lookup | equivalent_key_lookup |
|---|---|---|---|
| `true` | `true` | `[42]` | `[NULL]` |
### Expected behavior
`equivalent_key_lookup` should return `[42]`, matching `self_lookup`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
