Ishaan400 opened a new issue, #26197:
URL: https://github.com/apache/datafusion/issues/26197
### Describe the bug
`map[key]` (`get_field` on a map) does not coerce the lookup key to the
map's key type. Any map whose key type differs from the literal's type fails at
execution time.
The most common case is a map declared in SQL with `VARCHAR` keys. Since
`VARCHAR` maps to `Utf8View` by default, the map has `Utf8View` keys, while
`'a'` is a `Utf8` literal. Integer keys hit the same error: `MAP(INT, ...)`
with `m[1]`, where the literal is `Int64`.
`map_extract(m, 'a')` and `element_at(m, 'a')` work on the same column,
because `map_extract` coerces the key to the map key type in `coerce_types`.
### To Reproduce
```sql
CREATE TABLE t (m MAP(VARCHAR, BIGINT)) AS VALUES (MAP {'a': 1, 'b': 2});
SELECT m['a'] FROM t;
-- Execution error: The key type Utf8 does not match the map key type
Utf8View
SELECT map_extract(m, 'a') FROM t; -- [1]
SELECT element_at(m, 'a') FROM t; -- [1]
SELECT m[arrow_cast('a', 'Utf8View')] FROM t; -- 1
CREATE TABLE w (m MAP(INT, BIGINT)) AS VALUES (MAP {1: 10});
SELECT m[1] FROM w;
-- Execution error: The key type Int64 does not match the map key type Int32
```
Nested maps fail the same way (`b['a']['aa']` on `MAP(VARCHAR, MAP(VARCHAR,
INT))`), as do filters such as `WHERE m['a'] = 1`.
With `SET datafusion.sql_parser.map_string_types_to_utf8view = false`, the
`VARCHAR` case returns `1`.
### Expected behavior
`m['a']` returns `1` and `m[1]` returns `10`. The lookup key should be
coerced to the map key type, the same way `map_extract` does it.
### Additional context
- Tested on `main` at a4b8198.
- Not a regression from #25201. The commit before it (28c0dec) fails on the
same queries, with `Arrow error: Invalid argument error: Invalid comparison
operation: Utf8 == Utf8View` (and `Int64 == Int32`). #25201 only changed the
error message.
- `GetFieldFunc::coerce_types` returns the argument types unchanged
("validation happens in return_field_from_args"). `return_field_from_args` does
not check the map key type either. The key reaches `map_lookup`
(`datafusion/functions/src/utils.rs`), which requires the lookup key type to
equal the map key type, ignoring dictionary encoding.
- `MAP(BIGINT, ...)` with `m[1]` and `MAP(DOUBLE, ...)` with `m[1.5]` work,
because those literals already have the map's key type.
- The existing map tests (`map.slt`, and the `map(int, int)` / `map(varchar,
...)` table in `ddl.slt`) never look up a key on a map column declared in SQL.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
