Smallfu666 opened a new issue, #6584:
URL: https://github.com/apache/datafusion-comet/issues/6584
### Describe the bug
`CometLiteral.hasDuplicateMapKeys` keeps a folded map that holds a null key
on Spark, but it returns
early when the map has fewer than two keys, before it checks for null:
```scala
private def hasDuplicateMapKeys(keys: ArrayData, keyType: DataType): Boolean
= {
if (keys.numElements() < 2) return false
val sorted = keys.toObjectArray(keyType)
if (sorted.contains(null)) return true
...
```
So a folded map with one entry and a null key is expanded into
`CreateMap(Literal(null), v)`. That
`CreateMap` runs through the codegen dispatcher, where Spark's
`ArrayBasedMapBuilder.put` rejects the
null key. Spark itself never rebuilds the folded literal, so it returns a
value.
A TRY cast over a foldable map produces such a literal when the key cast
fails.
### Steps to reproduce
```sql
CREATE TABLE t(id int) USING parquet;
INSERT INTO t VALUES (1), (2);
SELECT id, element_at(try_cast(map(9999999999L, 20) AS map<int, int>), id)
FROM t ORDER BY id;
```
The optimized plan holds `element_at(map(keys: [null], values: [20]), id)`.
- Spark: `[1,null], [2,null]`
- Comet: `[NULL_MAP_KEY] Cannot use null as map key. SQLSTATE: 2200E`,
thrown from
`ArrayBasedMapBuilder.put` inside the dispatched kernel.
The same map with two entries, `map(1L, 10, 9999999999L, 20)`, folds to
`map(keys: [1,null], ...)`.
It reaches the null check, stays on Spark, and returns `[1,10], [2,null]` as
Spark does.
This needs `ConstantFolding`, so it does not reproduce in the SQL file
tests, which exclude it.
Reproduced on Spark 4.1 at `main` 8c93fbb73.
### Expected behavior
Same result as Spark. Checking for a null key before the size check keeps
the projection on Spark:
```scala
val sorted = keys.toObjectArray(keyType)
if (sorted.contains(null)) return true
if (sorted.length < 2) return false
```
### Additional context
Found while reviewing #6178. #5854 is related but a different path: it
enforces null-key rejection in
native map construction, not in the expansion of folded literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
