Ohad-Golan opened a new issue, #5512:
URL: https://github.com/apache/datafusion-comet/issues/5512
### Bug
`map_from_entries` crashes when the input array contains a null element
(e.g. `Seq((1, 10), null, (2, 20))`). Spark just returns `NULL` for that row —
Comet aborts the job instead.
Root cause, in `native/spark-expr/src/map_funcs/map_from_entries.rs`,
function `try_to_fix_struct_array`:
```rust
if key_array.null_count() > 0 {
return exec_err!(
"map_from_entries expects a list of structs with non-nullable keys,
but got nulls in the key array"
);
}
```
This checks "are there any nulls in the key column" — but doesn't check
whether those nulls belong to a null *entry* (the whole struct is absent) vs. a
real null *key* inside a valid entry. Arrow represents a null entry's key as
null too, so a harmless null array element gets misread as an illegal null key.
### Repro
```scala
Seq(Seq((1, 10), null, (2,
20))).toDF("a").selectExpr("map_from_entries(a)").show()
```
Spark: `NULL`. Comet: `CometNativeException: ... got nulls in the key array`.
### Impact
Null elements inside an array are common in real data (e.g. arrays built
from joins, optional JSON fields, or `collect_list` over nullable columns). Any
query calling `map_from_entries` on such an array crashes the whole Spark stage
instead of producing the correct `NULL` result for that row — this isn't a rare
edge case, it's a normal null-handling path that Spark itself supports without
error.
### Context
Found while re-auditing `DataFrameFunctionsSuite`'s currently-excluded tests
(#2480). Confirmed in isolation with a native Rust unit test — no Spark/JVM
involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
