andygrove opened a new issue, #5589:
URL: https://github.com/apache/datafusion-comet/issues/5589
Part of #5572.
`map_from_arrays` reports `Incompatible` whenever
`spark.sql.mapKeyDedupPolicy` is `LAST_WIN`:
```scala
if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
}
```
`spark/src/main/scala/org/apache/comet/serde/maps.scala:163-164`
With no `CodegenDispatchFallback` mixin, that means the whole projection
falls back to Spark for any session running that policy.
`CometMapFromEntries` sits directly below it in the same file and already
carries the mixin (`:211-213`) for its own analogous restriction, so the two
closely related functions behave differently for no good reason.
`MapFromArrays.doGenCode` builds the map through `ArrayBasedMapBuilder`,
which is where the dedup policy is actually implemented, so the dispatcher gets
`LAST_WIN` right by construction. `MapType` is supported by
`CometBatchKernelCodegen.isSupportedDataType` recursively.
Fix: mix `CodegenDispatchFallback` into `CometMapFromArrays`.
Not in scope: the separate null-key divergence documented in
`getCompatibleNotes` and tracked as #4680. That one is a `Compatible` path
today, so it is a correctness question rather than a fallback question — though
it is worth noting that routing through the dispatcher would fix it too, since
Spark's builder raises "Cannot use null as map key" itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
