andygrove opened a new issue, #5580: URL: https://github.com/apache/datafusion-comet/issues/5580
Part of #5572. `MapKeySupport.keySupport` (`spark/src/main/scala/org/apache/comet/serde/maps.scala:68`) returns `Unsupported` for three classes of map key: - floating point anywhere in the key type, because Spark normalizes `-0.0` to `+0.0` and canonicalizes `NaN` in `ArrayBasedMapBuilder` while native `map_extract` compares raw Arrow values - a non-default collation, because native compares as `UTF8_BINARY` - any complex key type, because `map_extract`'s coercion casts the lookup key to the map's exact Arrow key type Two serdes consume it, and neither mixes in `CodegenDispatchFallback`: - `CometMapExtract` (`GetMapValue`, i.e. `map_col[key]`) — `serde/maps.scala:117`, gate at `:120` - `CometElementAt` on map input — `serde/arrays.scala:614` So `map_col[key]` and `element_at(map_col, key)` fail the whole projection back to Spark whenever the map has double, collated-string or struct keys. The analysis behind each decline is right — these really are cases the native lookup gets wrong — which is exactly why the dispatcher is the correct answer rather than a native fix. Spark's own `GetMapValue.doGenCode` and `ElementAt.doGenCode` give the normalization and interpreted-ordering equality for free, and `MapType` is fully supported by `CometBatchKernelCodegen.isSupportedDataType` (recursively, as long as the leaves are). Fix: mix `CodegenDispatchFallback` into both serdes. Note that `CometElementAt`'s other `Unsupported` arm, "Input must be an array or map", is unreachable — Spark's own type checking rejects that before Comet sees it — so it needs no handling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
