andygrove opened a new issue, #5590:
URL: https://github.com/apache/datafusion-comet/issues/5590
Part of #5572.
`map_sort` falls the whole projection back to Spark for any non-scalar map
key:
```scala
if (!supportedScalarSortElementType(keyType)) {
Unsupported(Some(s"MapSort on map with key type $keyType is not
supported"))
}
```
`spark/src/main/spark-4.x/org/apache/comet/serde/CometMapSort.scala:40-41`
`supportedScalarSortElementType` (`serde/QueryPlanSerde.scala:1238`) admits
the scalar types only, and excludes collated strings, so `map_sort` on a map
keyed by a struct, array or collated string fails the operator.
`MapSort.doGenCode` sorts with Spark's interpreted ordering, which handles
all of those, and `MapType` is supported recursively by
`CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher covers the gap.
Fix: mix `CodegenDispatchFallback` into `CometMapSort`.
The serde's other arm — `Incompatible` under
`spark.comet.exec.strictFloatingPoint=true` for a floating-point key (`:42-46`)
— would be routed by the same mixin, and that is the right outcome: strict mode
is asking for Spark-exact behavior, which is precisely what the dispatcher
gives.
This is a Spark 4.x-only serde, so the test belongs with the other 4.x-gated
suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
