andygrove opened a new issue, #5583:
URL: https://github.com/apache/datafusion-comet/issues/5583
Part of #5572.
`CometArraysZip` has its own type gate, separate from `ArraysBase`, and
declines anything outside it:
```scala
if (!isTypeSupported(dt)) {
return Unsupported(Some(s"Unsupported child data type: $dt"))
}
```
`spark/src/main/scala/org/apache/comet/serde/arrays.scala:837-839`, with the
predicate at `:822`
That predicate accepts the scalar types plus nested arrays and structs, but
not `MapType`, so `arrays_zip` over an array of maps fails the whole projection
back to Spark. The serde has no `CodegenDispatchFallback` mixin.
`ArraysZip` is a plain codegen expression, and `MapType` is supported by
`CometBatchKernelCodegen.isSupportedDataType` recursively, so the dispatcher
covers the gap.
Fix: mix `CodegenDispatchFallback` into `CometArraysZip`.
Same carve-out as the `ArraysBase` issue — `NullType` elements are not
dispatchable, since `NullType` is absent from `isSupportedDataType`.
`CometArraysZip` also declines inside `convert` at `:880` when a child fails
to serialize, which the dispatcher cannot see; see the prerequisite issue on
`convert`-side declines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
