andygrove opened a new issue, #5586:
URL: https://github.com/apache/datafusion-comet/issues/5586
Part of #5572.
`named_struct` with duplicate field names falls the whole projection back to
Spark:
```scala
if (expr.names.length != expr.names.distinct.length) {
Unsupported(Some(duplicateNamesReason))
}
```
`spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43`
`docs/source/user-guide/latest/expressions.md` records it as "Duplicate
field names fall back".
Spark permits duplicate field names in a struct and
`CreateNamedStruct.doGenCode` builds the row positionally, so the dispatcher
reproduces it exactly. `StructType` is supported by
`CometBatchKernelCodegen.isSupportedDataType` recursively, and duplicate names
are not a problem on the Arrow side — the kernel's output field construction is
positional too. Worth confirming with a test that the resulting Arrow schema
round-trips, since duplicate field names in an Arrow struct are legal but
unusual.
Fix: mix `CodegenDispatchFallback` into `CometCreateNamedStruct`.
`CometCreateNamedStruct` also declines inside `convert` at `:66` when a
child fails to serialize, which the dispatcher cannot see; see the prerequisite
issue on `convert`-side declines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
