andygrove opened a new issue, #5578:
URL: https://github.com/apache/datafusion-comet/issues/5578
Part of #5572.
`to_csv` never runs inside Comet by default. Every path through
`CometStructsToCsv.getSupportLevel` returns `Unsupported` or `Incompatible`:
```scala
if (containsComplexType) {
return Unsupported(...)
}
if (containsIncompatibleDataTypes) {
return Incompatible(...)
}
// https://github.com/apache/datafusion-comet/issues/3232
Incompatible()
```
`spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288`
The serde has no `CodegenDispatchFallback` mixin, so with the default
configuration the enclosing operator falls back to Spark for every `to_csv`
call. The native path is reachable only with
`spark.comet.expression.StructsToCsv.allowIncompatible=true`.
The rest of the structured-text family does not behave this way. `to_json`
is Hybrid, and `from_csv` and `schema_of_csv` are both plain
`CometCodegenDispatch` (`serde/csv.scala:24`, `:26`). `to_csv` is the odd one
out.
There is also a docs bug here:
`docs/source/user-guide/latest/expressions.md` lists `to_csv` as "✅ Native"
with no note, which reads as "runs natively by default". It does not. Adding
the mixin fixes the docs too, since `GenerateDocs` derives the Implementation
column from the trait mixins and would then report it as Hybrid.
Fix: mix `CodegenDispatchFallback` into `CometStructsToCsv`. The
complex-type arm dispatches fine — arrays, structs and maps are all in
`CometBatchKernelCodegen.isSupportedDataType` as long as their leaves are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
