andygrove opened a new pull request, #4728: URL: https://github.com/apache/datafusion-comet/pull/4728
## Summary - `CodegenDispatchFallback` already keeps `Incompatible` cases inside the Comet pipeline by routing them through the JVM codegen dispatcher (Spark's own `doGenCode` invoked inside the kernel). This PR extends the same routing to `Unsupported` cases on those serdes: when `getSupportLevel` returns `Unsupported` and the serde mixes in `CodegenDispatchFallback`, run Spark's generated code via the dispatcher before falling back to Spark. - Affects `Concat` (non-string children), `SortArray` (nested arrays with Struct or Null children), `ArrayIntersect` (collated strings), `TruncDate` and `TruncTimestamp` (formats outside the native set). All five conditions are things Spark itself supports, so the projection now stays in Comet's pipeline instead of bouncing out to Spark. - Refreshes the expression compatibility docs: drops the "faster native" wording (no measured speed claim), clarifies that the default path for `NativeOptInAvailable` serdes runs in the JVM via Spark codegen, and renders `Unsupported` reasons differently for `CodegenDispatchFallback` serdes (always handled via JVM dispatch) vs everything else (still falls back to Spark). ## Behavior change For the five `CodegenDispatchFallback` serdes listed above, inputs that previously caused the entire projection to fall back to Spark now stay inside Comet via dispatch. Dispatch already returns `None` cleanly when its global flag (`spark.comet.exec.scalaUDF.codegen.enabled`) is disabled or `CometBatchKernelCodegen.canHandle` refuses the tree, so the safety net to Spark fallback is preserved. No `[COMET-INFO]` plan hint is emitted in the `Unsupported` arm — unlike `Incompatible`, there is no native opt-in for the user to flip. ## Test plan - [ ] `./mvnw test -pl spark -Pspark-3.5 -Dsuites="org.apache.comet.CometExpressionSuite"` (concat, sort_array, array_intersect, trunc coverage) - [ ] Spot-check extended explain on a `Concat` with non-string children — confirm no `[COMET: …fall back…]` and the projection remains a `CometProject` - [ ] Regenerate docs and review `string.md`, `array.md`, `datetime.md` for the new "no native implementation, runs in the JVM" wording -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
