andygrove opened a new pull request, #4514: URL: https://github.com/apache/datafusion-comet/pull/4514
## Which issue does this PR close? Closes #4513. ## Rationale for this change Comet's Arrow-direct codegen dispatcher routes eligible Spark `ScalaUDF` expressions through native execution, keeping the project, exchange, and sort operators around a UDF on the Comet path instead of forcing a fall back to Spark and a columnar-to-row roundtrip. Until now this was gated behind `spark.comet.exec.scalaUDF.codegen.enabled=false` and documented as experimental. The feature has broad type coverage (scalars, complex types with arbitrary nesting, higher-order functions) and is backed by end-to-end correctness, fuzz, and Iceberg test coverage. Enabling it by default lets users benefit without opt-in. ## What changes are included in this PR? - Flip the default of `spark.comet.exec.scalaUDF.codegen.enabled` to `true`. - Drop the "Experimental" prefix from the config documentation. - Update the Scala/Java UDF and Iceberg user guides to state the dispatcher is enabled by default and document how to disable it. Users can set `spark.comet.exec.scalaUDF.codegen.enabled=false` to restore the previous fall-back-to-Spark behavior. ## How are these changes tested? Covered by existing tests for the codegen dispatcher (`CometCodegenSuite`, `CometCodegenFuzzSuite`, `CometCodegenHOFSuite`, `CometTemporalExpressionSuite`, the datetime SQL file tests, and `CometIcebergRewriteActionSuite`), which exercise both the enabled and disabled paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
