Satyr09 opened a new pull request, #5764: URL: https://github.com/apache/datafusion-comet/pull/5764
## Which issue does this PR close? Closes #5579. ## Rationale for this change Queries such as `lpad(name, 10, pad_col)` currently fall the projection back to Spark because the native padding implementation cannot handle a column padding argument. Literal string inputs have the same restriction. Spark can generate code for both cases, so they can run through the existing codegen dispatcher inside Comet. ## What changes are included in this PR? Enroll `CometStringLPad` and `CometStringRPad` in `CodegenDispatchFallback`, preserving the existing native path for supported inputs. Update padding tests and add explicit coverage for dispatcher execution, native execution and fallback when codegen is disabled. Resolve Spark's padding expression builders in `GenerateDocs` so the generated implementation column identifies both functions as `Hybrid`. Add a benchmark for both dispatcher shapes and the native shape. ## How are these changes tested? The full `CometStringExpressionSuite` and the four padding SQL files pass on Spark 3.4.3, 3.5.9, 4.0.4 and 4.1.3 using the native release library. Spark 3.4 and 3.5 each pass 43 tests, with one existing Spark 4 collation test skipped; Spark 4.0 and 4.1 each pass 44 tests. A separate Spark 4.0 regression check passes all 14 targeted padding tests with the fix. Removing only the two dispatcher mixins makes exactly the two new dispatch tests fail because Spark retains a `Project` operator. Scalafix, Spotless, Scalastyle, license checks, benchmark/suite registration and documentation generation pass on Spark 3.5 and 4.0. Both formatting runs produce an empty patch. The release benchmark runs all six cases on 1,048,576 rows using Spark 4.0.4 and JDK 17. Best Spark/Comet times in milliseconds are 373/352 and 331/315 for column padding, 240/212 and 220/203 for literal string inputs, and 279/66 and 267/61 for the existing native shapes, listed as lpad then rpad. The harness reports no fallback or optimizer warnings. These are results from one CI runner. [Builds, tests and benchmark](https://github.com/Satyr09/datafusion-comet/actions/runs/34134288626), [formatting and generated documentation](https://github.com/Satyr09/datafusion-comet/actions/runs/34133797063), [regression proof](https://github.com/Satyr09/datafusion-comet/actions/runs/34131812395). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
