manuzhang opened a new pull request, #5267: URL: https://github.com/apache/datafusion-comet/pull/5267
## Which issue does this PR close? Relates to https://github.com/apache/datafusion-comet/issues/4180. ## Rationale for this change `spark.sql.function.concatBinaryAsString=true` makes Spark insert `BinaryType`-to-`StringType` casts before `concat`. Spark preserves malformed bytes in those casts, while Comet's native Arrow string path normalizes them to UTF-8 replacement bytes. For example, Spark keeps `FFFE41`, but Comet produced `EFBFBDEFBFBD41`. ## What changes are included in this PR? - Route `concat` trees containing `BinaryType`-to-`StringType` casts through Spark's generated code inside the Comet pipeline. - Add a two-value SQL config matrix covering result bytes, malformed UTF-8, empty values, and nulls. - Add a Scala assertion for result types and direct proof that both config values use the JVM codegen dispatcher without operator fallback. - Record the supported behavior in the Spark configuration contributor guide. ## How are these changes tested? - `make core` - Focused `CometSqlFileTestSuite concat_binary_as_string` on Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2 - Focused `CometCodegenSuite concatBinaryAsString` on the same versions - `./mvnw spotless:check -Pspark-4.1 -Pjdk17 -DskipTests -Dscalastyle.skip=true` - Dynamic documentation generation for all four Spark profiles -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
