sunchao commented on code in PR #6170:
URL: https://github.com/apache/datafusion-comet/pull/6170#discussion_r4188935638
##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -1751,6 +1751,22 @@ class CometNativeCastSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
}
}
+ test("collation casts retain Spark fallback") {
+ assume(CometSparkSessionExtensions.isSpark40Plus)
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+ withParquetTable(Seq(("a", "A"), ("x ", "x")), "collation_cast") {
+ for (collation <- Seq("UTF8_LCASE", "UTF8_BINARY_RTRIM")) {
+ val a = s"CAST(_1 AS STRING COLLATE $collation)"
+ val b = s"CAST(_2 AS STRING COLLATE $collation)"
+ checkSparkAnswerAndFallbackReason(
+ s"SELECT array_contains(array($a), $b), " +
+ s"arrays_overlap(array($a), array($b)) FROM collation_cast",
+ "Cast from StringType")
Review Comment:
Fixed in 7d80ca40cf. The fallback assertion now uses
`CometCast.nonDefaultCollationReason`, matching the serializer’s explicit
collation guard. The focused Spark 4.1 collation-cast test and strict Spark 3.5
test compilation pass. Prior-head hosted Spark 4.0/4.1/4.2 expression logs
confirmed this stale assertion; broader CI is rerunning on the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
