viirya commented on code in PR #6170:
URL: https://github.com/apache/datafusion-comet/pull/6170#discussion_r4570023899
##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -1751,6 +1751,22 @@ class CometNativeCastSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
}
}
+ test("collation casts retain Spark fallback") {
+ assume(CometSparkSessionExtensions.isSpark40Plus)
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+ withParquetTable(Seq(("a", "A"), ("x ", "x")), "collation_cast") {
+ for (collation <- Seq("UTF8_LCASE", "UTF8_BINARY_RTRIM")) {
+ val a = s"CAST(_1 AS STRING COLLATE $collation)"
+ val b = s"CAST(_2 AS STRING COLLATE $collation)"
+ checkSparkAnswerAndFallbackReason(
+ s"SELECT array_contains(array($a), $b), " +
+ s"arrays_overlap(array($a), array($b)) FROM collation_cast",
+ CometCast.nonDefaultCollationReason)
Review Comment:
After #6471, `array_contains` and `arrays_overlap` report `Incompatible` for
collated elements and fall back before their children are converted, so this
reason is never recorded. Run as written against current main, the test fails
with only the `array_contains` and `arrays_overlap` reasons.
`array_element_equality_collation_disabled.sql` from #6471 and
`CometCastCollatedStringSuite` from #5302 already cover both sides. Could this
test be dropped, or changed to a query where the cast is the only reason to
fall back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
