andygrove commented on code in PR #6480:
URL: https://github.com/apache/datafusion-comet/pull/6480#discussion_r4149881758
##########
spark/src/main/scala/org/apache/comet/serde/strings.scala:
##########
@@ -218,7 +218,78 @@ object CometStringReplace
object CometSubstring extends CometScalarFunction[Substring]("substring")
-object CometSubstringIndex extends CometExpressionSerde[SubstringIndex] {
+/**
+ * Support level for native string kernels that search, trim or compare
strings by raw bytes
+ * (`instr`, `substring_index`, `trim` with a trim string, `greatest`,
`least`). They ignore a
+ * non-UTF8_BINARY collation, for example UTF8_LCASE where 'a' equals 'A', and
give wrong answers.
+ * Reporting Incompatible routes the expression through the JVM codegen
dispatcher, which runs
+ * Spark's collation-aware implementation.
+ */
+private[serde] object StringCollationSupport extends CometTypeShim {
+ def collationReason(name: String): String =
+ "Spark evaluates non-UTF8_BINARY collated string input under its
collation, while " +
+ s"Comet's native $name compares raw bytes"
+
+ def getSupportLevel(name: String, operands: Seq[Expression]): SupportLevel =
+ if (operands.exists(op => hasNonDefaultStringCollation(op.dataType))) {
+ Incompatible(Some(collationReason(name)))
+ } else {
+ Compatible()
+ }
+}
+
+object CometStringInstr
+ extends CometScalarFunction[StringInstr]("instr")
+ with CodegenDispatchFallback {
Review Comment:
Could you update the seven `expressions.md` implementation entries from
`Native` to `Hybrid`? Adding `CodegenDispatchFallback` changes the documented
execution model for `instr`, `substring_index`, `trim`, `ltrim`, `rtrim`,
`greatest`, and `least`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
