This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 92f0f33e75 feat: fall back to Spark for collated predicate operands
(#4948)
92f0f33e75 is described below
commit 92f0f33e752243c1fe5ef72dd05858d224b7984c
Author: Oleks V <[email protected]>
AuthorDate: Fri Jul 17 14:39:47 2026 -0700
feat: fall back to Spark for collated predicate operands (#4948)
* feat: fall back to Spark for collated predicate operands
Comet's native equality, ordering, and hashing compare raw UTF-8 bytes,
so a predicate operand carrying a non-UTF8_BINARY collation (Spark 4+)
would produce wrong answers on the native path -- for example,
`'a' = 'A'` under UNICODE_CI returns true in Spark but false byte-wise.
Add a `ComparisonUtils.collationSupportLevel` shared helper and wire it
into `getSupportLevel` for the six binary comparison serdes plus
`In`/`InSet`. `CometNot`'s special-case rewrites for
`Not(EqualTo|EqualNullSafe|In)` now guard on the same helper so they
fall through to the generic path (and thus to Spark) when any operand
is collated. The helper walks nested types via
`hasNonDefaultStringCollation`, so collated strings inside
array/map/struct operands are caught too.
* feat: dispatch collated str_to_map through JVM codegen
Mix CodegenDispatchFallback into CometStrToMap so that when the
input string or a delimiter carries a non-UTF8_BINARY collation
the expression is executed by Spark's own doGenCode inside the
Comet kernel rather than falling the whole projection back to
Spark. Update the str_to_map_collation.sql fixture to drop the
expect_fallback(...) assertion accordingly.
* test: dispatch legacy-truncate str_to_map through codegen
Update str_to_map_legacy_truncate.sql to drop the expect_fallback
assertions -- with CodegenDispatchFallback on CometStrToMap the
expression now runs via Spark's own doGenCode inside the Comet
kernel instead of falling the projection back to Spark.
* feat: fall back to Spark for collated Like/Contains/StartsWith/EndsWith
Extends 903a772b6 to the string-predicate serdes flagged in review.
CometLike, CometContains, CometStartsWith, and CometEndsWith now route
their `getSupportLevel` through `ComparisonUtils.collationSupportLevel`
so any non-UTF8_BINARY operand triggers a clean fallback. Previously
Contains / StartsWith / EndsWith reached the native scalar-function
bridge via `CometScalarFunction` with no collation guard, and Like's
`getSupportLevel` only rejected custom escape characters. Native
kernels compare raw UTF-8 bytes, so a case- or accent-insensitive
collation would yield wrong answers (e.g. `'Hello' LIKE '%hello%'`
returns true in Spark under UNICODE_CI but false byte-wise).
`collationSupportLevel` grows an optional `exprName` so each predicate
surfaces a fallback reason that names itself in EXPLAIN.
Expands `sql-tests/.../collation.sql` with predicate coverage: the six
binary comparisons under UTF8_LCASE and UNICODE_CI, `IN`/`NOT IN`
(scalar and `InSet`-threshold), CometNot's `NOT (=)` / `NOT (<=>)`
rewrites, `LIKE`, `Contains`, `StartsWith`, `EndsWith`, collated
strings buried in `array`/`map`/`struct` operands (exercising
`hasNonDefaultStringCollation`'s recursion), and NULL propagation
through each of the above.
* test: expect fallback for collated-predicate SQL tests
Every predicate query in `collation.sql` now uses
`query expect_fallback(non-UTF8_BINARY collated operands)` instead of
plain `query`. The predicate serdes correctly return `Unsupported` for
collated operands, so the enclosing Project falls back to Spark --
which is the whole point of the fix -- but that violates the default
`CheckCoverageAndAnswer` assertion mode's "only Comet native operators"
check. `expect_fallback` still verifies the answer against Spark and
additionally pins the shared fallback-reason substring, so a regressed
guard shows up as either a wrong answer or a missing fallback.
Also drops the map test: `CometCreateMap` routes through the JVM
codegen dispatcher, so its interaction with the outer EqualTo's
collation guard is not the clean path this file is meant to pin down.
* test: drop collation cases Spark folds or rejects at analysis
Two classes of tests never reached the predicate serdes and so could
not exercise the fix:
- NULL-literal RHS: Spark's `NullPropagation` rewrites `x <=> NULL`
to `IsNull(x)`, folds `x = NULL` and `NULL IN (...)` to `Literal(null)`,
etc., so the predicate serde is never asked about a collated NULL
operand. The row `(7, NULL)` in `test_collated_predicates` still
exercises NULL propagation through every remaining predicate query.
- UNICODE_CI + LIKE / Contains / StartsWith / EndsWith: Spark 4.0
restricts these to `StringTypeNonCSAICollation`, which rejects
UNICODE_CI at analysis time with a DATATYPE_MISMATCH. UTF8_LCASE
is accepted and already covers the same fallback path.
* test: tighten comments in collation.sql
Drop the "what" narration and shrink each remaining note to a single
sentence about *why* the case exists (NullPropagation folding,
NonCSAICollation restriction, InSet threshold, etc.). No SQL changes.
* collation fallback
* feat: route collated predicates through JVM codegen dispatcher
Addresses two review comments on #4948.
CollationAwareBinaryPredicate, CometIn, CometInSet, and CometLike now
mix in CodegenDispatchFallback, so an `Unsupported` result from a
collated operand stays in the Comet pipeline via Spark's own
`doGenCode` rather than falling the whole projection or filter back to
Spark. CometContains / CometStartsWith / CometEndsWith inherit the
mixin through the trait. Collation semantics still come from Spark's
codegen path, so answers stay Spark-correct.
Each of these serdes now overrides `getUnsupportedReasons` so the
collation case (and, for Like, the custom-escape case) is picked up by
GenerateDocs and surfaces in the compatibility guide instead of being
undocumented.
The SQL tests in `collation.sql` switch from
`expect_fallback(non-UTF8_BINARY collated operands)` back to plain
`query`: with the dispatcher taking over, the operator stays native
and the fallback-coverage assertion catches any regression that would
force it back to Spark.
---
.../org/apache/comet/serde/QueryPlanSerde.scala | 6 +-
.../main/scala/org/apache/comet/serde/maps.scala | 11 +-
.../scala/org/apache/comet/serde/predicates.scala | 104 +++++++++++++----
.../scala/org/apache/comet/serde/strings.scala | 36 +++++-
.../expressions/map/str_to_map_collation.sql | 9 +-
.../expressions/map/str_to_map_legacy_truncate.sql | 19 ++--
.../sql-tests/expressions/string/collation.sql | 123 +++++++++++++++++++++
7 files changed, 265 insertions(+), 43 deletions(-)
diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
index 5eee0c5cad..284094de35 100644
--- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
@@ -218,8 +218,8 @@ object QueryPlanSerde extends Logging with CometExprShim
with CometTypeShim {
classOf[Chr] -> CometScalarFunction("char"),
classOf[ConcatWs] -> CometConcatWs,
classOf[Concat] -> CometConcat,
- classOf[Contains] -> CometScalarFunction("contains"),
- classOf[EndsWith] -> CometScalarFunction("ends_with"),
+ classOf[Contains] -> CometContains,
+ classOf[EndsWith] -> CometEndsWith,
classOf[GetJsonObject] -> CometGetJsonObject,
classOf[InitCap] -> CometInitCap,
classOf[Length] -> CometLength,
@@ -233,7 +233,7 @@ object QueryPlanSerde extends Logging with CometExprShim
with CometTypeShim {
classOf[RegExpReplace] -> CometRegExpReplace,
classOf[Reverse] -> CometReverse,
classOf[RLike] -> CometRLike,
- classOf[StartsWith] -> CometScalarFunction("starts_with"),
+ classOf[StartsWith] -> CometStartsWith,
classOf[StringInstr] -> CometScalarFunction("instr"),
classOf[StringRepeat] -> CometStringRepeat,
classOf[StringReplace] -> CometStringReplace,
diff --git a/spark/src/main/scala/org/apache/comet/serde/maps.scala
b/spark/src/main/scala/org/apache/comet/serde/maps.scala
index a5ae24d215..c8304440c3 100644
--- a/spark/src/main/scala/org/apache/comet/serde/maps.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/maps.scala
@@ -166,12 +166,17 @@ object CometMapFromEntries
}
}
-object CometStrToMap extends CometScalarFunction[StringToMap]("str_to_map")
with CometTypeShim {
+object CometStrToMap
+ extends CometScalarFunction[StringToMap]("str_to_map")
+ with CometTypeShim
+ with CodegenDispatchFallback {
// Spark 4.1.1+ honours spark.sql.legacy.truncateForEmptyRegexSplit by
truncating trailing
// empty entries from the split result. Comet's native str_to_map always
behaves as if the flag
- // were false, so it is incompatible when legacy truncation is enabled. Read
by string key so it
- // resolves on older Spark versions where the config is not registered.
+ // were false. When the flag is true, mark this Incompatible so the
CodegenDispatchFallback
+ // trait routes the expression through the JVM codegen dispatcher (Spark's
own doGenCode inside
+ // the Comet kernel) rather than falling the entire projection back to
Spark. Read by string
+ // key so it resolves on older Spark versions where the config is not
registered.
private val legacyTruncateConfig =
"spark.sql.legacy.truncateForEmptyRegexSplit"
private val legacyTruncateReason =
diff --git a/spark/src/main/scala/org/apache/comet/serde/predicates.scala
b/spark/src/main/scala/org/apache/comet/serde/predicates.scala
index 63b64fbcf2..c9a70a830f 100644
--- a/spark/src/main/scala/org/apache/comet/serde/predicates.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/predicates.scala
@@ -21,7 +21,7 @@ package org.apache.comet.serde
import scala.jdk.CollectionConverters._
-import org.apache.spark.sql.catalyst.expressions.{And, Attribute,
EqualNullSafe, EqualTo, Expression, GreaterThan, GreaterThanOrEqual, In, InSet,
IsNaN, IsNotNull, IsNull, LessThan, LessThanOrEqual, Literal, Not, Or}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute,
BinaryExpression, EqualNullSafe, EqualTo, Expression, GreaterThan,
GreaterThanOrEqual, In, InSet, IsNaN, IsNotNull, IsNull, LessThan,
LessThanOrEqual, Literal, Not, Or}
import org.apache.spark.sql.types.BooleanType
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
@@ -34,26 +34,34 @@ object CometNot extends CometExpressionSerde[Not] {
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
+ import ComparisonUtils.hasCollatedOperand
+
expr.child match {
- case expr: EqualTo =>
+ case inner: EqualTo if !hasCollatedOperand(inner.left, inner.right) =>
createBinaryExpr(
- expr,
- expr.left,
- expr.right,
+ inner,
+ inner.left,
+ inner.right,
inputs,
binding,
(builder, binaryExpr) => builder.setNeq(binaryExpr))
- case expr: EqualNullSafe =>
+ case inner: EqualNullSafe if !hasCollatedOperand(inner.left,
inner.right) =>
createBinaryExpr(
- expr,
- expr.left,
- expr.right,
+ inner,
+ inner.left,
+ inner.right,
inputs,
binding,
(builder, binaryExpr) => builder.setNeqNullSafe(binaryExpr))
- case expr: In =>
- ComparisonUtils.in(expr, expr.value, expr.list, inputs, binding,
negate = true)
+ case inner: In if !hasCollatedOperand((inner.value +: inner.list): _*) =>
+ ComparisonUtils.in(inner, inner.value, inner.list, inputs, binding,
negate = true)
case _ =>
+ // Includes the collated variants of EqualTo / EqualNullSafe / In
above: fall through so
+ // the child expression's own serde is consulted, which now returns
`Unsupported` for
+ // non-UTF8_BINARY operands (see `CometEqualTo.getSupportLevel` and
siblings). That makes
+ // `exprToProtoInternal` return None for the child, which cascades
this Not to None and
+ // falls the enclosing operator back to Spark — the only way to honour
collation-aware
+ // (in)equality without a native path.
createUnaryExpr(
expr,
expr.child,
@@ -103,7 +111,7 @@ object CometOr extends CometExpressionSerde[Or] {
}
}
-object CometEqualTo extends CometExpressionSerde[EqualTo] {
+object CometEqualTo extends CollationAwareBinaryPredicate[EqualTo] {
override def convert(
expr: EqualTo,
inputs: Seq[Attribute],
@@ -118,7 +126,7 @@ object CometEqualTo extends CometExpressionSerde[EqualTo] {
}
}
-object CometEqualNullSafe extends CometExpressionSerde[EqualNullSafe] {
+object CometEqualNullSafe extends CollationAwareBinaryPredicate[EqualNullSafe]
{
override def convert(
expr: EqualNullSafe,
inputs: Seq[Attribute],
@@ -133,7 +141,7 @@ object CometEqualNullSafe extends
CometExpressionSerde[EqualNullSafe] {
}
}
-object CometGreaterThan extends CometExpressionSerde[GreaterThan] {
+object CometGreaterThan extends CollationAwareBinaryPredicate[GreaterThan] {
override def convert(
expr: GreaterThan,
inputs: Seq[Attribute],
@@ -148,7 +156,7 @@ object CometGreaterThan extends
CometExpressionSerde[GreaterThan] {
}
}
-object CometGreaterThanOrEqual extends
CometExpressionSerde[GreaterThanOrEqual] {
+object CometGreaterThanOrEqual extends
CollationAwareBinaryPredicate[GreaterThanOrEqual] {
override def convert(
expr: GreaterThanOrEqual,
inputs: Seq[Attribute],
@@ -163,7 +171,7 @@ object CometGreaterThanOrEqual extends
CometExpressionSerde[GreaterThanOrEqual]
}
}
-object CometLessThan extends CometExpressionSerde[LessThan] {
+object CometLessThan extends CollationAwareBinaryPredicate[LessThan] {
override def convert(
expr: LessThan,
inputs: Seq[Attribute],
@@ -178,7 +186,7 @@ object CometLessThan extends CometExpressionSerde[LessThan]
{
}
}
-object CometLessThanOrEqual extends CometExpressionSerde[LessThanOrEqual] {
+object CometLessThanOrEqual extends
CollationAwareBinaryPredicate[LessThanOrEqual] {
override def convert(
expr: LessThanOrEqual,
inputs: Seq[Attribute],
@@ -233,7 +241,14 @@ object CometIsNaN extends CometExpressionSerde[IsNaN] {
}
}
-object CometIn extends CometExpressionSerde[In] {
+object CometIn extends CometExpressionSerde[In] with CodegenDispatchFallback {
+
+ override def getSupportLevel(expr: In): SupportLevel =
+ ComparisonUtils.collationSupportLevel("In", (expr.value +: expr.list): _*)
+
+ override def getUnsupportedReasons(): Seq[String] =
+ Seq(ComparisonUtils.nonDefaultCollationDocReason)
+
override def convert(
expr: In,
inputs: Seq[Attribute],
@@ -242,7 +257,14 @@ object CometIn extends CometExpressionSerde[In] {
}
}
-object CometInSet extends CometExpressionSerde[InSet] {
+object CometInSet extends CometExpressionSerde[InSet] with
CodegenDispatchFallback {
+
+ override def getSupportLevel(expr: InSet): SupportLevel =
+ ComparisonUtils.collationSupportLevel("InSet", expr.child)
+
+ override def getUnsupportedReasons(): Seq[String] =
+ Seq(ComparisonUtils.nonDefaultCollationDocReason)
+
override def convert(
expr: InSet,
inputs: Seq[Attribute],
@@ -257,8 +279,52 @@ object CometInSet extends CometExpressionSerde[InSet] {
}
}
+/**
+ * Mixin for serdes of binary predicates whose native kernel compares raw
bytes: any operand
+ * carrying a non-UTF8_BINARY collation is unsupported natively. Mixing in
+ * `CodegenDispatchFallback` routes these through the JVM codegen dispatcher
(Spark's own
+ * `doGenCode`), so the collation-aware evaluation runs inline in the Comet
pipeline instead of
+ * falling the whole operator back to Spark.
+ */
+trait CollationAwareBinaryPredicate[T <: BinaryExpression]
+ extends CometExpressionSerde[T]
+ with CodegenDispatchFallback {
+ override def getSupportLevel(expr: T): SupportLevel =
+ ComparisonUtils.collationSupportLevel(expr.prettyName, expr.left,
expr.right)
+
+ override def getUnsupportedReasons(): Seq[String] =
+ Seq(ComparisonUtils.nonDefaultCollationDocReason)
+}
+
object ComparisonUtils {
+ // Comet's native equality/ordering/hashing compare raw bytes, so any
predicate operand carrying
+ // a non-UTF8_BINARY collation (Spark 4+) would produce wrong answers on the
native path — e.g.
+ // `'a' = 'A'` under `UNICODE_CI` returns true in Spark but false byte-wise.
Every binary
+ // comparison and `In`/`InSet` serde routes its `getSupportLevel` through
here, marking the case
+ // `Unsupported`; `CodegenDispatchFallback` then routes those cases through
Spark's own
+ // `doGenCode` inside the Comet pipeline. `hasNonDefaultStringCollation`
walks nested types too,
+ // so collated strings inside array/map/struct operands are also caught.
+ def nonDefaultCollationReason(exprName: String): String =
+ s"$exprName does not support non-UTF8_BINARY collated operands; " +
+ "native comparison is byte-wise and cannot honour collation semantics."
+
+ // Doc-friendly variant of `nonDefaultCollationReason` for
`getUnsupportedReasons`. The compat
+ // guide already scopes the reason to a specific expression, so no per-expr
name is needed.
+ val nonDefaultCollationDocReason: String =
+ "Non-UTF8_BINARY collated operands are routed through the JVM codegen
dispatcher " +
+ "(Spark's own `doGenCode`) because native comparison is byte-wise."
+
+ def hasCollatedOperand(operands: Expression*): Boolean =
+ operands.exists(op => hasNonDefaultStringCollation(op.dataType))
+
+ def collationSupportLevel(exprName: String, operands: Expression*):
SupportLevel =
+ if (hasCollatedOperand(operands: _*)) {
+ Unsupported(Some(nonDefaultCollationReason(exprName)))
+ } else {
+ Compatible()
+ }
+
def in(
expr: Expression,
value: Expression,
diff --git a/spark/src/main/scala/org/apache/comet/serde/strings.scala
b/spark/src/main/scala/org/apache/comet/serde/strings.scala
index 07b3c0c99f..1e124dbd42 100644
--- a/spark/src/main/scala/org/apache/comet/serde/strings.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/strings.scala
@@ -19,7 +19,7 @@
package org.apache.comet.serde
-import org.apache.spark.sql.catalyst.expressions.{Attribute, Base64,
BitLength, Cast, Concat, ConcatWs, Elt, Empty2Null, Expression, FindInSet,
FormatNumber, FormatString, GetJsonObject, InitCap, Left, Length, Levenshtein,
Like, Literal, Lower, Mask, OctetLength, Overlay, RegExpExtract,
RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike, SoundEx,
StringLocate, StringLPad, StringRepeat, StringReplace, StringRPad, StringSplit,
StringTranslate, Substring, SubstringIndex, ToCharacter [...]
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Base64,
BitLength, Cast, Concat, ConcatWs, Contains, Elt, Empty2Null, EndsWith,
Expression, FindInSet, FormatNumber, FormatString, GetJsonObject, InitCap,
Left, Length, Levenshtein, Like, Literal, Lower, Mask, OctetLength, Overlay,
RegExpExtract, RegExpExtractAll, RegExpInStr, RegExpReplace, Right, RLike,
SoundEx, StartsWith, StringLocate, StringLPad, StringRepeat, StringReplace,
StringRPad, StringSplit, StringTranslate, Substr [...]
import org.apache.spark.sql.types.{BinaryType, DataTypes, LongType, StringType}
import org.apache.comet.CometConf
@@ -288,13 +288,21 @@ object CometConcatWs extends
CometExpressionSerde[ConcatWs] {
}
}
-object CometLike extends CometExpressionSerde[Like] {
+object CometLike extends CometExpressionSerde[Like] with
CodegenDispatchFallback {
+
+ private val customEscapeReason =
+ "LIKE with a custom escape character (only `\\` is supported natively)"
+
+ override def getUnsupportedReasons(): Seq[String] =
+ Seq(customEscapeReason, ComparisonUtils.nonDefaultCollationDocReason)
override def getSupportLevel(expr: Like): SupportLevel = {
- if (expr.escapeChar == '\\') {
- Compatible()
- } else {
+ if (ComparisonUtils.hasCollatedOperand(expr.left, expr.right)) {
+ Unsupported(Some(ComparisonUtils.nonDefaultCollationReason("Like")))
+ } else if (expr.escapeChar != '\\') {
Unsupported(Some(s"custom escape character ${expr.escapeChar} not
supported in LIKE"))
+ } else {
+ Compatible()
}
}
@@ -309,6 +317,24 @@ object CometLike extends CometExpressionSerde[Like] {
}
}
+/**
+ * Serdes for `Contains` / `StartsWith` / `EndsWith` that reject
non-UTF8_BINARY collated operands
+ * and otherwise delegate to the generic `contains` / `starts_with` /
`ends_with` scalar-function
+ * bridge. The native kernels compare raw bytes and cannot honour case- or
accent-insensitive
+ * collations, so a collated operand must fall back to Spark.
+ */
+object CometContains
+ extends CometScalarFunction[Contains]("contains")
+ with CollationAwareBinaryPredicate[Contains]
+
+object CometStartsWith
+ extends CometScalarFunction[StartsWith]("starts_with")
+ with CollationAwareBinaryPredicate[StartsWith]
+
+object CometEndsWith
+ extends CometScalarFunction[EndsWith]("ends_with")
+ with CollationAwareBinaryPredicate[EndsWith]
+
/**
* `rlike` runs Spark's own implementation through the codegen dispatcher by
default, for
* byte-exact results. The native (rust) regexp engine is faster but has
different semantics from
diff --git
a/spark/src/test/resources/sql-tests/expressions/map/str_to_map_collation.sql
b/spark/src/test/resources/sql-tests/expressions/map/str_to_map_collation.sql
index 7816a1442c..41594f612c 100644
---
a/spark/src/test/resources/sql-tests/expressions/map/str_to_map_collation.sql
+++
b/spark/src/test/resources/sql-tests/expressions/map/str_to_map_collation.sql
@@ -17,17 +17,16 @@
-- MinSparkVersion: 4.0
--- Spark 4.0+ applies collation-aware delimiter matching in str_to_map.
Comet's native
--- implementation matches raw delimiter bytes, so non-UTF8_BINARY collations
must fall back.
+-- Spark 4.0+ applies collation-aware delimiter matching in str_to_map.
-- collated input string
-query expect_fallback(`str_to_map` does not support non-UTF8_BINARY collations)
+query
SELECT str_to_map('a:1,b:2' COLLATE UTF8_LCASE, ',', ':')
-- collated pair delimiter
-query expect_fallback(`str_to_map` does not support non-UTF8_BINARY collations)
+query
SELECT str_to_map('aX1,bX2', 'x' COLLATE UTF8_LCASE, ':')
-- collated key-value delimiter
-query expect_fallback(`str_to_map` does not support non-UTF8_BINARY collations)
+query
SELECT str_to_map('aX1,bX2', ',', 'x' COLLATE UTF8_LCASE)
diff --git
a/spark/src/test/resources/sql-tests/expressions/map/str_to_map_legacy_truncate.sql
b/spark/src/test/resources/sql-tests/expressions/map/str_to_map_legacy_truncate.sql
index a5b5eb9f16..8ab8058920 100644
---
a/spark/src/test/resources/sql-tests/expressions/map/str_to_map_legacy_truncate.sql
+++
b/spark/src/test/resources/sql-tests/expressions/map/str_to_map_legacy_truncate.sql
@@ -15,19 +15,22 @@
-- specific language governing permissions and limitations
-- under the License.
--- Tests that str_to_map falls back to Spark when
--- spark.sql.legacy.truncateForEmptyRegexSplit is enabled. In legacy mode
Spark truncates trailing
--- empty entries from the split result, which Comet's native str_to_map does
not honour.
+-- Tests that str_to_map routes through the JVM codegen dispatcher when
+-- spark.sql.legacy.truncateForEmptyRegexSplit is enabled. In legacy mode
Spark truncates
+-- trailing empty entries from the split result, which Comet's native
str_to_map does not
+-- honour. `CometStrToMap` marks the expression Incompatible when the flag is
on and mixes in
+-- [[CodegenDispatchFallback]], so the projection stays native (Spark's own
`doGenCode` runs
+-- inside the Comet kernel) while producing Spark-exact results.
-- See https://github.com/apache/datafusion-comet/issues/4477
-- Config: spark.sql.legacy.truncateForEmptyRegexSplit=true
--- trailing pair delimiter: legacy mode truncates the trailing empty entry, so
Comet must fall
--- back to Spark
-query expect_fallback(truncateForEmptyRegexSplit)
+-- trailing pair delimiter: legacy mode truncates the trailing empty entry;
Comet delegates to
+-- the codegen dispatcher.
+query
SELECT str_to_map('a:1,b:2,', ',', ':')
--- column input also falls back
+-- column input is also handled via the codegen dispatcher
statement
CREATE TABLE test_str_to_map_legacy(s STRING, pair_delim STRING,
key_value_delim STRING) USING parquet
@@ -37,5 +40,5 @@ INSERT INTO test_str_to_map_legacy VALUES
('x:1;y:2;', ';', ':'),
(NULL, ',', ':')
-query expect_fallback(truncateForEmptyRegexSplit)
+query
SELECT str_to_map(s, pair_delim, key_value_delim) FROM test_str_to_map_legacy
diff --git
a/spark/src/test/resources/sql-tests/expressions/string/collation.sql
b/spark/src/test/resources/sql-tests/expressions/string/collation.sql
index 89dd95b9e7..534725b83f 100644
--- a/spark/src/test/resources/sql-tests/expressions/string/collation.sql
+++ b/spark/src/test/resources/sql-tests/expressions/string/collation.sql
@@ -51,3 +51,126 @@ SELECT concat('Hello' COLLATE UNICODE_CI, 'World' COLLATE
UNICODE_CI)
query
SELECT reverse('Hello' COLLATE UNICODE_CI)
+
+-- ============================================================================
+-- Collated predicate operands route through the JVM codegen dispatcher.
+-- The predicate serdes mark collated cases `Unsupported`;
`CodegenDispatchFallback` runs
+-- Spark's own `doGenCode` inside the Comet pipeline so the operator stays
native while
+-- matching Spark's collation-aware answer.
+-- ============================================================================
+
+statement
+CREATE TABLE test_collated_predicates(id INT, s STRING) USING parquet
+
+statement
+INSERT INTO test_collated_predicates VALUES
+ (1, 'a'),
+ (2, 'A'),
+ (3, 'hello'),
+ (4, 'HELLO'),
+ (5, 'World'),
+ (6, 'b'),
+ (7, NULL)
+
+-- ---------- EqualTo / EqualNullSafe / inequality ---------------------------
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) = 'A' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE CAST(s AS STRING COLLATE
UTF8_LCASE) = 'A' ORDER BY id
+
+-- UNICODE_CI covers a second ICU collation to prove the guard is not
UTF8_LCASE-only.
+query
+SELECT id, CAST(s AS STRING COLLATE UNICODE_CI) = 'A' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) != 'A' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) <> 'A' FROM
test_collated_predicates ORDER BY id
+
+-- `x <=> NULL` folds via NullPropagation, so we test only a non-NULL RHS.
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) <=> 'A' FROM
test_collated_predicates ORDER BY id
+
+-- ---------- Ordering ------------------------------------------------------
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) < 'b' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) <= 'a' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) > 'a' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) >= 'A' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, CAST(s AS STRING COLLATE UNICODE_CI) < 'b' FROM
test_collated_predicates ORDER BY id
+
+-- ---------- In / NotIn / InSet ---------------------------------------------
+
+query
+SELECT id FROM test_collated_predicates WHERE CAST(s AS STRING COLLATE
UTF8_LCASE) IN ('A', 'HELLO') ORDER BY id
+
+-- NOT IN exercises CometNot's collated fall-through into the generic path.
+query
+SELECT id FROM test_collated_predicates WHERE CAST(s AS STRING COLLATE
UTF8_LCASE) NOT IN ('A', 'HELLO') ORDER BY id
+
+-- 12 elements forces InSet (spark.sql.optimizer.inSetConversionThreshold=10
by default).
+query
+SELECT id FROM test_collated_predicates WHERE CAST(s AS STRING COLLATE
UTF8_LCASE) IN ('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'HELLO')
ORDER BY id
+
+-- ---------- NOT (EqualTo / EqualNullSafe) rewrites ------------------------
+
+query
+SELECT id FROM test_collated_predicates WHERE NOT (CAST(s AS STRING COLLATE
UTF8_LCASE) = 'A') ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE NOT (CAST(s AS STRING COLLATE
UTF8_LCASE) <=> 'A') ORDER BY id
+
+-- ---------- Like -----------------------------------------------------------
+
+query
+SELECT id, CAST(s AS STRING COLLATE UTF8_LCASE) LIKE '%LLO' FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE CAST(s AS STRING COLLATE
UTF8_LCASE) LIKE 'H%' ORDER BY id
+
+-- Spark 4.0 rejects UNICODE_CI for LIKE / Contains / StartsWith / EndsWith at
analysis time
+-- (StringTypeNonCSAICollation), so only UTF8_LCASE is covered for these.
+
+-- ---------- Contains / StartsWith / EndsWith -------------------------------
+
+query
+SELECT id, contains(CAST(s AS STRING COLLATE UTF8_LCASE), 'HE') FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, startswith(CAST(s AS STRING COLLATE UTF8_LCASE), 'HE') FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id, endswith(CAST(s AS STRING COLLATE UTF8_LCASE), 'LO') FROM
test_collated_predicates ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE contains(CAST(s AS STRING
COLLATE UTF8_LCASE), 'ELL') ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE startswith(CAST(s AS STRING
COLLATE UTF8_LCASE), 'HE') ORDER BY id
+
+query
+SELECT id FROM test_collated_predicates WHERE endswith(CAST(s AS STRING
COLLATE UTF8_LCASE), 'O') ORDER BY id
+
+-- ---------- Nested-collated types ------------------------------------------
+
+-- hasNonDefaultStringCollation recurses into array/struct element types.
+query
+SELECT id, array(CAST(s AS STRING COLLATE UTF8_LCASE)) = array('A' COLLATE
UTF8_LCASE) FROM test_collated_predicates ORDER BY id
+
+query
+SELECT id, struct(CAST(s AS STRING COLLATE UTF8_LCASE)) = struct('A' COLLATE
UTF8_LCASE) FROM test_collated_predicates ORDER BY id
+
+-- NULL-literal operands are omitted: NullPropagation folds them before Comet
sees them. The
+-- (7, NULL) row covers NULL-in-data on every query above.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
