This is an automated email from the ASF dual-hosted git repository.
dtenedor pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7fd360913344 [SPARK-60003][SQL] Do not promote collation-only CAST
when applying LCT
7fd360913344 is described below
commit 7fd3609133445be25b934485bde6c4704d578d42
Author: Serge Rielau <[email protected]>
AuthorDate: Tue Oct 6 22:38:22 2026 -0700
[SPARK-60003][SQL] Do not promote collation-only CAST when applying LCT
### What changes were proposed in this pull request?
SPARK-58798 nested every `Cast` when applying string-family LCT so
CHAR/VARCHAR truncation and padding stay on the inner node. That is correct
when the length constraint changes. For a collation-only change, wrapping
`CAST(non-string AS STRING COLLATE x)` makes the outer Cast see a string child,
so collation strength is promoted from Default to Implicit. A `CASE` of that
CAST next to a UTF8_BINARY literal then becomes Implicit UNICODE, which beats
an Implicit column in `concat` and fa [...]
This PR:
- Nests Cast only when the CHAR/VARCHAR constraint changes; collation-only
LCT retargets with `copy(dataType)` so a non-string CAST stays Default.
- Keeps SPARK-58798 same-collation LCT (`max(n, m)` including mixed
Implicit vs Default CHAR).
- Restores the pre-SPARK-58798 Default rule that UTF8_BINARY yields to the
other Default collation. Two Implicit columns, or two non-UTF8 Default CASTs,
still mismatch.
Does not change CAST collation strength to Implicit.
### Why are the changes needed?
`CREATE TABLE AS SELECT` of `a || ... || CASE ... TRY_CAST(b AS STRING
COLLATE UNICODE_CI_AI) ...` fails with `INDETERMINATE_COLLATION_IN_SCHEMA` when
`a` is already that collation. The analyzed CASE was Implicit UNICODE vs
Default UTF8_BINARY, so CASE became Implicit and then conflicted with concat.
### Does this PR introduce _any_ user-facing change?
Yes. After SPARK-58798, collation-only LCT wrapping could raise
`INDETERMINATE_COLLATION_IN_SCHEMA` on concat/CTAS that previously succeeded.
Those queries succeed again. Two Default non-UTF8 collated CASTs still conflict.
### How was this patch tested?
- `IndeterminateCollationTestSuite`: CASE/concat/CTAS repro (`TRY_CAST(date
AS STRING COLLATE UNICODE_CI_AI)`) and negative two Default non-UTF8 CASTs.
- `BasicCharVarcharTestSuite`: SPARK-58798 nested truncation `CAST('abcdef'
AS VARCHAR(2)` inside COALESCE and SPARK-58794 mixed-strength CHAR pad.
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt
-Dsbt.override.build.repos=true \
'sql/testOnly
org.apache.spark.sql.collation.IndeterminateCollationTestSuite
org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "collated mixed-length
LCT" -z "collated cast of non-string" -z "two Default non-UTF8" -z "least
common type for COALESCE"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
Closes #59260 from srielau/SPARK-60003.
Authored-by: Serge Rielau <[email protected]>
Signed-off-by: Daniel Tenedorio <[email protected]>
---
.../catalyst/analysis/CollationTypeCoercion.scala | 32 ++++++--------
.../apache/spark/sql/CharVarcharTestSuite.scala | 10 +++++
.../IndeterminateCollationTestSuite.scala | 51 ++++++++++++++++++++++
3 files changed, 74 insertions(+), 19 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
index ea1a1917d8f5..82160dec2a18 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
@@ -111,12 +111,10 @@ object CollationTypeCoercion extends SQLConfHelper {
/**
* Changes the data type of the expression to the given `newType`.
*
- * Never retarget an existing Cast (`cast.copy(dataType = ...)`). Explicit
CAST
- * truncation / overflow (ISO 6.13) and CHAR padding must stay on the inner
node;
- * LCT is an outer Cast. Uncollated TypeCoercion already nests.
- *
- * Literals: `copy(dataType)` is enough when only collation changes. When a
string
- * constraint changes (CHAR(2) to CHAR(4)), wrap in Cast so padding is
re-applied.
+ * Literal and Cast: wrap in a new Cast when the CHAR/VARCHAR constraint
+ * changes so truncation, overflow, and padding stay on the original node.
+ * Collation-only changes retarget with `copy(dataType)`, which keeps a
+ * non-string Cast child non-string and therefore Default strength.
*/
private def changeType(expr: Expression, newType: DataType): Expression = {
mergeTypes(expr.dataType, newType) match {
@@ -127,8 +125,10 @@ object CollationTypeCoercion extends SQLConfHelper {
case lit: Literal if stringConstraintChanged(lit.dataType,
newDataType) =>
Cast(lit, newDataType, timeZoneId =
Some(conf.sessionLocalTimeZone))
case lit: Literal => lit.copy(dataType = newDataType)
- case cast: Cast =>
+ case cast: Cast if stringConstraintChanged(cast.dataType,
newDataType) =>
Cast(cast, newDataType, timeZoneId =
Some(conf.sessionLocalTimeZone))
+ case cast: Cast =>
+ cast.copy(dataType = newDataType)
case subquery: SubqueryExpression =>
changeTypeInSubquery(subquery, newType)
@@ -448,19 +448,13 @@ object CollationTypeCoercion extends SQLConfHelper {
/**
* Resolves collation strength independently of CHAR/VARCHAR length.
*
- * This rule always runs. First-class CHAR/VARCHAR appear whenever
- * `charVarcharFirstClassTypes` is true (`standardSemantics` or
- * `preserveCharVarcharTypeInfo`), not only under `standardSemantics`.
- *
- * Same collation, including mixed strength: take the string-family LCT
`max(n, m)`
- * (pads, never truncates) and attach the stronger strength. Example:
- * `coalesce(CAST('a' AS CHAR(2) COLLATE UTF8_LCASE),
- * CAST(1 AS CHAR(4) COLLATE UTF8_LCASE))` is CHAR(4) COLLATE UTF8_LCASE
- * (Implicit CHAR(2) from a string CAST vs Default CHAR(4) from a non-string
CAST).
+ * Runs whenever first-class CHAR/VARCHAR are enabled (`standardSemantics` or
+ * `preserveCharVarcharTypeInfo`).
*
- * Different collations at equal strength: mismatch (error if Explicit, else
- * indeterminate). Different collations at unequal strength: the stronger
operand
- * wins in full, including its length (SQL collation precedence).
+ * Same collation: string-family LCT `max(n, m)` with the stronger strength.
+ * Different collations, equal strength: mismatch (error if Explicit, else
+ * indeterminate). Different collations, unequal strength: the stronger
+ * operand wins in full, including its length.
*/
private def getWinningStringType(
left: StringTypeWithContext,
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
index 9f5274d18701..ab553de65de6 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
@@ -1189,6 +1189,14 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
// LCT(NULL, T) = T
assert(sql("SELECT coalesce(null, cast('a' AS CHAR(5))) AS c")
.schema.head.dataType === CharType(5))
+ // SPARK-58798 intent: LCT widen nests Cast. Inner CAST('abcdef' AS
VARCHAR(2)/CHAR(2))
+ // truncates to "ab"; retargeting the inner Cast to length 4 would yield
"abcd".
+ checkAnswer(
+ sql("SELECT coalesce(cast('abcdef' AS VARCHAR(2)), cast('x' AS
VARCHAR(4))) AS c"),
+ Row("ab"))
+ checkAnswer(
+ sql("SELECT coalesce(cast('abcdef' AS CHAR(2)), cast('x' AS CHAR(4)))
AS c"),
+ Row("ab "))
}
}
@@ -1549,6 +1557,8 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
s"$key=$value same-strength mixed CHAR lengths")
assert(sql(mixedStrength).schema.head.dataType === CharType(4,
"UTF8_LCASE"),
s"$key=$value Implicit CHAR(2) vs Default CHAR(4) must widen, not
narrow")
+ // Mixed strength must still pad the Implicit CHAR(2) to CHAR(4), not
keep length 2.
+ checkAnswer(sql(mixedStrength), Row("a "))
}
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/collation/IndeterminateCollationTestSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/collation/IndeterminateCollationTestSuite.scala
index 8c5d2eb9d40f..f60153034f8b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/collation/IndeterminateCollationTestSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/collation/IndeterminateCollationTestSuite.scala
@@ -259,4 +259,55 @@ class IndeterminateCollationTestSuite extends
SharedSparkSession {
sql(s"SELECT c1 || c2 as col FROM $testTableName").show()
}
}
+
+ // SPARK-60003: a Default-strength collated CAST over a non-string child
+ // (TRY_CAST(date AS STRING COLLATE UNICODE_CI_AI)) in a CASE next to a
default literal
+ // must not poison an outer concat whose other operand is already
UNICODE_CI_AI.
+ test("collated cast of non-string in CASE concat stays determinate") {
+ withTable(testTableName, "tst_cast_collation_out") {
+ sql(
+ s"""
+ |CREATE TABLE $testTableName (
+ | a STRING COLLATE UNICODE_CI_AI,
+ | b DATE
+ |) USING $dataSource
+ |""".stripMargin)
+ sql(s"INSERT INTO $testTableName VALUES ('x', DATE'2020-01-01')")
+
+ checkAnswer(
+ sql(
+ s"""SELECT collation(
+ | a || 'X' ||
+ | (CASE WHEN b IS NULL THEN 'XXXXXX'
+ | ELSE TRY_CAST(b AS STRING COLLATE UNICODE_CI_AI) END) ||
'X')
+ |FROM $testTableName""".stripMargin),
+ Row("SYSTEM.BUILTIN.UNICODE_CI_AI"))
+
+ sql(
+ s"""CREATE TABLE tst_cast_collation_out USING $dataSource AS SELECT
+ | a || 'X' ||
+ | (CASE WHEN b IS NULL THEN 'XXXXXX'
+ | ELSE TRY_CAST(b AS STRING COLLATE UNICODE_CI_AI) END) ||
'X' AS o
+ |FROM $testTableName""".stripMargin)
+ }
+ }
+
+ // SPARK-58798 did not change this: two Default non-UTF8 collated CASTs
still conflict.
+ test("two Default non-UTF8 collated casts stay indeterminate") {
+ withTable(testTableName, "tst_two_default_casts") {
+ sql(
+ s"""
+ |CREATE TABLE $testTableName (
+ | a STRING COLLATE UNICODE_CI_AI,
+ | b DATE
+ |) USING $dataSource
+ |""".stripMargin)
+ assertIndeterminateCollationInSchemaError("o")(
+ sql(
+ s"""CREATE TABLE tst_two_default_casts USING $dataSource AS SELECT
+ | TRY_CAST(b AS STRING COLLATE UNICODE_CI_AI) ||
+ | TRY_CAST(b AS STRING COLLATE UTF8_LCASE) AS o
+ |FROM $testTableName""".stripMargin))
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
