This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6c5c5331bbb8 [SPARK-57412][SQL] Apply default collation to generated
column expression references
6c5c5331bbb8 is described below
commit 6c5c5331bbb8497edf3418994f67fda8b8ce4648
Author: Szehon Ho <[email protected]>
AuthorDate: Tue Jun 16 15:56:29 2026 -0700
[SPARK-57412][SQL] Apply default collation to generated column expression
references
### What changes were proposed in this pull request?
`ApplyDefaultCollation` now re-points default string/char/varchar
`AttributeReference`s inside a column's generation expression to the table's
default collation, in the same place it assigns the default collation to the
`ColumnDefinition` itself. The `ColumnDefinition` case guard is broadened so
this also runs when the generated column's own output type is not a default
string type (e.g. `c INT GENERATED ALWAYS AS (LENGTH(s))`).
### Why are the changes needed?
This is a regression from SPARK-55347, which moved generated-column
validation from the standalone `GeneratedColumnAnalyzer` (previously run at
planning time against the final, collation-applied schema) to inline analysis
plus `CheckAnalysis` / `GeneratedColumnExpression.validate`.
Generated columns are not allowed to involve non-UTF8-binary collated
string types.
**Before SPARK-55347 (worked):**
1. The generation expression was kept as a SQL string, not resolved during
analysis.
2. `ApplyDefaultCollation` applied the table's `DEFAULT COLLATION` to the
`ColumnDefinition`s.
3. Later, at planning time, the expression was parsed and resolved against
the **final** (collated) columns via `GeneratedColumnAnalyzer`.
4. The resolved `AttributeReference`s carried the real collation, so the
non-UTF8 collation check correctly rejected the table.
**After SPARK-55347 (bug):**
1. The generation expression is resolved early during analysis, so its
`AttributeReference`s capture the column types as `UTF8_BINARY`.
2. `ApplyDefaultCollation` applies the `DEFAULT COLLATION` to the
`ColumnDefinition`s, but not to those already-resolved `AttributeReference`s
inside the `GeneratedColumnExpression`.
3. `CheckAnalysis` runs `GeneratedColumnExpression.validate`, which walks
the expression tree; the `AttributeReference`s still show `UTF8_BINARY`.
4. The check passes and the table is wrongly created.
In short, the ordering flipped: before, columns were collated *then* the
expression was resolved; after, the expression is resolved *then* the columns
are collated, leaving the references stale.
Example that wrongly succeeded:
```sql
CREATE TABLE t (
c1 STRING,
c2 STRING GENERATED ALWAYS AS (SUBSTRING(c1, 0, 1))
) USING <v2source> DEFAULT COLLATION UTF8_LCASE;
```
The identical schema with `c1 STRING COLLATE UTF8_LCASE` is correctly
rejected.
### Does this PR introduce _any_ user-facing change?
Yes. `CREATE`/`REPLACE TABLE` with a generated column whose expression
involves a non-UTF8-binary collated column (via `DEFAULT COLLATION`) now
correctly fails with `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN`, consistent with
the behavior for an explicit per-column `COLLATE`. This is a fix within the
unreleased master/branch that carries SPARK-55347.
### How was this patch tested?
New tests in `CollationSuite` covering:
- a generated column rejected via table-level `DEFAULT COLLATION`,
- a non-string-output generated column (`INT GENERATED ALWAYS AS
(LENGTH(c1))`) rejected for both explicit and default collation,
- a generated column allowed under `DEFAULT COLLATION UTF8_BINARY`.
Existing `CollationSuite`,
`DefaultCollation{String,CharVarchar}TestSuite{V1,V2}`,
`DataSourceV2DataFrameSuite`, and `DataSourceV2SQLSuite` generated-column /
generation-expression tests all pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes #56465 from szehon-ho/fix-generated-column-collation.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../catalyst/analysis/ApplyDefaultCollation.scala | 50 ++++++-
.../spark/sql/collation/CollationSuite.scala | 165 ++++++++++++++++++++-
2 files changed, 210 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala
index ca9418a0c6ef..4485c299d218 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.SparkException
-import org.apache.spark.sql.catalyst.expressions.{Cast,
DefaultStringProducingExpression, Expression, Literal, SubqueryExpression}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Cast,
DefaultStringProducingExpression, Expression, Literal, SubqueryExpression}
import org.apache.spark.sql.catalyst.plans.logical.{AddColumns, AlterColumns,
AlterColumnSpec, AlterViewAs, ColumnDefinition, CreateTable,
CreateTableAsSelect, CreateTempView, CreateUserDefinedFunction, CreateView,
LogicalPlan, QualifiedColType, ReplaceColumns, ReplaceTable,
ReplaceTableAsSelect, TableSpec, V2CreateTablePlan}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.CurrentOrigin
@@ -245,6 +245,19 @@ object ApplyDefaultCollation extends Rule[LogicalPlan] {
private def transform(plan: LogicalPlan, collation: String): LogicalPlan = {
plan resolveOperators {
+ // The ResolvedIdentifier child of CreateTable/ReplaceTable exposes the
resolved table
+ // columns as output attributes. Re-point any default
string/char/varchar typed ones to the
+ // default collation so the resolved schema stays consistent with the
column definitions.
+ // Matched before the generic case below so it is not routed through
transformPlan, which
+ // does not rewrite bare output attributes.
+ case ri: ResolvedIdentifier
+ if ri.output.exists(a =>
hasDefaultStringCharOrVarcharType(a.dataType)) =>
+ ri.copy(output = ri.output.map {
+ case a if hasDefaultStringCharOrVarcharType(a.dataType) =>
+ a.withDataType(replaceDefaultStringCharAndVarcharTypes(a.dataType,
collation))
+ case a => a
+ })
+
case p if isCreateOrAlterPlan(p) ||
AnalysisContext.get.collation.isDefined =>
transformPlan(p, collation)
@@ -348,9 +361,24 @@ object ApplyDefaultCollation extends Rule[LogicalPlan] {
* collated types.
*/
private def transformExpression: PartialFunction[Expression, String =>
Expression] = {
- case columnDef: ColumnDefinition if
hasDefaultStringCharOrVarcharType(columnDef.dataType) =>
- collation => columnDef.copy(dataType =
- replaceDefaultStringCharAndVarcharTypes(columnDef.dataType, collation))
+ case columnDef: ColumnDefinition
+ if hasDefaultStringCharOrVarcharType(columnDef.dataType) ||
+ generationExpressionNeedsCollation(columnDef) =>
+ collation =>
+ // Re-point any default string/char/varchar typed references inside
the generation
+ // expression to the default collation, matching the collation applied
to the columns
+ // they reference. Other nodes (e.g. literals and casts) are handled
by the dedicated
+ // cases below as part of the bottom-up traversal.
+ val newGenExpr = columnDef.generationExpression.map { genExpr =>
+ val newChild = genExpr.child.transform {
+ case a: AttributeReference if
hasDefaultStringCharOrVarcharType(a.dataType) =>
+
a.withDataType(replaceDefaultStringCharAndVarcharTypes(a.dataType, collation))
+ }
+ if (newChild eq genExpr.child) genExpr else genExpr.copy(child =
newChild)
+ }
+ columnDef.copy(
+ dataType =
replaceDefaultStringCharAndVarcharTypes(columnDef.dataType, collation),
+ generationExpression = newGenExpr)
case cast: Cast if hasDefaultStringCharOrVarcharType(cast.dataType) &&
cast.containsTag(Cast.USER_SPECIFIED_CAST) =>
@@ -371,6 +399,20 @@ object ApplyDefaultCollation extends Rule[LogicalPlan] {
subquery.withNewPlan(newPlan)
}
+ /**
+ * Returns true if the column's generation expression references a default
string/char/varchar
+ * type.
+ */
+ private def generationExpressionNeedsCollation(columnDef: ColumnDefinition):
Boolean = {
+ columnDef.generationExpression.exists { genExpr =>
+ // Only inspect AttributeReferences, avoiding calling dataType on
unresolved nodes.
+ genExpr.child.exists {
+ case a: AttributeReference =>
hasDefaultStringCharOrVarcharType(a.dataType)
+ case _ => false
+ }
+ }
+ }
+
/**
* Casts [[DefaultStringProducingExpression]] in the plan to the `newType`.
*/
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala
index a6f4de1c80e3..86f5add99c8e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala
@@ -22,8 +22,9 @@ import scala.jdk.CollectionConverters.MapHasAsJava
import com.ibm.icu.util.VersionInfo.ICU_VERSION
import org.apache.spark.SparkException
-import org.apache.spark.sql.{AnalysisException, Row}
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
import org.apache.spark.sql.catalyst.ExtendedAnalysisException
+import org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.util.CollationFactory
import org.apache.spark.sql.connector.{DatasourceV2SQLBase,
FakeV2ProviderWithCustomSchema}
@@ -981,6 +982,168 @@ class CollationSuite extends DatasourceV2SQLBase with
AdaptiveSparkPlanHelper {
"generation expression cannot contain non utf8 binary collated
string type"))
}
+ test("Generated column expressions using table default collation - errors
out") {
+ // The referenced columns are declared as plain STRING (default
UTF8_BINARY), but the
+ // table-level DEFAULT COLLATION makes them effectively non-UTF8 binary
collated. The
+ // generation expression must be rejected the same way an explicit
per-column COLLATE would be.
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING,
+ | c2 STRING GENERATED ALWAYS AS (SUBSTRING(c1, 0, 1))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UNICODE
+ |""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "SUBSTRING(c1, 0, 1)",
+ "reason" ->
+ "generation expression cannot contain non utf8 binary collated
string type"))
+
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING,
+ | c2 STRING GENERATED ALWAYS AS (LOWER(c1))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UTF8_LCASE
+ |""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "LOWER(c1)",
+ "reason" ->
+ "generation expression cannot contain non utf8 binary collated
string type"))
+
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | struct1 STRUCT<a: STRING>,
+ | c2 STRING GENERATED ALWAYS AS (UCASE(struct1.a))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UNICODE
+ |""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "UCASE(struct1.a)",
+ "reason" ->
+ "generation expression cannot contain non utf8 binary collated
string type"))
+ }
+
+ test("Generated column with non-string output referencing collated column -
errors out") {
+ // The non-UTF8 collation check is structural: a generation expression
that references a
+ // collated string column is rejected even when the generated column's own
type is not a
+ // string (here INT). This must hold both for an explicit per-column
collation and for a
+ // table-level DEFAULT COLLATION.
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING COLLATE UNICODE,
+ | c2 INT GENERATED ALWAYS AS (LENGTH(c1))
+ |)
+ |USING $v2Source
+ |""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "LENGTH(c1)",
+ "reason" ->
+ "generation expression cannot contain non utf8 binary collated
string type"))
+
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING,
+ | c2 INT GENERATED ALWAYS AS (LENGTH(c1))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UNICODE
+ |""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "LENGTH(c1)",
+ "reason" ->
+ "generation expression cannot contain non utf8 binary collated
string type"))
+ }
+
+ test("Generated column with UTF8_BINARY default collation is allowed") {
+ withTable("testcat.test_table") {
+ sql(
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING,
+ | c2 STRING GENERATED ALWAYS AS (SUBSTRING(c1, 0, 1))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UTF8_BINARY
+ |""".stripMargin)
+ }
+ }
+
+ test("Unresolved column in generated column under DEFAULT COLLATION reports
a friendly error") {
+ val query =
+ s"""
+ |CREATE TABLE testcat.test_table(
+ | c1 STRING,
+ | c2 INT GENERATED ALWAYS AS (LENGTH(c_typo))
+ |)
+ |USING $v2Source
+ |DEFAULT COLLATION UNICODE
+ |""".stripMargin
+ val start = query.indexOf("c_typo")
+ checkError(
+ exception = intercept[AnalysisException](sql(query)),
+ condition = "UNRESOLVED_COLUMN.WITH_SUGGESTION",
+ parameters = Map(
+ "objectName" -> "`c_typo`",
+ "proposal" -> "`c1`, `c2`"),
+ queryContext = Array(ExpectedContext("c_typo", start, start +
"c_typo".length - 1)))
+ }
+
+ test("CREATE TABLE with DEFAULT COLLATION correctly applies collation") {
+ val t = "testcat.ns1.tbl"
+ withTable(t) {
+ val queryExecutions = QueryTest.withQueryExecutionsCaptured(spark) {
+ sql(s"CREATE TABLE $t (c1 STRING, c2 STRING COLLATE UNICODE) " +
+ s"DEFAULT COLLATION UTF8_LCASE")
+ }
+ val resolvedIdentifier = queryExecutions.flatMap { qe =>
+ qe.analyzed.collectFirst { case ri: ResolvedIdentifier => ri }
+ }.headOption
+ assert(resolvedIdentifier.nonEmpty,
+ "Expected a ResolvedIdentifier in the analyzed CREATE TABLE plan")
+ val output = resolvedIdentifier.get.output
+ assert(output.nonEmpty,
+ "Expected the ResolvedIdentifier to expose the table columns as
output")
+
+ // c1 has no explicit collation, so it inherits the table default
collation.
+ assert(output.find(_.name ==
"c1").map(_.dataType).contains(StringType("UTF8_LCASE")))
+ // c2 has an explicit collation and must be left untouched.
+ assert(output.find(_.name ==
"c2").map(_.dataType).contains(StringType("UNICODE")))
+ }
+ }
+
test("Cast expression for collations") {
checkAnswer(
sql(s"SELECT collation(cast('a' as string collate utf8_lcase))"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
