This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a9472969d8f5 [SPARK-57777][SQL][CONNECT] Distinguish explicit
collation when rendering string literals to SQL
a9472969d8f5 is described below
commit a9472969d8f574b86e0deb33f9676f1e1e66508c
Author: Wenchen Fan <[email protected]>
AuthorDate: Wed Jul 1 22:11:39 2026 +0200
[SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering
string literals to SQL
### What changes were proposed in this pull request?
This PR makes `Literal.sql` render an explicit `collate` clause for any
string literal whose
type carries an explicit collation, **including an explicit
`UTF8_BINARY`**, while still rendering
the default (un-collated) `StringType` without a clause.
`Literal.sql` previously had two arms for string literals:
```scala
case (v: UTF8String, StringType) => // matched any UTF8_BINARY
StringType by value equality
"'" + escaped + "'"
case (v: UTF8String, st: StringType) => // only reached for
non-UTF8_BINARY collations
"'" + escaped + "'" + st.typeName.substring(6)
```
The first arm matches via the `StringType` case object's `equals` (which
compares `collationId`
and `constraint`), so it collapsed both the *default* `StringType` and an
*explicitly collated*
`UTF8_BINARY` `StringType` into the same clause-less output. The two arms
are now merged into one
that decides the clause via
`DataTypeUtils.isDefaultStringCharOrVarcharType` (the same
singleton-identity check used elsewhere, e.g. by `ApplyDefaultCollation`
and `SHOW CREATE TABLE`):
```scala
case (v: UTF8String, st: StringType) =>
val collateClause =
if (DataTypeUtils.isDefaultStringCharOrVarcharType(st)) "" else s"
collate ${st.collationName}"
"'" + escaped + "'" + collateClause
```
For non-default collations the produced string is byte-for-byte identical
to the previous
`st.typeName.substring(6)` output (`typeName` is `s"string collate
$collationName"`).
This PR also removes a test-only normalization in `PlanGenerationTestSuite`
that stamped
`UTF8_BINARY` onto every string-type proto whose `collation` field was
empty before writing the
golden files. That shim made the generated `query-tests` golden artifacts
(`.proto.bin` / `.json`,
and the downstream `.explain` files) misrepresent the real wire format: a
default `StringType` is
serialized with an **empty** `collation` field (the "undetermined / default
collation" sentinel),
not `UTF8_BINARY`. The affected golden files were regenerated so they now
reflect what a real
client actually sends.
### Why are the changes needed?
A default `StringType` and an explicitly-`UTF8_BINARY` `StringType` are
semantically different:
the former is "undetermined" and is eligible to inherit a default collation
during analysis (e.g.
`CREATE TABLE ... DEFAULT COLLATION UTF8_LCASE AS SELECT 'x'`), while the
latter is explicitly
pinned and must not inherit. `Literal.sql` is used in view text, `SHOW
CREATE TABLE`, error
messages, etc.; rendering an explicitly-collated `UTF8_BINARY` literal
without a `collate` clause
loses that distinction and is not faithful on re-parse. This aligns
string-literal SQL rendering
with how the rest of the engine already distinguishes explicit collation.
The test-normalization removal is a correctness fix for the Spark Connect
golden files: they are
meant to be a faithful record of the protocol, and they were showing
`collate UTF8_BINARY` on
default strings where the actual proto omits the collation.
### Does this PR introduce _any_ user-facing change?
Yes. `Literal.sql` now appends ` collate UTF8_BINARY` when a string
literal's type is an explicit
(non-default) `UTF8_BINARY` `StringType`. A plain default string literal is
unchanged (no clause),
and literals with other explicit collations are unchanged. This affects SQL
text generated from
literals (e.g. view definitions, `SHOW CREATE TABLE`, error messages).
### How was this patch tested?
Existing suites, all passing:
- `catalyst/testOnly
org.apache.spark.sql.catalyst.expressions.LiteralExpressionSuite` (49)
- `connect-client-jvm/testOnly
org.apache.spark.sql.PlanGenerationTestSuite` (727; golden files regenerated
and reviewed)
- `connect/testOnly
org.apache.spark.sql.connect.ProtoToParsedPlanTestSuite` (732; explain golden
regenerated)
- `sql/testOnly org.apache.spark.sql.SQLQueryTestSuite` for the collation
inputs (`collations-basic`, `view-with-default-collation`,
`collations-padding-trim`, `collations-string-functions`, `collations-aliases`,
`listagg-collations`) (12; no golden changes)
- `sql/testOnly
org.apache.spark.sql.execution.command.{v1,v2}.ShowCreateTableSuite` (47)
The regenerated golden diff is limited to dropping the test-stamped
`collation: "UTF8_BINARY"`
from default-string proto types and the corresponding `CAST(NULL AS STRING
COLLATE UTF8_BINARY)`
-> `CAST(NULL AS STRING)` in one explain file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56892 from cloud-fan/cloud-fan/string-literal-default-collation.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/expressions/literals.scala | 17 ++++++++++++-----
.../expressions/LiteralExpressionSuite.scala | 13 +++++++++++++
.../apache/spark/sql/PlanGenerationTestSuite.scala | 12 ------------
.../explain-results/function_typedLit.explain | 2 +-
.../query-tests/queries/csv_from_dataset.json | 1 -
.../query-tests/queries/csv_from_dataset.proto.bin | Bin 169 -> 156 bytes
.../query-tests/queries/function_lit_array.json | 2 --
.../queries/function_lit_array.proto.bin | Bin 5346 -> 5320 bytes
.../query-tests/queries/function_typedLit.json | 16 ----------------
.../query-tests/queries/function_typedLit.proto.bin | Bin 11681 -> 11471 bytes
.../query-tests/queries/json_from_dataset.json | 1 -
.../query-tests/queries/json_from_dataset.proto.bin | Bin 180 -> 167 bytes
12 files changed, 26 insertions(+), 38 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
index b1ed1a494be8..cea16999c437 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
@@ -592,13 +592,20 @@ case class Literal (value: Any, dataType: DataType)
extends LeafExpression {
override def sql: String = (value, dataType) match {
case (_, NullType | _: ArrayType | _: MapType | _: StructType) if value ==
null => "NULL"
case _ if value == null => s"CAST(NULL AS ${dataType.sql})"
- case (v: UTF8String, StringType) =>
- // Escapes all backslashes and single quotes.
- "'" + v.toString.replace("\\", "\\\\").replace("'", "\\'") + "'"
case (v: UTF8String, st: StringType) =>
+ // Only render a `collate` clause for an explicit collation (including
an explicit
+ // `UTF8_BINARY`). The default `StringType` (the case object) has no
explicit collation, so
+ // it must render without a clause and stay distinguishable from an
explicitly-collated
+ // string on re-parse (e.g. so that default-collation resolution does
not treat an
+ // explicitly-collated literal as eligible for inheriting a default
collation).
+ val collateClause =
+ if (DataTypeUtils.isDefaultStringCharOrVarcharType(st)) {
+ ""
+ } else {
+ s" collate ${st.collationName}"
+ }
// Escapes all backslashes and single quotes.
- "'" + v.toString.replace("\\", "\\\\").replace("'", "\\'") +
- "'" + st.typeName.substring(6)
+ "'" + v.toString.replace("\\", "\\\\").replace("'", "\\'") + "'" +
collateClause
case (v: Byte, ByteType) => s"${v}Y"
case (v: Short, ShortType) => s"${v}S"
case (v: Long, LongType) => s"${v}L"
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
index b69b29f78914..58fff9bad849 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
@@ -842,4 +842,17 @@ class LiteralExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(lit.dataType === GeometryType(0))
assert(lit.value.isInstanceOf[BinaryView])
}
+
+ test("SPARK-57777: render explicit collation in string literal SQL") {
+ // The default `StringType` (case object) has no explicit collation, so it
renders
+ // without a `collate` clause.
+ assert(Literal(UTF8String.fromString("x"), StringType).sql === "'x'")
+ // A non-default (non-singleton) `UTF8_BINARY` `StringType` is an explicit
collation, so it
+ // renders the clause and stays distinguishable from the default on
re-parse.
+ assert(Literal(UTF8String.fromString("x"), StringType("UTF8_BINARY")).sql
===
+ "'x' collate UTF8_BINARY")
+ // Other explicit collations are rendered as before.
+ assert(Literal(UTF8String.fromString("x"), StringType("UTF8_LCASE")).sql
===
+ "'x' collate UTF8_LCASE")
+ }
}
diff --git
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
index a74b25459bad..36b2cab4d586 100644
---
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
+++
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
@@ -198,7 +198,6 @@ class PlanGenerationTestSuite extends ConnectFunSuite with
Logging {
/**
* Normalize proto messages for stable comparison:
* - Trim JVM origin fields (lines, stack traces, anonymous function names)
- * - Populate default StringType collation when missing (UTF8_BINARY)
*/
private def normalizeProtoForComparison[T <: protobuf.Message](message: T):
T = {
def trim(builder: proto.JvmOrigin.Builder): Unit = {
@@ -221,17 +220,6 @@ class PlanGenerationTestSuite extends ConnectFunSuite with
Logging {
val builder = message.toBuilder
builder match {
- // For comparison only, we add UTF8_BINARY when StringType collation is
missing
- // to ensure deterministic plan equality across environments.
- case dt: proto.DataType.Builder if dt.getKindCase ==
proto.DataType.KindCase.STRING =>
- val sb = dt.getStringBuilder
- if (sb.getCollation.isEmpty) {
- val defaultCollationName =
- CollationFactory
- .fetchCollation(CollationFactory.UTF8_BINARY_COLLATION_ID)
- .collationName
- sb.setCollation(defaultCollationName)
- }
case exp: proto.Relation.Builder
if exp.hasCommon && exp.getCommon.hasOrigin &&
exp.getCommon.getOrigin.hasJvmOrigin =>
trim(exp.getCommonBuilder.getOriginBuilder.getJvmOriginBuilder)
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_typedLit.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_typedLit.explain
index 5a827ca88ee7..3c878be34143 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_typedLit.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_typedLit.explain
@@ -1,2 +1,2 @@
-Project [id#0L, id#0L, 1 AS 1#0, null AS NULL#0, true AS true#0, 68 AS 68#0,
9872 AS 9872#0, -8726532 AS -8726532#0, 7834609328726532 AS
7834609328726532#0L, 2.718281828459045 AS 2.718281828459045#0, -0.8 AS -0.8#0,
89.97620 AS 89.97620#0, 89889.7667231 AS 89889.7667231#0, connect! AS
connect!#0, T AS T#0, ABCDEFGHIJ AS ABCDEFGHIJ#0,
0x78797A7B7C7D7E7F808182838485868788898A8B8C8D8E AS
X'78797A7B7C7D7E7F808182838485868788898A8B8C8D8E'#0, 0x0806 AS X'0806'#0, [8,6]
AS ARRAY(8, 6)#0, null A [...]
+Project [id#0L, id#0L, 1 AS 1#0, null AS NULL#0, true AS true#0, 68 AS 68#0,
9872 AS 9872#0, -8726532 AS -8726532#0, 7834609328726532 AS
7834609328726532#0L, 2.718281828459045 AS 2.718281828459045#0, -0.8 AS -0.8#0,
89.97620 AS 89.97620#0, 89889.7667231 AS 89889.7667231#0, connect! AS
connect!#0, T AS T#0, ABCDEFGHIJ AS ABCDEFGHIJ#0,
0x78797A7B7C7D7E7F808182838485868788898A8B8C8D8E AS
X'78797A7B7C7D7E7F808182838485868788898A8B8C8D8E'#0, 0x0806 AS X'0806'#0, [8,6]
AS ARRAY(8, 6)#0, null A [...]
+- LocalRelation <empty>, [id#0L, a#0, b#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.json
b/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.json
index e4b31258f984..d34fcb6f758e 100644
---
a/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.json
+++
b/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.json
@@ -18,7 +18,6 @@
"name": "c1",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.proto.bin
b/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.proto.bin
index c39243a10a8e..5f8bd50685ca 100644
Binary files
a/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.proto.bin
and
b/sql/connect/common/src/test/resources/query-tests/queries/csv_from_dataset.proto.bin
differ
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.json
b/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.json
index 9b77a8bd9b42..9e5227af8685 100644
---
a/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.json
+++
b/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.json
@@ -521,7 +521,6 @@
"array": {
"elementType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"containsNull": true
@@ -579,7 +578,6 @@
"array": {
"elementType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"containsNull": true
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.proto.bin
b/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.proto.bin
index 20b6d81c3cee..db81bd4e7eba 100644
Binary files
a/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.proto.bin
and
b/sql/connect/common/src/test/resources/query-tests/queries/function_lit_array.proto.bin
differ
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.json
b/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.json
index 41ca771596ef..eba5b99fe495 100644
---
a/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.json
+++
b/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.json
@@ -82,7 +82,6 @@
},
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
}
},
@@ -958,7 +957,6 @@
"name": "_2",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -1023,7 +1021,6 @@
"name": "_2",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -1106,7 +1103,6 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
@@ -1157,7 +1153,6 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
@@ -1202,7 +1197,6 @@
"name": "_1",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -1545,7 +1539,6 @@
"name": "_2",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -1555,7 +1548,6 @@
"array": {
"elementType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"containsNull": true
@@ -1629,7 +1621,6 @@
"name": "b",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -1707,7 +1698,6 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
@@ -1813,14 +1803,12 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
"array": {
"elementType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"containsNull": true
@@ -1896,7 +1884,6 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
@@ -1992,7 +1979,6 @@
"map": {
"keyType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueType": {
@@ -2010,7 +1996,6 @@
"name": "_1",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
@@ -2024,7 +2009,6 @@
},
"valueType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"valueContainsNull": true
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.proto.bin
b/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.proto.bin
index 5068b513a927..4388912bdb1f 100644
Binary files
a/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.proto.bin
and
b/sql/connect/common/src/test/resources/query-tests/queries/function_typedLit.proto.bin
differ
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.json
b/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.json
index f29245374e6e..d6f992d09a5c 100644
---
a/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.json
+++
b/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.json
@@ -18,7 +18,6 @@
"name": "c1",
"dataType": {
"string": {
- "collation": "UTF8_BINARY"
}
},
"nullable": true
diff --git
a/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.proto.bin
b/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.proto.bin
index 1ce2e676ce30..0fce9d9ff8c7 100644
Binary files
a/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.proto.bin
and
b/sql/connect/common/src/test/resources/query-tests/queries/json_from_dataset.proto.bin
differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
