cloud-fan opened a new pull request, #56892:
URL: https://github.com/apache/spark/pull/56892
### What changes were proposed in this pull request?
This PR makes `Literal.sql` render an explicit `collate` clause for any
string literal whose
type carries an explicit collation, **including an explicit `UTF8_BINARY`**,
while still rendering
the default (un-collated) `StringType` without a clause.
`Literal.sql` previously had two arms for string literals:
```scala
case (v: UTF8String, StringType) => // matched any UTF8_BINARY
StringType by value equality
"'" + escaped + "'"
case (v: UTF8String, st: StringType) => // only reached for
non-UTF8_BINARY collations
"'" + escaped + "'" + st.typeName.substring(6)
```
The first arm matches via the `StringType` case object's `equals` (which
compares `collationId`
and `constraint`), so it collapsed both the *default* `StringType` and an
*explicitly collated*
`UTF8_BINARY` `StringType` into the same clause-less output. The two arms
are now merged into one
that decides the clause via `DataTypeUtils.isDefaultStringCharOrVarcharType`
(the same
singleton-identity check used elsewhere, e.g. by `ApplyDefaultCollation` and
`SHOW CREATE TABLE`):
```scala
case (v: UTF8String, st: StringType) =>
val collateClause =
if (DataTypeUtils.isDefaultStringCharOrVarcharType(st)) "" else s"
collate ${st.collationName}"
"'" + escaped + "'" + collateClause
```
For non-default collations the produced string is byte-for-byte identical to
the previous
`st.typeName.substring(6)` output (`typeName` is `s"string collate
$collationName"`).
This PR also removes a test-only normalization in `PlanGenerationTestSuite`
that stamped
`UTF8_BINARY` onto every string-type proto whose `collation` field was empty
before writing the
golden files. That shim made the generated `query-tests` golden artifacts
(`.proto.bin` / `.json`,
and the downstream `.explain` files) misrepresent the real wire format: a
default `StringType` is
serialized with an **empty** `collation` field (the "undetermined / default
collation" sentinel),
not `UTF8_BINARY`. The affected golden files were regenerated so they now
reflect what a real
client actually sends.
### Why are the changes needed?
A default `StringType` and an explicitly-`UTF8_BINARY` `StringType` are
semantically different:
the former is "undetermined" and is eligible to inherit a default collation
during analysis (e.g.
`CREATE TABLE ... DEFAULT COLLATION UTF8_LCASE AS SELECT 'x'`), while the
latter is explicitly
pinned and must not inherit. `Literal.sql` is used in view text, `SHOW
CREATE TABLE`, error
messages, etc.; rendering an explicitly-collated `UTF8_BINARY` literal
without a `collate` clause
loses that distinction and is not faithful on re-parse. This aligns
string-literal SQL rendering
with how the rest of the engine already distinguishes explicit collation.
The test-normalization removal is a correctness fix for the Spark Connect
golden files: they are
meant to be a faithful record of the protocol, and they were showing
`collate UTF8_BINARY` on
default strings where the actual proto omits the collation.
### Does this PR introduce _any_ user-facing change?
Yes. `Literal.sql` now appends ` collate UTF8_BINARY` when a string
literal's type is an explicit
(non-default) `UTF8_BINARY` `StringType`. A plain default string literal is
unchanged (no clause),
and literals with other explicit collations are unchanged. This affects SQL
text generated from
literals (e.g. view definitions, `SHOW CREATE TABLE`, error messages).
### How was this patch tested?
Existing suites, all passing:
- `catalyst/testOnly
org.apache.spark.sql.catalyst.expressions.LiteralExpressionSuite` (49)
- `connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite`
(727; golden files regenerated and reviewed)
- `connect/testOnly org.apache.spark.sql.connect.ProtoToParsedPlanTestSuite`
(732; explain golden regenerated)
- `sql/testOnly org.apache.spark.sql.SQLQueryTestSuite` for the collation
inputs (`collations-basic`, `view-with-default-collation`,
`collations-padding-trim`, `collations-string-functions`, `collations-aliases`,
`listagg-collations`) (12; no golden changes)
- `sql/testOnly
org.apache.spark.sql.execution.command.{v1,v2}.ShowCreateTableSuite` (47)
The regenerated golden diff is limited to dropping the test-stamped
`collation: "UTF8_BINARY"`
from default-string proto types and the corresponding `CAST(NULL AS STRING
COLLATE UTF8_BINARY)`
-> `CAST(NULL AS STRING)` in one explain file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
