srielau opened a new pull request, #59260:
URL: https://github.com/apache/spark/pull/59260
### What changes were proposed in this pull request?
SPARK-58798 nested every `Cast` when applying string-family LCT so
CHAR/VARCHAR truncation and padding stay on the inner node. That is correct
when the length constraint changes. For a collation-only change, wrapping
`CAST(non-string AS STRING COLLATE x)` makes the outer Cast see a string child,
so collation strength is promoted from Default to Implicit. A `CASE` of that
CAST next to a UTF8_BINARY literal then becomes Implicit UNICODE, which beats
an Implicit column in `concat` and fails CTAS with
`INDETERMINATE_COLLATION_IN_SCHEMA`.
This PR:
- Nests Cast only when the CHAR/VARCHAR constraint changes; collation-only
LCT retargets with `copy(dataType)` so a non-string CAST stays Default.
- Keeps SPARK-58798 same-collation LCT (`max(n, m)` including mixed Implicit
vs Default CHAR).
- Restores the pre-SPARK-58798 Default rule that UTF8_BINARY yields to the
other Default collation. Two Implicit columns, or two non-UTF8 Default CASTs,
still mismatch.
Does not change CAST collation strength to Implicit.
### Why are the changes needed?
`CREATE TABLE AS SELECT` of `a || ... || CASE ... TRY_CAST(b AS STRING
COLLATE UNICODE_CI_AI) ...` fails with `INDETERMINATE_COLLATION_IN_SCHEMA` when
`a` is already that collation. The analyzed CASE was Implicit UNICODE vs
Default UTF8_BINARY, so CASE became Implicit and then conflicted with concat.
### Does this PR introduce _any_ user-facing change?
Yes. After SPARK-58798, collation-only LCT wrapping could raise
`INDETERMINATE_COLLATION_IN_SCHEMA` on concat/CTAS that previously succeeded.
Those queries succeed again. Two Default non-UTF8 collated CASTs still conflict.
### How was this patch tested?
- `IndeterminateCollationTestSuite`: CASE/concat/CTAS repro (`TRY_CAST(date
AS STRING COLLATE UNICODE_CI_AI)`) and negative two Default non-UTF8 CASTs.
- `BasicCharVarcharTestSuite`: SPARK-58798 nested truncation `CAST('abcdef'
AS VARCHAR(2)` inside COALESCE and SPARK-58794 mixed-strength CHAR pad.
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt
-Dsbt.override.build.repos=true \
'sql/testOnly
org.apache.spark.sql.collation.IndeterminateCollationTestSuite
org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "collated mixed-length
LCT" -z "collated cast of non-string" -z "two Default non-UTF8" -z "least
common type for COALESCE"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
