This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ac5bba0dec7c [SPARK-58798][SQL] Fix collated CHAR LCT and cover
compare/IN/set-ops
ac5bba0dec7c is described below
commit ac5bba0dec7c8908b54dfdb0e585d8ce4951a2f7
Author: Serge Rielau <[email protected]>
AuthorDate: Thu Aug 20 16:58:26 2026 +0000
[SPARK-58798][SQL] Fix collated CHAR LCT and cover compare/IN/set-ops
### What changes were proposed in this pull request?
Covers the string-family LCT lattice
([SPARK-58798](https://issues.apache.org/jira/browse/SPARK-58798)). Parent:
[SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Also includes
explicit character CAST truncation
([SPARK-58797](https://issues.apache.org/jira/browse/SPARK-58797)).
Rebased onto `master` after #58033 merged. Unique commits vs master:
https://github.com/srielau/spark/compare/master...SPARK-58794-lct-compare
Follow-up that drills into least-common-type, comparison, and `IN` under
`spark.sql.charVarchar.standardSemantics.enabled`.
`CollationTypeCoercion` is not gated on `standardSemantics` alone
(`charVarcharFirstClassTypes` is also true under
`preserveCharVarcharTypeInfo`). The equal-collation length LCT therefore
applies in preserve-only mode as well; tests cover both flags.
**Bug fix.** `CollationTypeCoercion.getWinningStringType` required
`sameType` at equal collation strength. Same-collation `CHAR(2)` and `CHAR(4)`
therefore looked like a collation mismatch and became `IndeterminateStringType`
(`string collate null`). At equal strength, take
`StringHelper.tightestCommonString` instead so every child is cast to `max(n,
m)` (padding, never truncating).
**Explicit CAST vs store assignment (ISO 6.13 / 9.2).** Under the flag:
- Character-to-`CHAR`/`VARCHAR` `CAST` / `TRY_CAST` **truncates** to `n`
characters (then `CHAR` pads). Example: `CAST('abcdef' AS VARCHAR(2))` is
`'ab'`, not `EXCEED_LIMIT_LENGTH`.
- Numeric (and other non-string) `CAST` to `CHAR`/`VARCHAR` still
**errors** when the formatted literal does not fit (`CAST(12345 AS VARCHAR(4))`
-> `EXCEED_LIMIT_LENGTH` / 54006). That matches ISO 22001, not silent digit
clipping.
- Store assignment (`INSERT`, `Dataset.to`, encoder write-side) is
unchanged: non-space overflow still errors. Project store-assignment casts to
unconstrained `STRING` then `stringLengthCheck`, so they do not inherit CAST
truncation.
**Coverage / pinned behavior.**
- Set ops: `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT`, plus multi-row
`VALUES`, all share the string-family LCT.
- Comparison and `IN`: every participant is cast to that LCT, including the
`IN` left-hand side (`InTypeCoercion` uses `findWiderCommonType` over `value +:
list`). Casting to `CHAR` pads, so unequal `CHAR` lengths compare equal after
widen; casting to `VARCHAR`/`STRING` keeps the `CHAR` pad, so `CHAR 'a'`
(stored as `'a '`) is not equal to `VARCHAR`/`STRING 'a'` unless the other side
carries the same trailing blank. Ignoring trailing blanks is a collation
concern (`RTRIM`), not a type [...]
- Parameterized lengths (`CHAR(:n)` / `VARCHAR(?)`): markers in length
position already bind through the parser; under the flag, character CAST
truncates to the bound `n`, numeric CAST still errors if it does not fit, and
DDL keeps the bound `n`. Negative / non-integral lengths fail at parse after
substitution.
### Why are the changes needed?
#58033 left the collated mixed-length LCT gap as a known issue, and had
only thin coverage of `UNION ALL` / `INTERSECT` with no compare / `IN`-LHS /
`VALUES` / `EXCEPT` matrix. Without pinning the LCT-cast rule, CHAR vs CHAR vs
VARCHAR equality is easy to misread as PAD SPACE.
Explicit character CAST was still using the write-side length check, so
`CAST('abcdef' AS VARCHAR(2))` failed the same way as `INSERT`. ISO 6.13
truncates that CAST; 9.2 keeps the INSERT error. Numeric CAST must still fail
when the literal does not fit (ISO 22001).
### Does this PR introduce _any_ user-facing change?
Yes, when the flag is on:
- Collated `CHAR`/`VARCHAR` values of different lengths but the same
collation now widen to `CHAR`/`VARCHAR(max(n,m))` instead of an indeterminate
collation. Compare / `IN` / set-op results for mixed `CHAR` lengths follow the
LCT-cast rule above (this matches what the engine already did for non-collated
paths; the new tests lock it in).
- Explicit character-to-`CHAR`/`VARCHAR` CAST truncates instead of raising
`EXCEED_LIMIT_LENGTH`. `INSERT` / encoder overflow and numeric CAST overflow
still error.
### How was this patch tested?
- New `BasicCharVarcharTestSuite` cases covering compare, `IN` (including
analyzed-plan assertion that the LHS widens), `RTRIM`, collated LCT, `UNION` /
`INTERSECT` / `EXCEPT`, multi-row `VALUES`, parameterized `CHAR`/`VARCHAR`
lengths, character CAST truncation, and numeric CAST overflow under the flag.
- Expanded golden file `charvarchar-standard-semantics.sql`.
- Locally: `BasicCharVarcharTestSuite` and the golden file regeneration/run.
Closes #58080 from srielau/SPARK-58794-lct-compare.
Authored-by: Serge Rielau <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
.../sql/catalyst/util/CharVarcharCodegenUtils.java | 24 ++
.../catalyst/analysis/CollationTypeCoercion.scala | 66 +++-
.../catalyst/analysis/TableOutputResolver.scala | 9 +-
.../spark/sql/catalyst/expressions/Cast.scala | 4 +
.../sql/catalyst/expressions/ToStringBase.scala | 45 ++-
.../plans/logical/basicLogicalOperators.scala | 33 +-
.../spark/sql/catalyst/util/CharVarcharUtils.scala | 9 +
.../catalyst/util/ResolveDefaultColumnsUtil.scala | 3 +-
.../charvarchar-standard-semantics.sql.out | 377 ++++++++++++++++++++
.../inputs/charvarchar-standard-semantics.sql | 115 +++++-
.../results/charvarchar-standard-semantics.sql.out | 393 ++++++++++++++++++++-
.../apache/spark/sql/CharVarcharTestSuite.scala | 211 ++++++++++-
12 files changed, 1226 insertions(+), 63 deletions(-)
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
index 659833851ccd..4c1d1014f5bb 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
@@ -54,6 +54,30 @@ public class CharVarcharCodegenUtils {
}
}
+ /**
+ * Applies the SQL explicit-cast rules for a character string source and
CHAR target.
+ *
+ * Unlike store assignment, an explicit character-to-character cast
truncates non-space
+ * characters instead of raising a right-truncation exception.
+ */
+ public static UTF8String charTypeCast(UTF8String inputStr, int limit) {
+ int numChars = inputStr.numChars();
+ if (numChars == limit) {
+ return inputStr;
+ } else if (numChars < limit) {
+ return inputStr.rpad(limit, SPACE);
+ } else {
+ return inputStr.substring(0, limit);
+ }
+ }
+
+ /**
+ * Applies the SQL explicit-cast rules for a character string source and
VARCHAR target.
+ */
+ public static UTF8String varcharTypeCast(UTF8String inputStr, int limit) {
+ return inputStr.numChars() > limit ? inputStr.substring(0, limit) :
inputStr;
+ }
+
public static UTF8String readSidePadding(UTF8String inputStr, int limit) {
int numChars = inputStr.numChars();
if (numChars == limit) {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
index f43dfb77a9c0..ea1a1917d8f5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
@@ -25,7 +25,10 @@ import
org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Proj
import org.apache.spark.sql.catalyst.trees.TreeNodeTag
import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLExpr
import org.apache.spark.sql.errors.QueryCompilationErrors
-import org.apache.spark.sql.types.{ArrayType, DataType,
IndeterminateStringType, MapType, NullType, StringType, StructType}
+import org.apache.spark.sql.types.{
+ ArrayType, DataType, IndeterminateStringType, MapType, NullType,
StringHelper,
+ StringType, StructType
+}
import org.apache.spark.sql.util.SchemaUtils
/**
@@ -107,6 +110,13 @@ object CollationTypeCoercion extends SQLConfHelper {
/**
* Changes the data type of the expression to the given `newType`.
+ *
+ * Never retarget an existing Cast (`cast.copy(dataType = ...)`). Explicit
CAST
+ * truncation / overflow (ISO 6.13) and CHAR padding must stay on the inner
node;
+ * LCT is an outer Cast. Uncollated TypeCoercion already nests.
+ *
+ * Literals: `copy(dataType)` is enough when only collation changes. When a
string
+ * constraint changes (CHAR(2) to CHAR(4)), wrap in Cast so padding is
re-applied.
*/
private def changeType(expr: Expression, newType: DataType): Expression = {
mergeTypes(expr.dataType, newType) match {
@@ -114,8 +124,11 @@ object CollationTypeCoercion extends SQLConfHelper {
assert(!newDataType.existsRecursively(_.isInstanceOf[StringTypeWithContext]))
expr match {
+ case lit: Literal if stringConstraintChanged(lit.dataType,
newDataType) =>
+ Cast(lit, newDataType, timeZoneId =
Some(conf.sessionLocalTimeZone))
case lit: Literal => lit.copy(dataType = newDataType)
- case cast: Cast => cast.copy(dataType = newDataType)
+ case cast: Cast =>
+ Cast(cast, newDataType, timeZoneId =
Some(conf.sessionLocalTimeZone))
case subquery: SubqueryExpression =>
changeTypeInSubquery(subquery, newType)
@@ -127,6 +140,24 @@ object CollationTypeCoercion extends SQLConfHelper {
}
}
+ /**
+ * True when CHAR/VARCHAR length (or nested length) differs between `from`
and `to`.
+ * Collation-only differences are not a constraint change.
+ */
+ private def stringConstraintChanged(from: DataType, to: DataType): Boolean =
{
+ (from, to) match {
+ case (f: StringType, t: StringType) => f.constraint != t.constraint
+ case (ArrayType(fe, _), ArrayType(te, _)) => stringConstraintChanged(fe,
te)
+ case (MapType(fk, fv, _), MapType(tk, tv, _)) =>
+ stringConstraintChanged(fk, tk) || stringConstraintChanged(fv, tv)
+ case (fs: StructType, ts: StructType) if fs.length == ts.length =>
+ fs.fields.indices.exists { i =>
+ stringConstraintChanged(fs.fields(i).dataType, ts.fields(i).dataType)
+ }
+ case _ => false
+ }
+ }
+
/**
* Changes the data type of the expression in the subquery to the given
`newType`.
* Currently only supports subqueries with [[Project]] and [[Aggregate]]
plan.
@@ -414,7 +445,23 @@ object CollationTypeCoercion extends SQLConfHelper {
}
}
- /** Determines the winning StringTypeWithContext based on the strength of
the collation. */
+ /**
+ * Resolves collation strength independently of CHAR/VARCHAR length.
+ *
+ * This rule always runs. First-class CHAR/VARCHAR appear whenever
+ * `charVarcharFirstClassTypes` is true (`standardSemantics` or
+ * `preserveCharVarcharTypeInfo`), not only under `standardSemantics`.
+ *
+ * Same collation, including mixed strength: take the string-family LCT
`max(n, m)`
+ * (pads, never truncates) and attach the stronger strength. Example:
+ * `coalesce(CAST('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ * CAST(1 AS CHAR(4) COLLATE UTF8_LCASE))` is CHAR(4) COLLATE UTF8_LCASE
+ * (Implicit CHAR(2) from a string CAST vs Default CHAR(4) from a non-string
CAST).
+ *
+ * Different collations at equal strength: mismatch (error if Explicit, else
+ * indeterminate). Different collations at unequal strength: the stronger
operand
+ * wins in full, including its length (SQL collation precedence).
+ */
private def getWinningStringType(
left: StringTypeWithContext,
right: StringTypeWithContext): StringTypeWithContext = {
@@ -427,14 +474,13 @@ object CollationTypeCoercion extends SQLConfHelper {
}
}
- (left.strength.priority, right.strength.priority) match {
- case (leftPriority, rightPriority) if leftPriority == rightPriority =>
- if (left.sameType(right)) left
- else handleMismatch()
+ val winner =
+ if (left.strength.priority <= right.strength.priority) left else right
- case (leftPriority, rightPriority) =>
- if (leftPriority < rightPriority) left
- else right
+ StringHelper.tightestCommonString(left.stringType, right.stringType) match
{
+ case Some(lct) => StringTypeWithContext(lct, winner.strength)
+ case None if left.strength.priority == right.strength.priority =>
handleMismatch()
+ case None => winner
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
index b50c77eee479..933513654dc1 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
@@ -236,7 +236,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
if (canWriteValue) {
val nullCheckedValue = checkNullability(value, attr, conf, colPath)
val casted = cast(nullCheckedValue, attrTypeWithoutCharVarchar, conf,
colPath.quoted)
- val exprWithStrLenCheck = if (conf.charVarcharAsString ||
!attrTypeHasCharVarchar) {
+ val exprWithStrLenCheck = if (!attrTypeHasCharVarchar ||
+ !CharVarcharUtils.shouldApplyWriteSideLengthCheck(conf)) {
casted
} else {
CharVarcharUtils.stringLengthCheck(casted, attr.dataType)
@@ -333,7 +334,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
expectedCol: Attribute,
conf: SQLConf): NamedExpression = {
val rawType =
CharVarcharUtils.getRawType(expectedCol.metadata).getOrElse(expectedCol.dataType)
- val checked = if (!conf.charVarcharAsString &&
CharVarcharUtils.hasCharVarchar(rawType)) {
+ val checked = if (CharVarcharUtils.hasCharVarchar(rawType) &&
+ CharVarcharUtils.shouldApplyWriteSideLengthCheck(conf)) {
val value = defaultExpr match {
case a: Alias => a.child
case other => other
@@ -854,7 +856,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
} else {
val udtUnwrapped = unwrapUDT(queryExpr)
val casted = cast(udtUnwrapped, attrTypeWithoutCharVarchar, conf,
colPath.quoted)
- if (conf.charVarcharAsString || !attrTypeHasCharVarchar) {
+ if (!attrTypeHasCharVarchar ||
+ !CharVarcharUtils.shouldApplyWriteSideLengthCheck(conf)) {
casted
} else {
CharVarcharUtils.stringLengthCheck(casted, tableAttr.dataType)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index b1fcee354b83..5e5bb00df6c2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -692,6 +692,10 @@ case class Cast(
override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression =
copy(timeZoneId = Option(timeZoneId))
+ // Parser, Column.cast, and Connect set USER_SPECIFIED_CAST.
Analyzer-inserted Casts do not.
+ override protected def truncateCharVarcharOnCast: Boolean =
+ containsTag(Cast.USER_SPECIFIED_CAST)
+
override protected def withNewChildInternal(newChild: Expression): Cast =
copy(child = newChild)
// CAST_TO_TIMESTAMP must be set on a superset of the targets accepted by
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
index 97f41b296055..096c1788df63 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
@@ -35,6 +35,13 @@ import org.apache.spark.util.SparkStringUtils
trait ToStringBase { self: UnaryExpression with TimeZoneAwareExpression =>
+ /**
+ * ISO 6.13 truncation applies only to user-written CAST / TRY_CAST.
Implicit and
+ * store-assignment Casts keep the write-side length check.
[[ToPrettyString]] is never
+ * a CAST, so it stays false.
+ */
+ protected def truncateCharVarcharOnCast: Boolean = false
+
private lazy val dateFormatter = DateFormatter()
private lazy val timeFormatter = new FractionTimeFormatter()
private lazy val timestampFormatter =
TimestampFormatter.getFractionFormatter(zoneId)
@@ -57,14 +64,22 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
// Returns a function to convert a value to pretty string. The function
assumes input is not null.
protected final def castToString(
- from: DataType, to: StringConstraint = NoConstraint): Any => UTF8String =
- to match {
- case FixedLength(length) =>
- s =>
CharVarcharCodegenUtils.charTypeWriteSideCheck(castToString(from)(s), length)
- case MaxLength(length) =>
- s =>
CharVarcharCodegenUtils.varcharTypeWriteSideCheck(castToString(from)(s), length)
- case NoConstraint => castToString(from)
+ from: DataType, to: StringConstraint = NoConstraint): Any => UTF8String
= {
+ val toUTF8String = castToString(from)
+ (to, from) match {
+ case (FixedLength(length), _: StringType)
+ if SQLConf.get.charVarcharStandardSemantics &&
truncateCharVarcharOnCast =>
+ s => CharVarcharCodegenUtils.charTypeCast(toUTF8String(s), length)
+ case (MaxLength(length), _: StringType)
+ if SQLConf.get.charVarcharStandardSemantics &&
truncateCharVarcharOnCast =>
+ s => CharVarcharCodegenUtils.varcharTypeCast(toUTF8String(s), length)
+ case (FixedLength(length), _) =>
+ s => CharVarcharCodegenUtils.charTypeWriteSideCheck(toUTF8String(s),
length)
+ case (MaxLength(length), _) =>
+ s =>
CharVarcharCodegenUtils.varcharTypeWriteSideCheck(toUTF8String(s), length)
+ case (NoConstraint, _) => toUTF8String
}
+ }
// The Types Framework is the single integration point for framework types'
cast-to-string, via
// the zone-less formatUTF8. The cast's session zone is threaded into the
lookup so TIMESTAMP_LTZ
@@ -196,14 +211,22 @@ trait ToStringBase { self: UnaryExpression with
TimeZoneAwareExpression =>
(c, evPrim) => {
val tmpVar = ctx.freshVariable("tmp", classOf[UTF8String])
val castToString = castToStringCode(from, ctx)(c, tmpVar)
- val maintainConstraint = to match {
- case FixedLength(length) =>
+ val maintainConstraint = (to, from) match {
+ case (FixedLength(length), _: StringType)
+ if SQLConf.get.charVarcharStandardSemantics &&
truncateCharVarcharOnCast =>
+ code"""$evPrim =
org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils
+ .charTypeCast($tmpVar, $length);""".stripMargin
+ case (MaxLength(length), _: StringType)
+ if SQLConf.get.charVarcharStandardSemantics &&
truncateCharVarcharOnCast =>
+ code"""$evPrim =
org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils
+ .varcharTypeCast($tmpVar, $length);""".stripMargin
+ case (FixedLength(length), _) =>
code"""$evPrim =
org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils
.charTypeWriteSideCheck($tmpVar, $length);""".stripMargin
- case MaxLength(length) =>
+ case (MaxLength(length), _) =>
code"""$evPrim =
org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils
.varcharTypeWriteSideCheck($tmpVar, $length);""".stripMargin
- case NoConstraint => code"$evPrim = $tmpVar;"
+ case (NoConstraint, _) => code"$evPrim = $tmpVar;"
}
code"""
UTF8String $tmpVar;
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
index 2798e2a82d24..fb2dffed92b7 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
@@ -184,13 +184,44 @@ object Project {
if (other == target) {
col
} else if (Cast.canANSIStoreAssign(other, target)) {
- Cast(col, target, Option(conf.sessionLocalTimeZone), ansiEnabled =
true)
+ storeAssignCast(col, other, target, conf)
} else {
throw
QueryCompilationErrors.invalidColumnOrFieldDataTypeError(columnPath, other,
target)
}
}
}
+ /**
+ * Cast `col` for ANSI store assignment without using character-to-character
CAST
+ * truncation (ISO 6.13).
+ *
+ * For CHAR/VARCHAR targets the plan is Cast to unconstrained STRING, then
+ * `stringLengthCheck` (write-side overflow). Avoid
replaceCharVarcharWithString
+ * so first-class types stay CHAR/VARCHAR.
+ */
+ private def storeAssignCast(
+ col: Expression,
+ other: DataType,
+ target: DataType,
+ conf: SQLConf): Expression = {
+ val (castTarget, lengthCheckType) = target match {
+ case c: CharType => (c.toStringType, Some(c: DataType))
+ case v: VarcharType => (v.toStringType, Some(v: DataType))
+ case otherType => (otherType, None)
+ }
+ val casted = if (other == castTarget) {
+ col
+ } else {
+ Cast(col, castTarget, Option(conf.sessionLocalTimeZone), ansiEnabled =
true)
+ }
+ lengthCheckType match {
+ case Some(dt) if CharVarcharUtils.shouldApplyWriteSideLengthCheck(conf)
=>
+ CharVarcharUtils.stringLengthCheck(casted, dt)
+ case _ =>
+ casted
+ }
+ }
+
private def reorderFields(
fields: Seq[(String, Expression)],
expected: Seq[StructField],
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
index 1b8f68175878..00adcfe69bc5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
@@ -167,6 +167,15 @@ object CharVarcharUtils extends Logging with
SparkCharVarcharUtils {
}.getOrElse(expr)
}
+ /**
+ * Write-side CHAR/VARCHAR length checks apply unless the session is on the
legacy
+ * `charVarcharAsString` path with no first-class types. `standardSemantics`
and
+ * `preserveCharVarcharTypeInfo` keep first-class types even if the legacy
flag is also on.
+ */
+ def shouldApplyWriteSideLengthCheck(conf: SQLConf): Boolean = {
+ !conf.charVarcharAsString || conf.charVarcharFirstClassTypes
+ }
+
def stringLengthCheck(expr: Expression, dt: DataType): Expression = {
processStringForCharVarchar(
expr,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
index 68529e41937e..b9c0a0255f31 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
@@ -439,7 +439,8 @@ object ResolveDefaultColumns extends QueryErrorsBase
throw QueryCompilationErrors.defaultValuesDataTypeError(
statementType, colName, defaultSQL, dataType, other.dataType))
}
- if (!conf.charVarcharAsString && CharVarcharUtils.hasCharVarchar(dataType)
&& ret.foldable) {
+ if (CharVarcharUtils.hasCharVarchar(dataType) &&
+ CharVarcharUtils.shouldApplyWriteSideLengthCheck(conf) &&
ret.foldable) {
CharVarcharUtils.stringLengthCheck(ret, dataType).eval(EmptyRow)
}
ret
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
index 81fb27a17e34..85688308aa2e 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
@@ -55,6 +55,71 @@ Project [try_cast(abcdef as varchar(2)) AS TRY_CAST(abcdef
AS VARCHAR(2))#x]
+- OneRowRelation
+-- !query
+SELECT CAST(12345 AS VARCHAR(4))
+-- !query analysis
+Project [cast(12345 as varchar(4)) AS CAST(12345 AS VARCHAR(4))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT CAST(12345 AS VARCHAR(5))
+-- !query analysis
+Project [cast(12345 as varchar(5)) AS CAST(12345 AS VARCHAR(5))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT try_cast(12345 AS VARCHAR(4))
+-- !query analysis
+Project [try_cast(12345 as varchar(4)) AS TRY_CAST(12345 AS VARCHAR(4))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT coalesce(CAST('abcdef' AS VARCHAR(2)), CAST('x' AS VARCHAR(4)))
+-- !query analysis
+Project [coalesce(cast(cast(abcdef as varchar(2)) as varchar(4)), cast(x as
varchar(4))) AS coalesce(CAST(abcdef AS VARCHAR(2)), CAST(x AS VARCHAR(4)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT CASE WHEN true THEN CAST('abcdef' AS VARCHAR(2)) ELSE CAST('x' AS
VARCHAR(4)) END
+-- !query analysis
+Project [CASE WHEN true THEN cast(cast(abcdef as varchar(2)) as varchar(4))
ELSE cast(x as varchar(4)) END AS CASE WHEN true THEN CAST(abcdef AS
VARCHAR(2)) ELSE CAST(x AS VARCHAR(4)) END#x]
++- OneRowRelation
+
+
+-- !query
+SELECT CAST('abcdef' AS VARCHAR(2)) IN (CAST('ab' AS VARCHAR(4)))
+-- !query analysis
+Project [cast(cast(abcdef as varchar(2)) as varchar(4)) IN (cast(cast(ab as
varchar(4)) as varchar(4))) AS (CAST(abcdef AS VARCHAR(2)) IN (CAST(ab AS
VARCHAR(4))))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT coalesce(
+ CAST('abcdef' AS VARCHAR(2) COLLATE UTF8_LCASE),
+ CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE))
+-- !query analysis
+Project [coalesce(cast(cast(abcdef as varchar(2) collate UTF8_LCASE) as
varchar(4) collate UTF8_LCASE), cast(x as varchar(4) collate UTF8_LCASE)) AS
coalesce(CAST(abcdef AS VARCHAR(2) COLLATE UTF8_LCASE), CAST(x AS VARCHAR(4)
COLLATE UTF8_LCASE))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT coalesce(try_cast(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)))
+-- !query analysis
+Project [coalesce(cast(try_cast(12345 as varchar(4)) as varchar(5)), cast(x as
varchar(5))) AS coalesce(TRY_CAST(12345 AS VARCHAR(4)), CAST(x AS
VARCHAR(5)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT coalesce(CAST(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)))
+-- !query analysis
+Project [coalesce(cast(cast(12345 as varchar(4)) as varchar(5)), cast(x as
varchar(5))) AS coalesce(CAST(12345 AS VARCHAR(4)), CAST(x AS VARCHAR(5)))#x]
++- OneRowRelation
+
+
-- !query
SELECT typeof(coalesce(cast('hello' AS VARCHAR(5)), cast('world' AS
VARCHAR(10))))
-- !query analysis
@@ -322,6 +387,48 @@ Project [typeof(coalesce(cast(a as char(2) collate
UTF8_LCASE), cast(bb as char(
+- OneRowRelation
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)))
+-- !query analysis
+Project [typeof(coalesce(cast(cast(a as char(2) collate UTF8_LCASE) as char(4)
collate UTF8_LCASE), cast(bb as char(4) collate UTF8_LCASE))) AS
typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(bb AS CHAR(4)
COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)))
+-- !query analysis
+Project [hex(cast(coalesce(cast(cast(a as char(2) collate UTF8_LCASE) as
char(4) collate UTF8_LCASE), cast(bb as char(4) collate UTF8_LCASE)) as string
collate UTF8_LCASE)) AS hex(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE),
CAST(bb AS CHAR(4) COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS VARCHAR(4) COLLATE
UTF8_LCASE)))
+-- !query analysis
+Project [typeof(coalesce(cast(cast(a as char(2) collate UTF8_LCASE) as
varchar(4) collate UTF8_LCASE), cast(bb as varchar(4) collate UTF8_LCASE))) AS
typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(bb AS VARCHAR(4)
COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)))
+-- !query analysis
+Project [typeof(coalesce(cast(cast(a as char(2) collate UTF8_LCASE) as char(4)
collate UTF8_LCASE), cast(1 as char(4) collate UTF8_LCASE))) AS
typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(1 AS CHAR(4)
COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)))
+-- !query analysis
+Project [hex(cast(coalesce(cast(cast(a as char(2) collate UTF8_LCASE) as
char(4) collate UTF8_LCASE), cast(1 as char(4) collate UTF8_LCASE)) as string
collate UTF8_LCASE)) AS hex(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE),
CAST(1 AS CHAR(4) COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
-- !query
SELECT typeof(c) FROM (
SELECT cast('a' AS VARCHAR(3)) AS c
@@ -341,6 +448,276 @@ GlobalLimit 1
+- OneRowRelation
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t LIMIT 1
+-- !query analysis
+GlobalLimit 1
++- LocalLimit 1
+ +- Project [typeof(c#x) AS typeof(c)#x]
+ +- SubqueryAlias t
+ +- Union false, false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(a as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(bb as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [concat(<, cast(c#x as string), >) AS concat(<, c, >)#x]
++- SubqueryAlias t
+ +- Union false, false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(a as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(bb as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [typeof(c#x) AS typeof(c)#x]
++- SubqueryAlias t
+ +- Distinct
+ +- Union false, false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(a as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(a as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [concat(<, cast(c#x as string), >) AS concat(<, c, >)#x]
++- SubqueryAlias t
+ +- Distinct
+ +- Union false, false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(a as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(a as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [typeof(c#x) AS typeof(c)#x]
++- SubqueryAlias t
+ +- Intersect false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(ab as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(ab as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [concat(<, cast(c#x as string), >) AS concat(<, c, >)#x]
++- SubqueryAlias t
+ +- Intersect false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(ab as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(ab as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [typeof(c#x) AS typeof(c)#x]
++- SubqueryAlias t
+ +- Except false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(ab as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(xy as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t
+-- !query analysis
+Project [concat(<, cast(c#x as string), >) AS concat(<, c, >)#x]
++- SubqueryAlias t
+ +- Except false
+ :- Project [cast(c#x as char(4)) AS c#x]
+ : +- Project [cast(ab as char(2)) AS c#x]
+ : +- OneRowRelation
+ +- Project [cast(xy as char(4)) AS c#x]
+ +- OneRowRelation
+
+
+-- !query
+SELECT typeof(c) FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c)
+-- !query analysis
+Project [typeof(c#x) AS typeof(c)#x]
++- SubqueryAlias t
+ +- Project [col1#x AS c#x]
+ +- LocalRelation [col1#x]
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c)
+-- !query analysis
+Project [concat(<, cast(c#x as string), >) AS concat(<, c, >)#x]
++- SubqueryAlias t
+ +- Project [col1#x AS c#x]
+ +- LocalRelation [col1#x]
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a' AS CHAR(4))
+-- !query analysis
+Project [(cast(cast(a as char(2)) as char(4)) = cast(a as char(4))) AS (CAST(a
AS CHAR(2)) = CAST(a AS CHAR(4)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a' AS VARCHAR(2))
+-- !query analysis
+Project [(cast(cast(a as char(2)) as varchar(2)) = cast(a as varchar(2))) AS
(CAST(a AS CHAR(2)) = CAST(a AS VARCHAR(2)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a ' AS VARCHAR(2))
+-- !query analysis
+Project [(cast(cast(a as char(2)) as varchar(2)) = cast(a as varchar(2))) AS
(CAST(a AS CHAR(2)) = CAST(a AS VARCHAR(2)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = 'a'
+-- !query analysis
+Project [(cast(cast(a as char(2)) as string collate UTF8_BINARY) = a) AS
(CAST(a AS CHAR(2)) = a)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = 'a '
+-- !query analysis
+Project [(cast(cast(a as char(2)) as string collate UTF8_BINARY) = a ) AS
(CAST(a AS CHAR(2)) = a )#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = 'a'
+-- !query analysis
+Project [(cast(a as char(2) collate UTF8_BINARY_RTRIM) = cast(a as char(2)
collate UTF8_BINARY_RTRIM)) AS (CAST(a AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) =
a)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) =
+ cast('a' AS CHAR(4) COLLATE UTF8_BINARY_RTRIM)
+-- !query analysis
+Project [(cast(cast(a as char(2) collate UTF8_BINARY_RTRIM) as char(4) collate
UTF8_BINARY_RTRIM) = cast(a as char(4) collate UTF8_BINARY_RTRIM)) AS (CAST(a
AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = CAST(a AS CHAR(4) COLLATE
UTF8_BINARY_RTRIM))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)))
+-- !query analysis
+Project [cast(cast(a as char(2)) as char(4)) IN (cast(cast(a as char(4)) as
char(4))) AS (CAST(a AS CHAR(2)) IN (CAST(a AS CHAR(4))))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS VARCHAR(2)))
+-- !query analysis
+Project [cast(cast(a as char(2)) as varchar(2)) IN (cast(cast(a as varchar(2))
as varchar(2))) AS (CAST(a AS CHAR(2)) IN (CAST(a AS VARCHAR(2))))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a ' AS VARCHAR(2)))
+-- !query analysis
+Project [cast(cast(a as char(2)) as varchar(2)) IN (cast(cast(a as
varchar(2)) as varchar(2))) AS (CAST(a AS CHAR(2)) IN (CAST(a AS
VARCHAR(2))))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN ('a', 'b')
+-- !query analysis
+Project [cast(cast(a as char(2)) as string collate UTF8_BINARY) IN (cast(a as
string collate UTF8_BINARY),cast(b as string collate UTF8_BINARY)) AS (CAST(a
AS CHAR(2)) IN (a, b))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN ('a ', 'b')
+-- !query analysis
+Project [cast(cast(a as char(2)) as string collate UTF8_BINARY) IN (cast(a as
string collate UTF8_BINARY),cast(b as string collate UTF8_BINARY)) AS (CAST(a
AS CHAR(2)) IN (a , b))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)), cast('b' AS VARCHAR(3)))
+-- !query analysis
+Project [cast(cast(a as char(2)) as varchar(4)) IN (cast(cast(a as char(4)) as
varchar(4)),cast(cast(b as varchar(3)) as varchar(4))) AS (CAST(a AS CHAR(2))
IN (CAST(a AS CHAR(4)), CAST(b AS VARCHAR(3))))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) = cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE)
+-- !query analysis
+Project [(cast(cast(a as char(2) collate UTF8_LCASE) as varchar(2) collate
UTF8_LCASE) = cast(a as varchar(2) collate UTF8_LCASE)) AS (CAST(a AS CHAR(2)
COLLATE UTF8_LCASE) = CAST(a AS VARCHAR(2) COLLATE UTF8_LCASE))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) IN (cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE))
+-- !query analysis
+Project [cast(cast(a as char(2) collate UTF8_LCASE) as varchar(2) collate
UTF8_LCASE) IN (cast(a as varchar(2) collate UTF8_LCASE)) AS (CAST(a AS CHAR(2)
COLLATE UTF8_LCASE) IN (CAST(a AS VARCHAR(2) COLLATE UTF8_LCASE)))#x]
++- OneRowRelation
+
+
-- !query
SELECT typeof(array(cast('a' AS CHAR(2)), cast('bb' AS CHAR(3))))
-- !query analysis
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
b/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
index 6d591bc7e8cf..25a52737bcb2 100644
---
a/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
+++
b/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
@@ -5,12 +5,25 @@ SELECT typeof(CAST('ab' AS CHAR(5)));
SELECT typeof(CAST('hello' AS VARCHAR(5)));
SELECT 'X' || CAST('5' AS CHAR(5)) || 'X';
--- CAST length enforcement: trailing spaces are trimmed, real overflow errors
+-- CAST length: character-to-character truncates (ISO 6.13); numeric overflow
errors
SELECT CAST('ab ' AS CHAR(2));
SELECT CAST('abcdef' AS CHAR(2));
SELECT CAST('abcdef' AS VARCHAR(2));
SELECT try_cast('abcdef' AS CHAR(2));
SELECT try_cast('abcdef' AS VARCHAR(2));
+SELECT CAST(12345 AS VARCHAR(4));
+SELECT CAST(12345 AS VARCHAR(5));
+SELECT try_cast(12345 AS VARCHAR(4));
+
+-- Explicit CAST inside LCT must keep inner truncation / overflow (do not
retarget).
+SELECT coalesce(CAST('abcdef' AS VARCHAR(2)), CAST('x' AS VARCHAR(4)));
+SELECT CASE WHEN true THEN CAST('abcdef' AS VARCHAR(2)) ELSE CAST('x' AS
VARCHAR(4)) END;
+SELECT CAST('abcdef' AS VARCHAR(2)) IN (CAST('ab' AS VARCHAR(4)));
+SELECT coalesce(
+ CAST('abcdef' AS VARCHAR(2) COLLATE UTF8_LCASE),
+ CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE));
+SELECT coalesce(try_cast(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)));
+SELECT coalesce(CAST(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)));
-- R2: least common type (COALESCE / CASE)
SELECT typeof(coalesce(cast('hello' AS VARCHAR(5)), cast('world' AS
VARCHAR(10))));
@@ -66,22 +79,108 @@ SELECT typeof(reverse(array(1, 2)));
SELECT typeof(str_to_map(cast('a:1,b:2' AS CHAR(7))));
SELECT typeof(c0) FROM (SELECT json_tuple(cast('{"a":"1"}' AS CHAR(9)), 'a')
AS c0);
--- R2 with collation. A declared collation survives the CAST and an LCT over
equally constrained
--- operands. The mixed-length case (CHAR(2) with CHAR(4), same collation) is
deliberately not
--- covered here: CollationTypeCoercion reads the differing lengths as a
collation mismatch and
--- yields an indeterminate collation. That predates this change (it reproduces
under
--- spark.sql.preserveCharVarcharTypeInfo) and is tracked separately, so
goldening it would
--- normalize the bug.
+-- Collation survives CAST and LCT. Mixed lengths with the same collation
widen to max(n, m);
+-- they must not collapse to an indeterminate collation.
SELECT typeof(cast('a' AS CHAR(2) COLLATE UTF8_LCASE));
SELECT typeof(coalesce(
cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(2) COLLATE
UTF8_LCASE)));
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)));
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)));
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS VARCHAR(4) COLLATE
UTF8_LCASE)));
+-- Mixed strength, same collation: Implicit string CAST CHAR(2) vs Default
+-- non-string CAST CHAR(4). Length still widens to max(n, m); the COLLATE
+-- operator itself is STRING, so it is not used here.
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)));
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)));
--- UNION LCT
+-- Set operations and multi-row VALUES share the same LCT as COALESCE.
SELECT typeof(c) FROM (
SELECT cast('a' AS VARCHAR(3)) AS c
UNION ALL
SELECT cast('abcd' AS VARCHAR(8)) AS c
) t LIMIT 1;
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t LIMIT 1;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t;
+-- Non-empty EXCEPT: after widening, 'ab ' is not 'xy '.
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c);
+SELECT concat('<', c, '>') FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c);
+
+-- Comparison and IN: both sides (including the IN left-hand side) are cast to
the LCT of all
+-- participants. Casting to CHAR pads, so CHAR vs CHAR of different lengths
compares equal after
+-- widening; casting to VARCHAR/STRING keeps the CHAR pad, so CHAR 'a' (stored
as 'a ') is not equal
+-- to VARCHAR/STRING 'a' unless the other side carries the same trailing
blank. Trailing-blank
+-- ignoring is a collation concern (RTRIM), not a type-level PAD SPACE policy.
+SELECT cast('a' AS CHAR(2)) = cast('a' AS CHAR(4));
+SELECT cast('a' AS CHAR(2)) = cast('a' AS VARCHAR(2));
+SELECT cast('a' AS CHAR(2)) = cast('a ' AS VARCHAR(2));
+SELECT cast('a' AS CHAR(2)) = 'a';
+SELECT cast('a' AS CHAR(2)) = 'a ';
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = 'a';
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) =
+ cast('a' AS CHAR(4) COLLATE UTF8_BINARY_RTRIM);
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)));
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS VARCHAR(2)));
+SELECT cast('a' AS CHAR(2)) IN (cast('a ' AS VARCHAR(2)));
+SELECT cast('a' AS CHAR(2)) IN ('a', 'b');
+SELECT cast('a' AS CHAR(2)) IN ('a ', 'b');
+-- Three-part IN: LHS CHAR(2) and list CHAR(4)/VARCHAR(3) all widen to
VARCHAR(4).
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)), cast('b' AS VARCHAR(3)));
+-- Same CHAR-pad rule under a non-RTRIM collation: UTF8_LCASE must not make
+-- CHAR 'a' equal VARCHAR 'a'. The analyzer must nest CHAR then VARCHAR, not
+-- retarget CAST('a' AS CHAR(2) COLLATE UTF8_LCASE) to VARCHAR(2).
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) = cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE);
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) IN (cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE));
-- Nested types keep CHAR/VARCHAR
SELECT typeof(array(cast('a' AS CHAR(2)), cast('bb' AS CHAR(3))));
diff --git
a/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
b/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
index 1830a80cac51..168b5ef173be 100644
---
a/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
@@ -34,21 +34,38 @@ ab
-- !query
SELECT CAST('abcdef' AS CHAR(2))
-- !query schema
-struct<>
+struct<CAST(abcdef AS CHAR(2)):char(2)>
-- !query output
-org.apache.spark.SparkRuntimeException
-{
- "errorClass" : "EXCEED_LIMIT_LENGTH",
- "sqlState" : "54006",
- "messageParameters" : {
- "limit" : "2"
- }
-}
+ab
-- !query
SELECT CAST('abcdef' AS VARCHAR(2))
-- !query schema
+struct<CAST(abcdef AS VARCHAR(2)):varchar(2)>
+-- !query output
+ab
+
+
+-- !query
+SELECT try_cast('abcdef' AS CHAR(2))
+-- !query schema
+struct<TRY_CAST(abcdef AS CHAR(2)):char(2)>
+-- !query output
+ab
+
+
+-- !query
+SELECT try_cast('abcdef' AS VARCHAR(2))
+-- !query schema
+struct<TRY_CAST(abcdef AS VARCHAR(2)):varchar(2)>
+-- !query output
+ab
+
+
+-- !query
+SELECT CAST(12345 AS VARCHAR(4))
+-- !query schema
struct<>
-- !query output
org.apache.spark.SparkRuntimeException
@@ -56,27 +73,84 @@ org.apache.spark.SparkRuntimeException
"errorClass" : "EXCEED_LIMIT_LENGTH",
"sqlState" : "54006",
"messageParameters" : {
- "limit" : "2"
+ "limit" : "4"
}
}
-- !query
-SELECT try_cast('abcdef' AS CHAR(2))
+SELECT CAST(12345 AS VARCHAR(5))
-- !query schema
-struct<TRY_CAST(abcdef AS CHAR(2)):char(2)>
+struct<CAST(12345 AS VARCHAR(5)):varchar(5)>
-- !query output
-NULL
+12345
-- !query
-SELECT try_cast('abcdef' AS VARCHAR(2))
+SELECT try_cast(12345 AS VARCHAR(4))
-- !query schema
-struct<TRY_CAST(abcdef AS VARCHAR(2)):varchar(2)>
+struct<TRY_CAST(12345 AS VARCHAR(4)):varchar(4)>
-- !query output
NULL
+-- !query
+SELECT coalesce(CAST('abcdef' AS VARCHAR(2)), CAST('x' AS VARCHAR(4)))
+-- !query schema
+struct<coalesce(CAST(abcdef AS VARCHAR(2)), CAST(x AS VARCHAR(4))):varchar(4)>
+-- !query output
+ab
+
+
+-- !query
+SELECT CASE WHEN true THEN CAST('abcdef' AS VARCHAR(2)) ELSE CAST('x' AS
VARCHAR(4)) END
+-- !query schema
+struct<CASE WHEN true THEN CAST(abcdef AS VARCHAR(2)) ELSE CAST(x AS
VARCHAR(4)) END:varchar(4)>
+-- !query output
+ab
+
+
+-- !query
+SELECT CAST('abcdef' AS VARCHAR(2)) IN (CAST('ab' AS VARCHAR(4)))
+-- !query schema
+struct<(CAST(abcdef AS VARCHAR(2)) IN (CAST(ab AS VARCHAR(4)))):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT coalesce(
+ CAST('abcdef' AS VARCHAR(2) COLLATE UTF8_LCASE),
+ CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE))
+-- !query schema
+struct<coalesce(CAST(abcdef AS VARCHAR(2) COLLATE UTF8_LCASE), CAST(x AS
VARCHAR(4) COLLATE UTF8_LCASE)):varchar(4) collate UTF8_LCASE>
+-- !query output
+ab
+
+
+-- !query
+SELECT coalesce(try_cast(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)))
+-- !query schema
+struct<coalesce(TRY_CAST(12345 AS VARCHAR(4)), CAST(x AS
VARCHAR(5))):varchar(5)>
+-- !query output
+x
+
+
+-- !query
+SELECT coalesce(CAST(12345 AS VARCHAR(4)), CAST('x' AS VARCHAR(5)))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "EXCEED_LIMIT_LENGTH",
+ "sqlState" : "54006",
+ "messageParameters" : {
+ "limit" : "4"
+ }
+}
+
+
-- !query
SELECT typeof(coalesce(cast('hello' AS VARCHAR(5)), cast('world' AS
VARCHAR(10))))
-- !query schema
@@ -375,6 +449,53 @@ struct<typeof(coalesce(CAST(a AS CHAR(2) COLLATE
UTF8_LCASE), CAST(bb AS CHAR(2)
char(2) collate UTF8_LCASE
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)))
+-- !query schema
+struct<typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(bb AS
CHAR(4) COLLATE UTF8_LCASE))):string>
+-- !query output
+char(4) collate UTF8_LCASE
+
+
+-- !query
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)))
+-- !query schema
+struct<hex(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(bb AS CHAR(4)
COLLATE UTF8_LCASE))):string collate UTF8_LCASE>
+-- !query output
+61202020
+
+
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS VARCHAR(4) COLLATE
UTF8_LCASE)))
+-- !query schema
+struct<typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(bb AS
VARCHAR(4) COLLATE UTF8_LCASE))):string>
+-- !query output
+varchar(4) collate UTF8_LCASE
+
+
+-- !query
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)))
+-- !query schema
+struct<typeof(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(1 AS
CHAR(4) COLLATE UTF8_LCASE))):string>
+-- !query output
+char(4) collate UTF8_LCASE
+
+
+-- !query
+SELECT hex(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)))
+-- !query schema
+struct<hex(coalesce(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(1 AS CHAR(4)
COLLATE UTF8_LCASE))):string collate UTF8_LCASE>
+-- !query output
+61202020
+
+
-- !query
SELECT typeof(c) FROM (
SELECT cast('a' AS VARCHAR(3)) AS c
@@ -387,6 +508,248 @@ struct<typeof(c):string>
varchar(8)
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t LIMIT 1
+-- !query schema
+struct<typeof(c):string>
+-- !query output
+char(4)
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<concat(<, c, >):string>
+-- !query output
+<a >
+<bb >
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<typeof(c):string>
+-- !query output
+char(4)
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<concat(<, c, >):string>
+-- !query output
+<a >
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<typeof(c):string>
+-- !query output
+char(4)
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<concat(<, c, >):string>
+-- !query output
+<ab >
+
+
+-- !query
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<typeof(c):string>
+-- !query output
+char(4)
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t
+-- !query schema
+struct<concat(<, c, >):string>
+-- !query output
+<ab >
+
+
+-- !query
+SELECT typeof(c) FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c)
+-- !query schema
+struct<typeof(c):string>
+-- !query output
+char(4)
+char(4)
+
+
+-- !query
+SELECT concat('<', c, '>') FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c)
+-- !query schema
+struct<concat(<, c, >):string>
+-- !query output
+<a >
+<bb >
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a' AS CHAR(4))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) = CAST(a AS CHAR(4))):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a' AS VARCHAR(2))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) = CAST(a AS VARCHAR(2))):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = cast('a ' AS VARCHAR(2))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) = CAST(a AS VARCHAR(2))):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = 'a'
+-- !query schema
+struct<(CAST(a AS CHAR(2)) = a):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) = 'a '
+-- !query schema
+struct<(CAST(a AS CHAR(2)) = a ):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = 'a'
+-- !query schema
+struct<(CAST(a AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = a):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) =
+ cast('a' AS CHAR(4) COLLATE UTF8_BINARY_RTRIM)
+-- !query schema
+struct<(CAST(a AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = CAST(a AS CHAR(4)
COLLATE UTF8_BINARY_RTRIM)):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (CAST(a AS CHAR(4)))):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS VARCHAR(2)))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (CAST(a AS VARCHAR(2)))):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a ' AS VARCHAR(2)))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (CAST(a AS VARCHAR(2)))):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN ('a', 'b')
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (a, b)):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN ('a ', 'b')
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (a , b)):boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)), cast('b' AS VARCHAR(3)))
+-- !query schema
+struct<(CAST(a AS CHAR(2)) IN (CAST(a AS CHAR(4)), CAST(b AS
VARCHAR(3)))):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) = cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE)
+-- !query schema
+struct<(CAST(a AS CHAR(2) COLLATE UTF8_LCASE) = CAST(a AS VARCHAR(2) COLLATE
UTF8_LCASE)):boolean>
+-- !query output
+false
+
+
+-- !query
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) IN (cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE))
+-- !query schema
+struct<(CAST(a AS CHAR(2) COLLATE UTF8_LCASE) IN (CAST(a AS VARCHAR(2) COLLATE
UTF8_LCASE))):boolean>
+-- !query output
+false
+
+
-- !query
SELECT typeof(array(cast('a' AS CHAR(2)), cast('bb' AS CHAR(3))))
-- !query schema
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
index 34e268d3ba79..f2905da9a2d8 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
@@ -21,9 +21,9 @@ import scala.util.Try
import org.apache.spark.{SparkConf, SparkException, SparkRuntimeException}
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
-import org.apache.spark.sql.catalyst.expressions.{Attribute, EqualTo,
GreaterThan, ScalarSubquery, StringRPad}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, EqualTo,
GreaterThan, Literal, ScalarSubquery, StringRPad}
import org.apache.spark.sql.catalyst.expressions.Cast.toSQLId
-import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Project}
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.connector.SchemaRequiredDataSource
@@ -868,25 +868,80 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
assert(varcharDf.schema.head.dataType === VarcharType(5))
checkAnswer(varcharDf, Row("hello"))
+ // ISO 6.13: character-to-character CAST truncates rather than erroring.
+ checkAnswer(sql("SELECT CAST('hello!' AS VARCHAR(5)) AS v"),
Row("hello"))
+ checkAnswer(sql("SELECT CAST('abcdef' AS CHAR(2)) AS c"), Row("ab"))
+ checkAnswer(sql("SELECT CAST('abcdef' AS VARCHAR(2)) AS v"), Row("ab"))
+ checkAnswer(sql("SELECT try_cast('abcdef' AS CHAR(2)) AS c"), Row("ab"))
+ checkAnswer(sql("SELECT try_cast('abcdef' AS VARCHAR(2)) AS v"),
Row("ab"))
+
+ // Multi-byte characters: length is in characters, not octets.
+ // scalastyle:off nonascii
+ checkAnswer(sql("SELECT CAST('你好' AS VARCHAR(2)) AS v"), Row("你好"))
+ checkAnswer(sql("SELECT CAST('你好啊' AS VARCHAR(2)) AS v"), Row("你好"))
+ // scalastyle:on nonascii
+
+ // ISO 6.13 numeric-to-character CAST still errors when the literal does
not fit.
checkError(
exception = intercept[SparkRuntimeException] {
- sql("SELECT CAST('hello!' AS VARCHAR(5))").collect()
+ sql("SELECT CAST(12345 AS VARCHAR(4))").collect()
},
condition = "EXCEED_LIMIT_LENGTH",
- parameters = Map("limit" -> "5")
+ parameters = Map("limit" -> "4")
)
+ checkAnswer(sql("SELECT CAST(12345 AS VARCHAR(5)) AS v"), Row("12345"))
+ checkAnswer(sql("SELECT try_cast(12345 AS VARCHAR(4)) AS v"), Row(null))
- // Multi-byte characters: length is in characters, not octets.
- // scalastyle:off nonascii
- checkAnswer(sql("SELECT CAST('你好' AS VARCHAR(2)) AS v"), Row("你好"))
+ // LCT must wrap the inner CAST, not retarget it (truncation / overflow
stay).
+ checkAnswer(
+ sql("SELECT coalesce(CAST('abcdef' AS VARCHAR(2)), CAST('x' AS
VARCHAR(4))) AS c"),
+ Row("ab"))
+ checkAnswer(
+ sql("""SELECT CASE WHEN true THEN CAST('abcdef' AS VARCHAR(2))
+ |ELSE CAST('x' AS VARCHAR(4)) END AS c""".stripMargin),
+ Row("ab"))
+ checkAnswer(
+ sql("SELECT CAST('abcdef' AS VARCHAR(2)) IN (CAST('ab' AS
VARCHAR(4)))"),
+ Row(true))
+ checkAnswer(
+ sql("""SELECT coalesce(
+ | CAST('abcdef' AS VARCHAR(2) COLLATE UTF8_LCASE),
+ | CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin),
+ Row("ab"))
+ checkAnswer(
+ sql("SELECT coalesce(try_cast(12345 AS VARCHAR(4)), CAST('x' AS
VARCHAR(5))) AS c"),
+ Row("x"))
checkError(
exception = intercept[SparkRuntimeException] {
- sql("SELECT CAST('你好啊' AS VARCHAR(2))").collect()
+ sql("SELECT coalesce(CAST(12345 AS VARCHAR(4)), CAST('x' AS
VARCHAR(5)))").collect()
},
condition = "EXCEED_LIMIT_LENGTH",
- parameters = Map("limit" -> "2")
+ parameters = Map("limit" -> "4")
)
- // scalastyle:on nonascii
+ }
+ }
+
+ test("SPARK-58797: store assignment with standardSemantics and
charVarcharAsString") {
+ withSQLConf(
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true",
+ SQLConf.LEGACY_CHAR_VARCHAR_AS_STRING.key -> "true") {
+ val wide = new StructType().add("c", CharType(10))
+ val df = spark.createDataFrame(java.util.Arrays.asList(Row("spark")),
wide)
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ df.to(new StructType().add("c", CharType(3))).collect()
+ },
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "3"))
+ withTable("std_and_as_string") {
+ sql("CREATE TABLE std_and_as_string (v VARCHAR(2)) USING parquet")
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ sql("INSERT INTO std_and_as_string VALUES ('abc')")
+ },
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "2"))
+ }
}
}
@@ -1201,6 +1256,91 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
}
}
+ test("SPARK-58794: collated mixed-length LCT ignores collation strength for
length") {
+ val mixedLength =
+ """SELECT coalesce(
+ | cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | cast('bb' AS CHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin
+ val mixedStrength =
+ """SELECT coalesce(
+ | cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | cast(1 AS CHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin
+
+ Seq(
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true",
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true").foreach { case
(key, value) =>
+ withSQLConf(key -> value) {
+ assert(sql(mixedLength).schema.head.dataType === CharType(4,
"UTF8_LCASE"),
+ s"$key=$value same-strength mixed CHAR lengths")
+ assert(sql(mixedStrength).schema.head.dataType === CharType(4,
"UTF8_LCASE"),
+ s"$key=$value Implicit CHAR(2) vs Default CHAR(4) must widen, not
narrow")
+ }
+ }
+ }
+
+ test("SPARK-58794: typed CHAR Literal is re-padded when LCT widens the
length") {
+ // CollationTypeCoercion.changeType used to `copy(dataType)` on Literal,
which would
+ // leave CHAR(2) "a " as a CHAR(4) value without the extra pad. SQL CAST
is a Cast
+ // node so goldens do not cover this; Literal.create does.
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val c2 = Column(Literal.create("a", CharType(2, "UTF8_LCASE")))
+ val c4 = Column(Literal.create("bb", CharType(4, "UTF8_LCASE")))
+ val coalesced = functions.coalesce(c2, c4)
+ val df = spark.range(1).select(coalesced.as("c"))
+ assert(df.schema.head.dataType === CharType(4, "UTF8_LCASE"))
+ checkAnswer(df, Row("a "))
+ }
+ }
+
+ test("SPARK-58794: parameterized CHAR/VARCHAR lengths under
standardSemantics") {
+ // Length positions accept parameter markers (`integerValue` ->
`parameterMarker`). Under
+ // standardSemantics the bound type stays first-class: CAST keeps
CHAR/VARCHAR, pads and
+ // enforces length, and DDL schemas retain the substituted n.
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val charDf = spark.sql("SELECT cast('ab' AS CHAR(:n)) AS c", Map("n" ->
5))
+ assert(charDf.schema.head.dataType === CharType(5))
+ checkAnswer(
+ spark.sql("SELECT concat('<', cast('ab' AS CHAR(:n)), '>')", Map("n"
-> 5)),
+ Row("<ab >"))
+
+ val varcharDf = spark.sql("SELECT cast('hello' AS VARCHAR(?)) AS c",
Array(5))
+ assert(varcharDf.schema.head.dataType === VarcharType(5))
+ checkAnswer(
+ spark.sql("SELECT cast('abcdef' AS VARCHAR(?))", Array(2)),
+ Row("ab"))
+
+ withTable("param_varchar", "param_char") {
+ spark.sql(
+ "CREATE TABLE param_varchar (c VARCHAR(:n)) USING parquet", Map("n"
-> 7))
+ assert(spark.table("param_varchar").schema.head.dataType ===
VarcharType(7))
+ spark.sql("CREATE TABLE param_char (c CHAR(?)) USING parquet",
Array(4))
+ assert(spark.table("param_char").schema.head.dataType === CharType(4))
+ }
+
+ // Non-integral / negative lengths fail when substituted into the length
position.
+ checkError(
+ exception = intercept[ParseException] {
+ spark.sql("SELECT cast('a' AS CHAR(:n))", Map("n" -> -1))
+ },
+ condition = "PARSE_SYNTAX_ERROR",
+ parameters = Map("error" -> "'-'", "hint" -> ""),
+ context = ExpectedContext(
+ fragment = "SELECT cast('a' AS CHAR(:n))",
+ start = 0,
+ stop = 27))
+ checkError(
+ exception = intercept[ParseException] {
+ spark.sql("SELECT cast('a' AS CHAR(:n))", Map("n" -> 1.5))
+ },
+ condition = "PARSE_SYNTAX_ERROR",
+ parameters = Map("error" -> "'1.5D'", "hint" -> ""),
+ context = ExpectedContext(
+ fragment = "SELECT cast('a' AS CHAR(:n))",
+ start = 0,
+ stop = 27))
+ }
+ }
+
test("SPARK-58802: single-pass resolver agrees with fixed-point under
standardSemantics") {
// Dual run defaults to on under tests, but pin it explicitly so this
coverage cannot be
// silently lost: the HybridAnalyzer compares output schema and normalized
plan across the
@@ -1219,7 +1359,18 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
VarcharType(5))
assert(sql("SELECT try_cast('abcdef' AS CHAR(2)) AS
c").schema.head.dataType ===
CharType(2))
- checkAnswer(sql("SELECT try_cast('abcdef' AS VARCHAR(2)) AS c"),
Row(null))
+ checkAnswer(sql("SELECT try_cast('abcdef' AS VARCHAR(2)) AS c"),
Row("ab"))
+ checkAnswer(
+ sql("SELECT coalesce(CAST('abcdef' AS VARCHAR(2)), CAST('x' AS
VARCHAR(4))) AS c"),
+ Row("ab"))
+ checkAnswer(
+ sql("SELECT CAST('abcdef' AS VARCHAR(2)) IN (CAST('ab' AS
VARCHAR(4)))"),
+ Row(true))
+ checkAnswer(
+ sql("""SELECT coalesce(
+ | CAST('abcdef' AS VARCHAR(2) COLLATE UTF8_LCASE),
+ | CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin),
+ Row("ab"))
// Least common type (R2): COALESCE / CASE / NULL / CHAR+VARCHAR /
CHAR+STRING.
assert(sql(
@@ -1289,9 +1440,41 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
.schema.head.dataType ===
StructType(Seq(StructField("f", CharType(2), nullable = false))))
- // Collated CAST keeps the declared collation under both analyzers.
- assert(sql("SELECT CAST('ab' AS CHAR(2) COLLATE UTF8_LCASE) AS c")
- .schema.head.dataType === CharType(2, "UTF8_LCASE"))
+ // Collated mixed-length LCT (the CollationTypeCoercion equal-strength
and
+ // mixed-strength paths). Set ops have a separate resolver path.
+ assert(sql(
+ """SELECT coalesce(
+ | CAST('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | CAST('bb' AS CHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin)
+ .schema.head.dataType === CharType(4, "UTF8_LCASE"))
+ assert(sql(
+ """SELECT coalesce(
+ | CAST('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | CAST(1 AS CHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin)
+ .schema.head.dataType === CharType(4, "UTF8_LCASE"))
+ checkAnswer(
+ sql("SELECT CAST('a' AS CHAR(2) COLLATE UTF8_LCASE) = " +
+ "CAST('a' AS CHAR(4) COLLATE UTF8_LCASE)"),
+ Row(true))
+ checkAnswer(
+ sql("SELECT CAST('a' AS CHAR(2) COLLATE UTF8_LCASE) IN " +
+ "(CAST('a' AS CHAR(4) COLLATE UTF8_LCASE))"),
+ Row(true))
+ checkAnswer(
+ sql("SELECT CAST('a' AS CHAR(2) COLLATE UTF8_LCASE) = " +
+ "CAST('a' AS VARCHAR(2) COLLATE UTF8_LCASE)"),
+ Row(false))
+ checkAnswer(
+ sql("SELECT CAST('a' AS CHAR(2) COLLATE UTF8_LCASE) IN " +
+ "(CAST('a' AS VARCHAR(2) COLLATE UTF8_LCASE))"),
+ Row(false))
+
+ val mixedCharUnion = sql(
+ """SELECT CAST('a' AS CHAR(2)) AS c
+ |UNION ALL
+ |SELECT CAST('bb' AS CHAR(4)) AS c""".stripMargin)
+ assert(mixedCharUnion.schema.head.dataType === CharType(4))
+ checkAnswer(mixedCharUnion, Seq(Row("a "), Row("bb ")))
// Bare column references keep the declared type (R3) through dual-run
analysis.
withTable("char_varchar_dual_run") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
