This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 85eb4c8866d0 [SPARK-58794][SQL] Promote CHAR/VARCHAR to STRING via
ImplicitTypeCasts
85eb4c8866d0 is described below
commit 85eb4c8866d06bac1fbc2969a3459f94bf4bd257
Author: Serge Rielau <[email protected]>
AuthorDate: Tue Aug 25 14:24:53 2026 +0800
[SPARK-58794][SQL] Promote CHAR/VARCHAR to STRING via ImplicitTypeCasts
### What changes were proposed in this pull request?
Under `spark.sql.charVarchar.standardSemantics.enabled`, an expression that
expects a plain string now gets `CHAR(n)` / `VARCHAR(n)` promoted to unbounded
`STRING` by the analyzer, the same way `SHORT` is promoted to `INT`. Previously
each transforming expression stripped the length constraint in its own
`dataType`.
- Move `charVarcharToPlainString` onto `TypeCoercionHelper` and recurse
into arrays when the expected type is `AbstractArrayType` / `TypeCollection`
(for example, `array_join`).
- Apply the promotion in `ImplicitTypeCasts` for both
`ImplicitCastInputTypes` and `ExpectsInputTypes`.
- Use collation-preserving `implicitCastToString` for `Concat` / `Elt`
instead of `implicitCast(e, StringType)`, which missed collated CHAR.
- Drop the per-expression `StringHelper.transformingStringResultType`
stripping; a result type now follows its promoted child.
- Use `ExpectsInputTypes` for `StringSplitSQL`.
- Give `JsonTuple` a CHAR/VARCHAR-only coercion arm so its existing INT and
untyped NULL validation remains unchanged.
### Why are the changes needed?
Stripping the constraint inside each expression missed every
`ExpectsInputTypes` site (`str_to_map`, `array_join`, and others) and collated
CHAR on `Concat`, and it had to be repeated in each new expression. Promoting
at the call site is the same mechanism Spark already uses for numeric widening,
so it applies uniformly and shows up in the analyzed plan as a `CAST(... AS
STRING)`.
### Does this PR introduce _any_ user-facing change?
Yes, when the flag is on: analyzed plans for transforming string operators
now contain an explicit `CAST(... AS STRING)` and those operators return
unbounded `STRING`. Result values are unchanged versus the previous
per-expression stripping.
### How was this patch tested?
- `TypeCoercionSuite`: legacy and ANSI `Concat` / `Elt` promotion,
including `CHAR(n) COLLATE UTF8_LCASE`; `ImplicitTypeCasts` promotion; and
unchanged `JsonTuple` INT / untyped NULL handling.
- `BasicCharVarcharTestSuite`: transforming string result types and
collated Concat / Elt values.
- Regenerated `charvarchar-standard-semantics.sql` result and analyzer
goldens.
### Was this patch authored or co-authored using generative AI tooling?
Yes (Cursor).
Closes #58130 from srielau/serge-rielau_data/SPARK-58794-r1-implicit.
Authored-by: Serge Rielau <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../org/apache/spark/sql/types/StringType.scala | 16 +--
.../sql/catalyst/analysis/AnsiTypeCoercion.scala | 2 +-
.../spark/sql/catalyst/analysis/TypeCoercion.scala | 2 +-
.../sql/catalyst/analysis/TypeCoercionBase.scala | 46 +--------
.../sql/catalyst/analysis/TypeCoercionHelper.scala | 113 ++++++++++++++++++---
.../expressions/collectionOperations.scala | 15 ++-
.../catalyst/expressions/complexTypeCreator.scala | 4 +-
.../sql/catalyst/expressions/jsonExpressions.scala | 5 +-
.../sql/catalyst/expressions/maskExpressions.scala | 4 +-
.../sql/catalyst/expressions/mathExpressions.scala | 6 +-
.../catalyst/expressions/regexpExpressions.scala | 8 +-
.../catalyst/expressions/stringExpressions.scala | 46 +++++----
.../sql/catalyst/analysis/TypeCoercionSuite.scala | 61 +++++++++++
.../charvarchar-standard-semantics.sql.out | 48 ++++++++-
.../inputs/charvarchar-standard-semantics.sql | 14 +++
.../results/charvarchar-standard-semantics.sql.out | 42 ++++++++
.../apache/spark/sql/CharVarcharTestSuite.scala | 107 +++++++++++++++----
17 files changed, 393 insertions(+), 146 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala
index 9de50cd758e8..736fbfead3d7 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala
@@ -163,8 +163,7 @@ case object StringHelper extends
PartialOrdering[StringConstraint] {
* Strip CHAR/VARCHAR length constraints, preserving collation.
*
* Used by transforming string expressions (upper, substr, concat, ...) so
their result type is
- * plain STRING even when inputs are CharType/VarcharType (SQL standard
CHAR/VARCHAR R1), when
- * standard semantics are on.
+ * plain STRING even when inputs are CharType/VarcharType, when standard
semantics are on.
*/
def plainStringType(dt: DataType): DataType = dt match {
case c: CharType => c.toStringType
@@ -178,19 +177,6 @@ case object StringHelper extends
PartialOrdering[StringConstraint] {
case other => other
}
- /**
- * Result type for transforming string expressions. Under
- * spark.sql.charVarchar.standardSemantics.enabled, always plain STRING
(R1). Under
- * preserveCharVarcharTypeInfo alone, keep child type (legacy leaky path).
- */
- def transformingStringResultType(dt: DataType): DataType = {
- if (SqlApiConf.get.charVarcharStandardSemantics) {
- plainStringType(dt)
- } else {
- dt
- }
- }
-
def isMoreConstrained(a: StringType, b: StringType): Boolean =
gteq(a.constraint, b.constraint)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala
index efac72d250ce..d64f9d48bee2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala
@@ -168,7 +168,7 @@ object AnsiTypeCoercion extends TypeCoercionBase {
private def implicitCast(
inType: DataType,
expectedType: AbstractDataType): Option[DataType] = {
- // R1 CHAR/VARCHAR promotion is checked first: the acceptsType case below
would otherwise
+ // CHAR/VARCHAR promotion is checked first: the acceptsType case below
would otherwise
// accept the constrained type unchanged, since CharType and VarcharType
extend StringType.
charVarcharToPlainString(inType, expectedType).foreach(dt => return
Some(dt))
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
index b6a7e8aed67b..755882325939 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
@@ -202,7 +202,7 @@ object TypeCoercion extends TypeCoercionBase {
private def implicitCast(inType: DataType, expectedType: AbstractDataType):
Option[DataType] = {
// Note that ret is nullable to avoid typing a lot of Some(...) in this
local scope.
// We wrap immediately an Option after this.
- // R1 CHAR/VARCHAR promotion is checked first: the acceptsType case below
would otherwise
+ // CHAR/VARCHAR promotion is checked first: the acceptsType case below
would otherwise
// accept the constrained type unchanged, since CharType and VarcharType
extend StringType.
charVarcharToPlainString(inType, expectedType).foreach(dt => return
Some(dt))
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala
index 1be2d52e8842..7c83a5282910 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala
@@ -51,54 +51,10 @@ import
org.apache.spark.sql.catalyst.util.ResolveDefaultColumns
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper
import org.apache.spark.sql.connector.catalog.procedures.BoundProcedure
import
org.apache.spark.sql.errors.DataTypeErrors.cannotMergeIncompatibleDataTypesError
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.internal.types.AbstractStringType
-import org.apache.spark.sql.types.{
- AbstractDataType,
- DataType,
- StringHelper,
- StringType,
- TypeCollection
-}
+import org.apache.spark.sql.types.DataType
abstract class TypeCoercionBase extends TypeCoercionHelper {
- /**
- * R1 promotion for CHAR(n)/VARCHAR(n): where a plain string is expected,
promote to STRING the
- * same way SHORT promotes to INT, and return the promoted type.
- *
- * CharType and VarcharType extend StringType, so an expectation such as
- * `StringTypeWithCollation` accepts them as-is and the implicit cast rules
leave the length
- * constraint in place. Expressions that then require all their string
inputs to share a single
- * type (`overlay`, `string_agg`, ...) cannot unify CHAR(n) with STRING, and
RuntimeReplaceable
- * ones (`right`) build literals from the constrained type that no longer
match their other
- * branches.
- *
- * The expectation must actually mention a string type. Promoting at an
`AnyDataType` site would
- * strip the length from pass-through expressions such as `max`, `lag`, and
`element_at`, which
- * are required to preserve CHAR/VARCHAR (R2/R3).
- */
- protected def charVarcharToPlainString(
- inType: DataType,
- expectedType: AbstractDataType): Option[DataType] = inType match {
- case st: StringType
- if SQLConf.get.charVarcharStandardSemantics &&
!StringHelper.isPlainString(st) =>
- val plain = StringHelper.plainStringType(st)
- if (expectsStringType(expectedType) && expectedType.acceptsType(plain)) {
- Some(plain)
- } else {
- None
- }
- case _ => None
- }
-
- private def expectsStringType(expectedType: AbstractDataType): Boolean =
expectedType match {
- case _: StringType => true
- case _: AbstractStringType => true
- case TypeCollection(types) => types.exists(expectsStringType)
- case _ => false
- }
-
/**
* Type coercion rule that combines multiple type coercion rules and applies
them in a single tree
* traversal.
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
index 93e1c602c072..f64f9a09a5e4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
@@ -39,6 +39,7 @@ import org.apache.spark.sql.catalyst.expressions.{
ImplicitCastInputTypes,
In,
InSubquery,
+ JsonTuple,
Least,
ListQuery,
Literal,
@@ -60,7 +61,11 @@ import
org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.types.DataTypeUtils
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.internal.types.{AbstractArrayType,
StringTypeWithCollation}
+import org.apache.spark.sql.internal.types.{
+ AbstractArrayType,
+ AbstractStringType,
+ StringTypeWithCollation
+}
import org.apache.spark.sql.types.{
AbstractDataType,
AnyDataType,
@@ -80,6 +85,7 @@ import org.apache.spark.sql.types.{
IntegralType,
MapType,
NullType,
+ StringHelper,
StringType,
StringTypeExpression,
StructType,
@@ -89,7 +95,8 @@ import org.apache.spark.sql.types.{
TimestampNTZType,
TimestampType,
TimestampTypeExpression,
- TimeType
+ TimeType,
+ TypeCollection
}
abstract class TypeCoercionHelper {
@@ -133,6 +140,69 @@ abstract class TypeCoercionHelper {
*/
def implicitCast(e: Expression, expectedType: AbstractDataType):
Option[Expression]
+ /**
+ * Where a plain string is expected, promote CHAR(n)/VARCHAR(n) to unbounded
STRING the same way
+ * SHORT promotes to INT, and return the promoted type.
+ *
+ * CharType and VarcharType extend StringType, so an expectation such as
+ * `StringTypeWithCollation` accepts them as-is and the implicit cast rules
leave the length
+ * constraint in place. Expressions that then require all their string
inputs to share a single
+ * type (`overlay`, `string_agg`, ...) cannot unify CHAR(n) with STRING, and
RuntimeReplaceable
+ * ones (`right`) build literals from the constrained type that no longer
match their other
+ * branches.
+ *
+ * The expectation must actually mention a string type (or an array of
strings). Promoting at an
+ * `AnyDataType` site would strip the length from pass-through expressions
such as `max`, `lag`,
+ * and `element_at`, which are required to preserve CHAR/VARCHAR.
+ */
+ protected def charVarcharToPlainString(
+ inType: DataType,
+ expectedType: AbstractDataType): Option[DataType] = {
+ if (!conf.charVarcharStandardSemantics) {
+ return None
+ }
+ inType match {
+ case st: StringType if !StringHelper.isPlainString(st) =>
+ val plain = StringHelper.plainStringType(st)
+ if (expectsStringType(expectedType) &&
expectedType.acceptsType(plain)) {
+ Some(plain)
+ } else {
+ None
+ }
+ case ArrayType(et, containsNull) =>
+ arrayElementExpectation(expectedType).flatMap { elemExpected =>
+ charVarcharToPlainString(et, elemExpected).map(ArrayType(_,
containsNull))
+ }
+ case _ => None
+ }
+ }
+
+ private def expectsStringType(expectedType: AbstractDataType): Boolean =
expectedType match {
+ case _: StringType => true
+ case _: AbstractStringType => true
+ case TypeCollection(types) => types.exists(expectsStringType)
+ case _ => false
+ }
+
+ private def arrayElementExpectation(expectedType: AbstractDataType):
Option[AbstractDataType] =
+ expectedType match {
+ case AbstractArrayType(elem) => Some(elem)
+ case ArrayType(elem, _) => Some(elem)
+ case TypeCollection(types) =>
types.view.flatMap(arrayElementExpectation).headOption
+ case _ => None
+ }
+
+ /**
+ * Concat/Elt stringify non-binary inputs. CHAR/VARCHAR go through
+ * [[charVarcharToPlainString]] so a collated constrained type promotes to
unbounded STRING
+ * with the same collation. Non-strings still target the default UTF8_BINARY
StringType.
+ */
+ protected def implicitCastToString(e: Expression): Expression = {
+ charVarcharToPlainString(e.dataType,
StringTypeWithCollation(supportsTrimCollation = true))
+ .map(dt => if (dt == e.dataType) e else Cast(e, dt))
+ .getOrElse(implicitCast(e, StringType).getOrElse(e))
+ }
+
/**
* Whether casting `from` as `to` is valid.
*/
@@ -512,9 +582,7 @@ abstract class TypeCoercionHelper {
case c @ Concat(children)
if conf.concatBinaryAsString ||
!children.map(_.dataType).forall(_ == BinaryType) =>
- val newChildren = c.children.map { e =>
- implicitCast(e, StringType).getOrElse(e)
- }
+ val newChildren = c.children.map(implicitCastToString)
c.copy(children = newChildren)
case other => other
}
@@ -562,9 +630,7 @@ abstract class TypeCoercionHelper {
val newInputs =
if (conf.eltOutputAsString ||
!children.tail.map(_.dataType).forall(_ == BinaryType)) {
- children.tail.map { e =>
- implicitCast(e, StringType).getOrElse(e)
- }
+ children.tail.map(implicitCastToString)
} else {
children.tail
}
@@ -649,16 +715,35 @@ abstract class TypeCoercionHelper {
}
e.withNewChildren(children)
+ // JsonTuple validates its own input types and rejects non-string
children with
+ // NON_STRING_TYPE, so it only takes the CHAR/VARCHAR promotion here. Do
not fold this into
+ // the ExpectsInputTypes arm below: that would also apply the NullType
rewrite and turn
+ // json_tuple(json, null) from an analysis error into a typed STRING
null.
+ case j: JsonTuple =>
+ val expected = StringTypeWithCollation(supportsTrimCollation = true)
+ val children = j.children.map { child =>
+ charVarcharToPlainString(child.dataType, expected)
+ .map(dt => if (dt == child.dataType) child else Cast(child, dt))
+ .getOrElse(child)
+ }
+ j.withNewChildren(children)
+
case e: ExpectsInputTypes if e.inputTypes.nonEmpty =>
// Convert NullType into some specific target type for
ExpectsInputTypes that don't do
- // general implicit casting.
+ // general implicit casting. Also promote CHAR/VARCHAR to STRING here:
these
+ // expressions skip ImplicitCastInputTypes, so without this the length
constraint would
+ // remain on the child.
val children: Seq[Expression] = e.children.zip(e.inputTypes).map {
case (in, expected) =>
- if (in.dataType == NullType && !expected.acceptsType(NullType)) {
- Literal.create(null, expected.defaultConcreteType)
- } else {
- in
- }
+ charVarcharToPlainString(in.dataType, expected)
+ .map(dt => if (dt == in.dataType) in else Cast(in, dt))
+ .getOrElse {
+ if (in.dataType == NullType &&
!expected.acceptsType(NullType)) {
+ Literal.create(null, expected.defaultConcreteType)
+ } else {
+ in
+ }
+ }
}
e.withNewChildren(children)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 8561ce3aae8e..80db7c88c6e3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -1430,11 +1430,9 @@ case class Reverse(child: Expression)
BinaryType,
ArrayType))
- // Reversing a string transforms its content, so a CHAR/VARCHAR input yields
plain STRING (R1).
- // Array and binary inputs are unaffected. ImplicitTypeCasts already
promotes the string branch
- // (its promotion looks inside a TypeCollection), so this covers the paths
that do not go
- // through implicit casting, such as an expression built directly.
- override def dataType: DataType =
StringHelper.transformingStringResultType(child.dataType)
+ // Reversing a string transforms its content, so ImplicitTypeCasts promotes
CHAR/VARCHAR to
+ // STRING. Array and binary inputs are unaffected.
+ override def dataType: DataType = child.dataType
private def resultArrayElementNullable =
dataType.asInstanceOf[ArrayType].containsNull
@@ -2513,10 +2511,9 @@ case class ArrayJoin(
}
}
- // The joined result concatenates every element plus delimiters, so it must
not inherit the
- // element's CHAR/VARCHAR length constraint (R1).
+ // After ImplicitTypeCasts, array elements that were CHAR/VARCHAR are STRING.
override def dataType: DataType =
-
StringHelper.transformingStringResultType(array.dataType.asInstanceOf[ArrayType].elementType)
+ array.dataType.asInstanceOf[ArrayType].elementType
override def prettyName: String = "array_join"
@@ -3203,7 +3200,7 @@ case class Concat(children: Seq[Expression]) extends
ComplexTypeMergingExpressio
if (children.isEmpty) {
StringType
} else {
- StringHelper.transformingStringResultType(super.dataType)
+ super.dataType
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
index bff416537d4b..2f5b05af043d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@@ -641,9 +641,9 @@ case class StringToMap(text: Expression, pairDelim:
Expression, keyValueDelim: E
Seq(StringTypeNonCSAICollation, StringTypeNonCSAICollation,
StringTypeNonCSAICollation)
// The entries are split out of the input, so they do not carry its
CHAR(n)/VARCHAR(n) length
- // (R1). ExpectsInputTypes does not insert a cast, so the constraint has to
be dropped here.
+ // constraint. ImplicitTypeCasts promotes CHAR/VARCHAR to STRING at this
ExpectsInputTypes site.
private lazy val entryType: DataType =
- StringHelper.transformingStringResultType(first.dataType)
+ first.dataType
override def dataType: DataType = MapType(entryType, entryType)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index cb8988af4235..a5f07d8eaa03 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -287,9 +287,10 @@ case class JsonTuple(children: Seq[Expression])
}
// The extracted fields are values from inside the JSON document, so they do
not carry the
- // CHAR(n)/VARCHAR(n) length of the document itself (R1).
+ // CHAR(n)/VARCHAR(n) length of the document itself. ImplicitTypeCoercion
promotes CHAR/VARCHAR
+ // children to STRING without applying general implicit casts or rewriting
untyped NULL.
private lazy val fieldType: DataType =
- StringHelper.transformingStringResultType(children.head.dataType)
+ children.head.dataType
override def elementSchema: StructType =
StructType(fieldExpressions.zipWithIndex.map {
case (_, idx) => StructField(s"c$idx", fieldType, nullable = true)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala
index dbf2331d6517..9d6f2c037df6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala
@@ -25,7 +25,7 @@ import
org.apache.spark.sql.catalyst.expressions.codegen.Block._
import org.apache.spark.sql.catalyst.plans.logical.{FunctionSignature,
InputParameter}
import org.apache.spark.sql.errors.QueryErrorsBase
import org.apache.spark.sql.internal.types.StringTypeWithCollation
-import org.apache.spark.sql.types.{AbstractDataType, DataType, StringHelper,
StringType}
+import org.apache.spark.sql.types.{AbstractDataType, DataType, StringType}
import org.apache.spark.unsafe.types.UTF8String
// scalastyle:off line.size.limit
@@ -291,7 +291,7 @@ case class Mask(
* the dataType of an unresolved expression (i.e., when `resolved` == false).
*/
override def dataType: DataType =
- StringHelper.transformingStringResultType(input.dataType)
+ input.dataType
/**
* Returns a Seq of the children of this node. Children should not change.
Immutability required
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
index 92ce8c20d391..988b1f73a6e6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
@@ -1282,9 +1282,9 @@ case class Hex(child: Expression)
Seq(TypeCollection(LongType, BinaryType,
StringTypeWithCollation(supportsTrimCollation = true)))
override def dataType: DataType = child.dataType match {
- // Hex expands each input character to two hex digits, so a
CHAR(n)/VARCHAR(n) input must not
- // carry its length constraint into the result (R1).
- case st: StringType => StringHelper.transformingStringResultType(st)
+ // After ImplicitTypeCasts, a CHAR/VARCHAR input is STRING. Keep collation
from the
+ // promoted child rather than DefaultStringProducingExpression's
UTF8_BINARY StringType.
+ case st: StringType => st
case _ => super.dataType
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
index 78c0eacf124c..5c1fb73cf1a8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
@@ -618,7 +618,7 @@ case class StringSplit(str: Expression, regex: Expression,
limit: Expression)
extends TernaryExpression with ImplicitCastInputTypes {
override def nullIntolerant: Boolean = true
override def dataType: DataType =
- ArrayType(StringHelper.transformingStringResultType(str.dataType),
containsNull = false)
+ ArrayType(str.dataType, containsNull = false)
override def inputTypes: Seq[AbstractDataType] =
Seq(StringTypeBinaryLcase, StringTypeWithCollation, IntegerType)
override def first: Expression = str
@@ -767,7 +767,7 @@ case class RegExpReplace(subject: Expression, regexp:
Expression, rep: Expressio
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(subject.dataType)
+ subject.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(StringTypeBinaryLcase,
StringTypeWithCollation, StringTypeBinaryLcase, IntegerType)
@@ -943,7 +943,7 @@ case class RegExpExtract(subject: Expression, regexp:
Expression, idx: Expressio
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(subject.dataType)
+ subject.dataType
override def prettyName: String = "regexp_extract"
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode = {
@@ -1020,7 +1020,7 @@ case class RegExpExtractAll(subject: Expression, regexp:
Expression, idx: Expres
}
override def dataType: DataType =
- ArrayType(StringHelper.transformingStringResultType(subject.dataType))
+ ArrayType(subject.dataType)
override def prettyName: String = "regexp_extract_all"
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode = {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index c9a287799a1a..afd894167ac9 100755
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -100,7 +100,7 @@ case class ConcatWs(children: Seq[Expression])
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(children.head.dataType)
+ children.head.dataType
override def nullable: Boolean = children.head.nullable
override def foldable: Boolean = children.forall(_.foldable)
@@ -455,7 +455,7 @@ trait String2StringExpression extends
ImplicitCastInputTypes {
def convert(v: UTF8String): UTF8String
override def dataType: DataType =
- StringHelper.transformingStringResultType(child.dataType)
+ child.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(StringTypeWithCollation(supportsTrimCollation = true))
override def contextIndependentFoldable: Boolean =
child.contextIndependentFoldable
@@ -1076,7 +1076,7 @@ case class StringReplace(srcExpr: Expression, searchExpr:
Expression, replaceExp
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(srcExpr.dataType)
+ srcExpr.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeNonCSAICollation(supportsTrimCollation = true),
@@ -1171,7 +1171,7 @@ case class Overlay(input: Expression, replace:
Expression, pos: Expression, len:
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(input.dataType)
+ input.dataType
override def inputTypes: Seq[AbstractDataType] = Seq(
TypeCollection(
@@ -1361,7 +1361,7 @@ case class StringTranslate(srcExpr: Expression,
matchingExpr: Expression, replac
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(srcExpr.dataType)
+ srcExpr.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeNonCSAICollation(supportsTrimCollation = true),
@@ -1442,7 +1442,7 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def children: Seq[Expression] = srcStr +: trimStr.toSeq
override def dataType: DataType =
- StringHelper.transformingStringResultType(srcStr.dataType)
+ srcStr.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq.fill(children.size)(StringTypeWithCollation(supportsTrimCollation =
true))
@@ -2000,7 +2000,7 @@ case class SubstringIndex(strExpr: Expression, delimExpr:
Expression, countExpr:
final lazy val collationId: Int =
first.dataType.asInstanceOf[StringType].collationId
override def dataType: DataType =
- StringHelper.transformingStringResultType(strExpr.dataType)
+ strExpr.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeNonCSAICollation(supportsTrimCollation = true),
@@ -2214,7 +2214,7 @@ case class StringLPad(str: Expression, len: Expression,
pad: Expression)
override def third: Expression = pad
override def dataType: DataType =
- StringHelper.transformingStringResultType(str.dataType)
+ str.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeWithCollation(supportsTrimCollation = true),
@@ -2311,7 +2311,7 @@ case class StringRPad(
override def third: Expression = pad
override def dataType: DataType =
- StringHelper.transformingStringResultType(str.dataType)
+ str.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeWithCollation(supportsTrimCollation = true),
@@ -2368,7 +2368,7 @@ case class FormatString(children: Expression*) extends
Expression with ImplicitC
override def contextIndependentFoldable: Boolean =
children.forall(_.contextIndependentFoldable)
override def nullable: Boolean = children(0).nullable
override def dataType: DataType =
- StringHelper.transformingStringResultType(children(0).dataType)
+ children(0).dataType
override def inputTypes: Seq[AbstractDataType] =
StringTypeWithCollation(supportsTrimCollation = true) ::
@@ -2491,7 +2491,7 @@ case class InitCap(child: Expression)
override def inputTypes: Seq[AbstractDataType] =
Seq(StringTypeWithCollation(supportsTrimCollation = true))
override def dataType: DataType =
- StringHelper.transformingStringResultType(child.dataType)
+ child.dataType
override def nullSafeEval(string: Any): Any = {
CollationSupport.InitCap.exec(string.asInstanceOf[UTF8String],
collationId, useICU)
@@ -2529,7 +2529,7 @@ case class StringRepeat(str: Expression, times:
Expression)
override def left: Expression = str
override def right: Expression = times
override def dataType: DataType =
- StringHelper.transformingStringResultType(str.dataType)
+ str.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
StringTypeWithCollation(supportsTrimCollation = true),
@@ -2642,7 +2642,7 @@ case class Substring(str: Expression, pos: Expression,
len: Expression)
}
override def dataType: DataType =
- StringHelper.transformingStringResultType(str.dataType)
+ str.dataType
override def inputTypes: Seq[AbstractDataType] =
Seq(
@@ -2706,11 +2706,9 @@ case class Substring(str: Expression, pos: Expression,
len: Expression)
case class Right(str: Expression, len: Expression) extends RuntimeReplaceable
with ImplicitCastInputTypes with BinaryLike[Expression] {
- // Type the literal branches after R1: Substring returns plain STRING for a
CHAR(n)/VARCHAR(n)
- // input, so deriving the literals from str.dataType would leave the If with
branches of
- // different types. CheckAnalysis does not see inside a RuntimeReplaceable's
replacement, so
- // that mismatch would surface later as COMPLEX_EXPRESSION_UNSUPPORTED_INPUT.
- private lazy val resultType: DataType =
StringHelper.transformingStringResultType(str.dataType)
+ // Type the literal branches after ImplicitTypeCasts promotes CHAR/VARCHAR
to STRING.
+ // Substring then returns STRING, so the If branches match.
+ private lazy val resultType: DataType = str.dataType
override lazy val replacement: Expression = If(
IsNull(str),
@@ -4086,7 +4084,7 @@ case class Sentences(
override def nullable: Boolean = true
override def dataType: DataType = {
- val elementType = StringHelper.transformingStringResultType(str.dataType)
+ val elementType = str.dataType
ArrayType(ArrayType(elementType, containsNull = false), containsNull =
false)
}
override def inputTypes: Seq[AbstractDataType] =
@@ -4122,9 +4120,13 @@ case class Sentences(
*/
case class StringSplitSQL(
str: Expression,
- delimiter: Expression) extends BinaryExpression {
+ delimiter: Expression) extends BinaryExpression with ExpectsInputTypes {
override def dataType: DataType =
- ArrayType(StringHelper.transformingStringResultType(str.dataType),
containsNull = false)
+ ArrayType(str.dataType, containsNull = false)
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(
+ StringTypeWithCollation(supportsTrimCollation = true),
+ StringTypeWithCollation(supportsTrimCollation = true))
final lazy val collationId: Int =
left.dataType.asInstanceOf[StringType].collationId
override def left: Expression = str
override def right: Expression = delimiter
@@ -4214,7 +4216,7 @@ case class Empty2Null(child: Expression) extends
UnaryExpression with String2Str
// Not a transforming function: every non-empty value is returned unchanged,
so this keeps the
// child's type rather than taking the plain-STRING result that
String2StringExpression gives
- // its transforming implementations (R1).
+ // its transforming implementations.
override def dataType: DataType = child.dataType
override def nullable: Boolean = true
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
index fc596766e189..577ff6de4f1c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
@@ -354,6 +354,22 @@ abstract class TypeCoercionSuiteBase extends AnalysisTest {
Concat(Seq(Literal("123".getBytes), Literal("456".getBytes))),
Concat(Seq(Literal("123".getBytes), Literal("456".getBytes))))
}
+
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val charLit = Literal.create("ab", CharType(2))
+ val collatedChar = Literal.create("ab", CharType(2, "UTF8_LCASE"))
+ val collatedString = StringType("UTF8_LCASE")
+ Seq(TypeCoercion.ConcatCoercion,
AnsiTypeCoercion.ConcatCoercion).foreach { r =>
+ ruleTest(r,
+ Concat(Seq(charLit, charLit)),
+ Concat(Seq(Cast(charLit, StringType), Cast(charLit, StringType))))
+ ruleTest(r,
+ Concat(Seq(collatedChar, collatedChar)),
+ Concat(Seq(
+ Cast(collatedChar, collatedString),
+ Cast(collatedChar, collatedString))))
+ }
+ }
}
test("type coercion for Elt") {
@@ -408,6 +424,23 @@ abstract class TypeCoercionSuiteBase extends AnalysisTest {
Elt(Seq(Literal(1), Literal("123".getBytes), Literal("456".getBytes))),
Elt(Seq(Literal(1), Literal("123".getBytes), Literal("456".getBytes))))
}
+
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val charLit = Literal.create("ab", CharType(5))
+ val collatedChar = Literal.create("ab", CharType(5, "UTF8_LCASE"))
+ val collatedString = StringType("UTF8_LCASE")
+ Seq(TypeCoercion.EltCoercion, AnsiTypeCoercion.EltCoercion).foreach { r
=>
+ ruleTest(r,
+ Elt(Seq(Literal(1), charLit, charLit)),
+ Elt(Seq(Literal(1), Cast(charLit, StringType), Cast(charLit,
StringType))))
+ ruleTest(r,
+ Elt(Seq(Literal(1), collatedChar, collatedChar)),
+ Elt(Seq(
+ Literal(1),
+ Cast(collatedChar, collatedString),
+ Cast(collatedChar, collatedString))))
+ }
+ }
}
test("Datetime operations") {
@@ -1083,6 +1116,34 @@ class TypeCoercionSuite extends TypeCoercionSuiteBase {
ruleTest(TypeCoercion.ImplicitTypeCasts,
NumericTypeUnaryExpression(Literal.create(null, NullType)),
NumericTypeUnaryExpression(Literal.create(null, DoubleType)))
+
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val charLit = Literal.create("ab", CharType(2))
+ ruleTest(TypeCoercion.ImplicitTypeCasts,
+ Upper(charLit),
+ Upper(Cast(charLit, StringType)))
+ }
+ }
+
+ test("coerce JsonTuple children without the NullType rewrite") {
+ val json = Literal("""{"a":1}""")
+ val nullField = Literal.create(null, NullType)
+ val intField = Literal(1)
+
+ // JsonTuple keeps its own NON_STRING_TYPE check, so these stay for
checkInputDataTypes.
+ ruleTest(TypeCoercion.ImplicitTypeCasts,
+ JsonTuple(Seq(json, nullField)),
+ JsonTuple(Seq(json, nullField)))
+ ruleTest(TypeCoercion.ImplicitTypeCasts,
+ JsonTuple(Seq(json, intField)),
+ JsonTuple(Seq(json, intField)))
+
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val charLit = Literal.create("ab", CharType(2))
+ ruleTest(TypeCoercion.ImplicitTypeCasts,
+ JsonTuple(Seq(charLit, charLit)),
+ JsonTuple(Seq(Cast(charLit, StringType), Cast(charLit, StringType))))
+ }
}
test("cast NullType for binary operators") {
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
index 070fba1bcd94..8081f9cecaaf 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/charvarchar-standard-semantics.sql.out
@@ -222,6 +222,44 @@ Project [typeof(concat(cast(cast(a as char(2)) as string),
cast(cast(b as char(3
+- OneRowRelation
+-- !query
+SELECT typeof(concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)))
+-- !query analysis
+Project [typeof(concat(cast(cast(cast(a as char(2) collate UTF8_LCASE) as
char(3) collate UTF8_LCASE) as string collate UTF8_LCASE), cast(cast(b as
char(3) collate UTF8_LCASE) as string collate UTF8_LCASE))) AS
typeof(concat(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(b AS CHAR(3) COLLATE
UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT concat('<', concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)), '>')
+-- !query analysis
+Project [concat(<, concat(cast(cast(cast(a as char(2) collate UTF8_LCASE) as
char(3) collate UTF8_LCASE) as string collate UTF8_LCASE), cast(cast(b as
char(3) collate UTF8_LCASE) as string collate UTF8_LCASE)), >) AS concat('<'
collate UTF8_LCASE, concat(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(b AS
CHAR(3) COLLATE UTF8_LCASE)), '>' collate UTF8_LCASE)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT typeof(elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)))
+-- !query analysis
+Project [typeof(elt(1, cast(cast(ab as char(5) collate UTF8_LCASE) as string
collate UTF8_LCASE), cast(cast(cast(x as char(1) collate UTF8_LCASE) as char(5)
collate UTF8_LCASE) as string collate UTF8_LCASE), true)) AS typeof(elt(1,
CAST(ab AS CHAR(5) COLLATE UTF8_LCASE), CAST(x AS CHAR(1) COLLATE
UTF8_LCASE)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT concat('<', elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)), '>')
+-- !query analysis
+Project [concat(<, elt(1, cast(cast(ab as char(5) collate UTF8_LCASE) as
string collate UTF8_LCASE), cast(cast(cast(x as char(1) collate UTF8_LCASE) as
char(5) collate UTF8_LCASE) as string collate UTF8_LCASE), true), >) AS
concat('<' collate UTF8_LCASE, elt(1, CAST(ab AS CHAR(5) COLLATE UTF8_LCASE),
CAST(x AS CHAR(1) COLLATE UTF8_LCASE)), '>' collate UTF8_LCASE)#x]
++- OneRowRelation
+
+
-- !query
SELECT typeof(trim(cast('ab ' AS CHAR(4))))
-- !query analysis
@@ -267,7 +305,7 @@ Project [typeof(split(cast(cast(a,b as char(3)) as string),
,, -1)) AS typeof(sp
-- !query
SELECT typeof(mask(cast('ab' AS CHAR(2))))
-- !query analysis
-Project [typeof(mask(cast(ab as char(2)), X, x, n, null)) AS
typeof(mask(CAST(ab AS CHAR(2)), X, x, n, NULL))#x]
+Project [typeof(mask(cast(cast(ab as char(2)) as string), X, x, n, null)) AS
typeof(mask(CAST(ab AS CHAR(2)), X, x, n, NULL))#x]
+- OneRowRelation
@@ -337,14 +375,14 @@ Project [hex(cast(cast(ab as char(5)) as string)) AS
hex(CAST(ab AS CHAR(5)))#x]
-- !query
SELECT typeof(array_join(array(cast('ab' AS CHAR(5)), cast('cd' AS CHAR(5))),
'-'))
-- !query analysis
-Project [typeof(array_join(array(cast(ab as char(5)), cast(cd as char(5))), -,
None)) AS typeof(array_join(array(CAST(ab AS CHAR(5)), CAST(cd AS CHAR(5))),
-))#x]
+Project [typeof(array_join(cast(array(cast(ab as char(5)), cast(cd as
char(5))) as array<string>), -, None)) AS typeof(array_join(array(CAST(ab AS
CHAR(5)), CAST(cd AS CHAR(5))), -))#x]
+- OneRowRelation
-- !query
SELECT concat('<', array_join(array(cast('ab' AS CHAR(5)), cast('cd' AS
CHAR(5))), '-'), '>')
-- !query analysis
-Project [concat(<, array_join(array(cast(ab as char(5)), cast(cd as char(5))),
-, None), >) AS concat(<, array_join(array(CAST(ab AS CHAR(5)), CAST(cd AS
CHAR(5))), -), >)#x]
+Project [concat(<, array_join(cast(array(cast(ab as char(5)), cast(cd as
char(5))) as array<string>), -, None), >) AS concat(<, array_join(array(CAST(ab
AS CHAR(5)), CAST(cd AS CHAR(5))), -), >)#x]
+- OneRowRelation
@@ -358,7 +396,7 @@ Project [typeof(reverse(array(1, 2))) AS
typeof(reverse(array(1, 2)))#x]
-- !query
SELECT typeof(str_to_map(cast('a:1,b:2' AS CHAR(7))))
-- !query analysis
-Project [typeof(str_to_map(cast(a:1,b:2 as char(7)), ,, :)) AS
typeof(str_to_map(CAST(a:1,b:2 AS CHAR(7)), ,, :))#x]
+Project [typeof(str_to_map(cast(cast(a:1,b:2 as char(7)) as string), ,, :)) AS
typeof(str_to_map(CAST(a:1,b:2 AS CHAR(7)), ,, :))#x]
+- OneRowRelation
@@ -368,7 +406,7 @@ SELECT typeof(c0) FROM (SELECT json_tuple(cast('{"a":"1"}'
AS CHAR(9)), 'a') AS
Project [typeof(c0#x) AS typeof(c0)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Project [c0#x]
- +- Generate json_tuple(cast({"a":"1"} as char(9)), a), false, [c0#x]
+ +- Generate json_tuple(cast(cast({"a":"1"} as char(9)) as string), a),
false, [c0#x]
+- OneRowRelation
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
b/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
index c48965f667fb..23baf68d2a98 100644
---
a/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
+++
b/sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql
@@ -45,6 +45,20 @@ SELECT typeof(cast('a' AS CHAR(1)) || cast('b' AS
VARCHAR(1)));
SELECT typeof(substr(cast('hello' AS VARCHAR(5)), 1, 2));
SELECT typeof(upper(coalesce(cast('a' AS CHAR(2)), cast('b' AS CHAR(4)))));
SELECT typeof(concat(cast('a' AS CHAR(2)), cast('b' AS CHAR(3))));
+SELECT typeof(concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)));
+SELECT concat('<', concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)), '>');
+SELECT typeof(elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)));
+SELECT concat('<', elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)), '>');
SELECT typeof(trim(cast('ab ' AS CHAR(4))));
SELECT typeof(lpad(cast('ab' AS CHAR(2)), 5, 'x'));
diff --git
a/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
b/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
index 538da5cffe49..c9592e8361c3 100644
---
a/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/charvarchar-standard-semantics.sql.out
@@ -264,6 +264,48 @@ struct<typeof(concat(CAST(a AS CHAR(2)), CAST(b AS
CHAR(3)))):string>
string
+-- !query
+SELECT typeof(concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)))
+-- !query schema
+struct<typeof(concat(CAST(a AS CHAR(2) COLLATE UTF8_LCASE), CAST(b AS CHAR(3)
COLLATE UTF8_LCASE))):string>
+-- !query output
+string collate UTF8_LCASE
+
+
+-- !query
+SELECT concat('<', concat(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast('b' AS CHAR(3) COLLATE UTF8_LCASE)), '>')
+-- !query schema
+struct<concat('<' collate UTF8_LCASE, concat(CAST(a AS CHAR(2) COLLATE
UTF8_LCASE), CAST(b AS CHAR(3) COLLATE UTF8_LCASE)), '>' collate
UTF8_LCASE):string collate UTF8_LCASE>
+-- !query output
+<a b >
+
+
+-- !query
+SELECT typeof(elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)))
+-- !query schema
+struct<typeof(elt(1, CAST(ab AS CHAR(5) COLLATE UTF8_LCASE), CAST(x AS CHAR(1)
COLLATE UTF8_LCASE))):string>
+-- !query output
+string collate UTF8_LCASE
+
+
+-- !query
+SELECT concat('<', elt(
+ 1,
+ cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ cast('x' AS CHAR(1) COLLATE UTF8_LCASE)), '>')
+-- !query schema
+struct<concat('<' collate UTF8_LCASE, elt(1, CAST(ab AS CHAR(5) COLLATE
UTF8_LCASE), CAST(x AS CHAR(1) COLLATE UTF8_LCASE)), '>' collate
UTF8_LCASE):string collate UTF8_LCASE>
+-- !query output
+<ab >
+
+
-- !query
SELECT typeof(trim(cast('ab ' AS CHAR(4))))
-- !query schema
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
index c375694d7576..a9e3c0626e0f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
@@ -21,10 +21,14 @@ import scala.util.Try
import org.apache.spark.{SparkConf, SparkException, SparkRuntimeException,
SparkThrowable}
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
-import org.apache.spark.sql.catalyst.expressions.{Attribute, EqualTo,
GreaterThan, Literal, ScalarSubquery, StringRPad}
+import org.apache.spark.sql.catalyst.analysis.resolver.ResolverGuard
+import org.apache.spark.sql.catalyst.expressions.{
+ ArrayJoin, Attribute, Concat, EqualTo, Expression, GreaterThan, Literal,
ScalarSubquery,
+ StringRPad, StringToMap, Upper
+}
import org.apache.spark.sql.catalyst.expressions.Cast.toSQLId
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Project}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter,
LogicalPlan, Project}
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.connector.SchemaRequiredDataSource
import org.apache.spark.sql.connector.catalog.{CatalogV2Util,
InMemoryPartitionTableCatalog}
@@ -982,6 +986,29 @@ class BasicCharVarcharTestSuite extends SharedSparkSession
{
checkAnswer(
sql("SELECT cast('he' AS CHAR(4)) || cast('llo' AS CHAR(3)) AS c"),
Row("he llo"))
+ // Collated CHAR must promote to the same collation, not UTF8_BINARY
STRING.
+ assert(sql(
+ """SELECT concat(
+ | cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | cast('b' AS CHAR(3) COLLATE UTF8_LCASE)) AS c""".stripMargin)
+ .schema.head.dataType === StringType("UTF8_LCASE"))
+ checkAnswer(
+ sql("""SELECT concat(
+ | cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ | cast('b' AS CHAR(3) COLLATE UTF8_LCASE)) AS c""".stripMargin),
+ Row("a b "))
+ assert(sql(
+ """SELECT elt(
+ | 1,
+ | cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ | cast('x' AS CHAR(1) COLLATE UTF8_LCASE)) AS c""".stripMargin)
+ .schema.head.dataType === StringType("UTF8_LCASE"))
+ checkAnswer(
+ sql("""SELECT elt(
+ | 1,
+ | cast('ab' AS CHAR(5) COLLATE UTF8_LCASE),
+ | cast('x' AS CHAR(1) COLLATE UTF8_LCASE)) AS c""".stripMargin),
+ Row("ab "))
assert(sql("SELECT substr(cast('hello' AS VARCHAR(5)), 1, 2) AS c")
.schema.head.dataType === StringType)
assert(sql(
@@ -1015,13 +1042,13 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
}
}
- test("SPARK-58796: preserve vs standardSemantics R1 matrix") {
+ test("SPARK-58796: preserve vs standardSemantics result-type matrix") {
// preserve-only: transforming ops may keep Char/Varchar (leaky
experimental path).
withSQLConf(SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true") {
assert(sql("SELECT upper(cast('ab' AS CHAR(2))) AS c")
.schema.head.dataType === CharType(2))
}
- // standardSemantics: R1 forces STRING even if preserve is also on.
+ // standardSemantics: transforming ops return STRING even if preserve is
also on.
withSQLConf(
SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true",
SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
@@ -1165,7 +1192,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
}
}
- test("SPARK-58794: R1 promotion unifies CHAR/VARCHAR with STRING at
plain-string inputs") {
+ test("SPARK-58794: promotion unifies CHAR/VARCHAR with STRING at
plain-string inputs") {
withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
withTable("std_promote") {
sql("CREATE TABLE std_promote (c CHAR(5), v VARCHAR(5)) USING parquet")
@@ -1180,7 +1207,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
"listagg(c, '-')" -> "ab ",
"elt(1, c, 'x')" -> "ab ",
// right() is RuntimeReplaceable; its literal branches must agree
with the substring
- // branch, which R1 has already reduced to STRING.
+ // branch, which promotion has already reduced to STRING.
"right(c, 2)" -> " ",
"left(c, 2)" -> "ab").foreach { case (expr, expected) =>
val df = sql(s"SELECT $expr AS r FROM std_promote")
@@ -1199,7 +1226,8 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
checkAnswer(df, Row(expected))
}
- // Promotion must not reach pass-through / LCT sites, which preserve
CHAR/VARCHAR (R2/R3).
+ // Promotion must not reach pass-through / least-common-type sites,
which preserve
+ // CHAR/VARCHAR.
Seq(
"c", "coalesce(c, c)", "case when true then c else c end", "max(c)",
"element_at(array(c), 1)", "transform(array(c), x -> x)[0]",
@@ -1208,19 +1236,19 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
assert(df.schema.head.dataType === CharType(5), s"$expr should stay
CHAR(5)")
}
- // reverse() on non-string inputs is unaffected by the R1 change.
+ // reverse() on non-string inputs is unaffected by the promotion.
assert(sql("SELECT reverse(array(1, 2)) AS r").schema.head.dataType ===
ArrayType(IntegerType, containsNull = false))
}
}
}
- // Allowlist for the inventory below: R2/R3 pass-through and container cases
that may keep
+ // Allowlist for the inventory below: pass-through and container cases that
may keep
// CHAR(n)/VARCHAR(n): aggregates/ordering that return an input unchanged,
null-handling,
// element access, array/map/struct constructors, and collection
rearrangements that keep
// element types. Coverage is limited to the seven fixed argumentShapes
templates in the test;
// a leak only at another arity or nested shape would not fail here. For
those shapes,
- // anything not listed must reduce to plain STRING (R1).
+ // anything not listed must reduce to plain STRING.
private val charVarcharPassThroughFunctions = Set(
"any_value", "approx_top_k", "approx_top_k_accumulate", "array",
"array_agg", "array_compact",
"array_distinct", "array_max", "array_min", "array_repeat", "array_sort",
"arrays_zip",
@@ -1347,6 +1375,33 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
}
}
+ test("SPARK-58794: ImplicitTypeCasts promotes CHAR/VARCHAR to STRING") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ def analyzed(sqlText: String): LogicalPlan =
sql(sqlText).queryExecution.analyzed
+ def exprs(plan: LogicalPlan): Seq[Expression] =
+ plan.flatMap(_.expressions.flatMap(e => e +: e.collect { case c => c
}))
+
+ val upper = exprs(analyzed("SELECT upper(cast('ab' AS CHAR(2)))"))
+ .collect { case u: Upper => u }.head
+ assert(upper.child.dataType === StringType)
+
+ val concat = exprs(analyzed(
+ "SELECT concat(cast('a' AS CHAR(2)), cast('b' AS CHAR(3)))"))
+ .collect { case c: Concat => c }.head
+ assert(concat.children.forall(_.dataType == StringType))
+
+ val arrayJoin = exprs(analyzed(
+ "SELECT array_join(array(cast('ab' AS CHAR(5))), '-')"))
+ .collect { case a: ArrayJoin => a }.head
+ assert(arrayJoin.array.dataType === ArrayType(StringType, containsNull =
false))
+
+ val stringToMap = exprs(analyzed("SELECT str_to_map(cast('a:1' AS
CHAR(5)))"))
+ .collect { case s: StringToMap => s }.head
+ assert(stringToMap.first.dataType === StringType)
+ assert(stringToMap.dataType === MapType(StringType, StringType,
valueContainsNull = true))
+ }
+ }
+
test("SPARK-58794: parameterized CHAR/VARCHAR lengths under
standardSemantics") {
// Length positions accept parameter markers (`integerValue` ->
`parameterMarker`). Under
// standardSemantics the bound type stays first-class: CAST keeps
CHAR/VARCHAR, pads and
@@ -1398,7 +1453,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
test("SPARK-58794: language surfaces keep CHAR/VARCHAR under
standardSemantics") {
withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
- // CTAS / CREATE VIEW inherit projected CHAR/VARCHAR (D13).
+ // CTAS / CREATE VIEW inherit projected CHAR/VARCHAR.
withTable("std_src", "std_ctas") {
sql("CREATE TABLE std_src (c CHAR(5), v VARCHAR(5)) USING parquet")
sql("INSERT INTO std_src VALUES ('ab', 'ab')")
@@ -1794,14 +1849,24 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
// silently lost: the HybridAnalyzer compares output schema and normalized
plan across the
// two analyzers and fails with HYBRID_ANALYZER_EXCEPTION on any
divergence. Resolver has no
// Char/Varchar-specific logic; it inherits Expression.dataType and shared
TypeCoercion, so
- // this matrix is the proof that LCT/CAST/R1 stay aligned (D19).
+ // this matrix is the proof that least common type, CAST, and promotion
stay aligned.
withSQLConf(
SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true",
SQLConf.ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER.key ->
"true",
SQLConf.ANALYZER_DUAL_RUN_SAMPLE_RATE.key -> "1.0",
SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key ->
"false",
SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_EXPOSE_RESOLVER_GUARD_FAILURE.key ->
"true") {
- // CAST / try_cast introduce the type (R3).
+ def sql(sqlText: String): DataFrame = {
+ val parsed = spark.sessionState.sqlParser.parsePlan(sqlText)
+ val unsupportedReason =
+ new
ResolverGuard(spark.sessionState.catalogManager).apply(parsed).planUnsupportedReason
+ assert(
+ unsupportedReason.isEmpty,
+ s"ResolverGuard skipped dual-run for [$sqlText]: $unsupportedReason")
+ spark.sql(sqlText)
+ }
+
+ // CAST / try_cast introduce the type.
assert(sql("SELECT CAST('ab' AS CHAR(5)) AS c").schema.head.dataType ===
CharType(5))
assert(sql("SELECT CAST('hello' AS VARCHAR(5)) AS
c").schema.head.dataType ===
VarcharType(5))
@@ -1820,7 +1885,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
| CAST('x' AS VARCHAR(4) COLLATE UTF8_LCASE)) AS c""".stripMargin),
Row("ab"))
- // Least common type (R2): COALESCE / CASE / NULL / CHAR+VARCHAR /
CHAR+STRING.
+ // Least common type: COALESCE / CASE / NULL / CHAR+VARCHAR /
CHAR+STRING.
assert(sql(
"SELECT coalesce(CAST('a' AS VARCHAR(3)), CAST('bb' AS VARCHAR(7))) AS
c")
.schema.head.dataType === VarcharType(7))
@@ -1845,7 +1910,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
sql("SELECT CAST('a' AS CHAR(2)) IN (CAST('a ' AS CHAR(2)), CAST('bbb'
AS VARCHAR(3)))"),
Row(true))
- // Transforming operators return STRING (R1).
+ // Transforming operators return STRING.
assert(sql("SELECT upper(CAST('ab' AS CHAR(2))) AS
c").schema.head.dataType === StringType)
assert(sql("SELECT lower(CAST('AB' AS VARCHAR(2))) AS
c").schema.head.dataType ===
StringType)
@@ -1861,7 +1926,7 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
assert(sql("SELECT mask(CAST('ab' AS CHAR(2))) AS
c").schema.head.dataType === StringType)
assert(sql("SELECT split(CAST('a,b' AS CHAR(3)), ',') AS
c").schema.head.dataType ===
ArrayType(StringType, containsNull = false))
- // R1 after LCT: coalesce stays CHAR, upper widens to STRING.
+ // Promotion after least common type: coalesce stays CHAR, upper widens
to STRING.
assert(sql(
"SELECT upper(coalesce(CAST('a' AS CHAR(2)), CAST('b' AS CHAR(4)))) AS
c")
.schema.head.dataType === StringType)
@@ -1924,10 +1989,10 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
assert(mixedCharUnion.schema.head.dataType === CharType(4))
checkAnswer(mixedCharUnion, Seq(Row("a "), Row("bb ")))
- // Bare column references keep the declared type (R3) through dual-run
analysis.
+ // Bare column references keep the declared type through dual-run
analysis.
withTable("char_varchar_dual_run") {
- sql("CREATE TABLE char_varchar_dual_run (c CHAR(5), v VARCHAR(5))
USING parquet")
- sql("INSERT INTO char_varchar_dual_run VALUES ('ab', 'ab')")
+ spark.sql("CREATE TABLE char_varchar_dual_run (c CHAR(5), v
VARCHAR(5)) USING parquet")
+ spark.sql("INSERT INTO char_varchar_dual_run VALUES ('ab', 'ab')")
val df = sql("SELECT c, v FROM char_varchar_dual_run")
assert(df.schema("c").dataType === CharType(5))
assert(df.schema("v").dataType === VarcharType(5))
@@ -2158,7 +2223,7 @@ class FileSourceCharVarcharTestSuite extends
CharVarcharTestSuite with SharedSpa
}
test("SPARK-58794: EXTERNAL TABLE CHAR/VARCHAR pad, assignment, and
oversize") {
- // D14: external file tables use the same store assignment on write and
pad +
+ // External file tables use the same store assignment on write and pad +
// length-check on scan. Hive TRANSFORM is out of scope for this epic.
withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
withTempPath { dir =>
@@ -2186,7 +2251,7 @@ class FileSourceCharVarcharTestSuite extends
CharVarcharTestSuite with SharedSpa
}
}
}
- // Bypass-writer oversize non-blank values fail on scan (D12).
+ // Bypass-writer oversize non-blank values fail on scan.
withTempPath { dir =>
val path = dir.getCanonicalPath
sql("SELECT 'abcdef' AS c").write.format(format).save(path)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
