mihailoale-db commented on code in PR #59271:
URL: https://github.com/apache/spark/pull/59271#discussion_r4260505384
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -1759,15 +1759,26 @@ case class Unpivot(
def canBeCoercioned: Boolean = values.exists(_.nonEmpty) &&
values.exists(_.forall(_.forall(_.resolved))) &&
+ // values are coerced by position, so every value group must have one
value per value column,
+ // otherwise CheckAnalysis reports UNPIVOT_VALUE_SIZE_MISMATCH
+ values.exists(_.forall(_.length == valueColumnNames.length)) &&
// when no ids are given, values must be Attributes (column names) to
allow detecting ids
// coercion will add aliases, would disallow detecting ids, so defer
coercion after id detection
ids.exists(_.forall(_.resolved))
- def valuesTypeCoercioned: Boolean = canBeCoercioned &&
- // all inner values at position idx must have the same data type
+ def valuesTypeCoercioned: Boolean = canBeCoercioned && {
+ // all inner values at position idx must have exactly the same data type,
including nested
+ // nullability, because the value column takes the data type of the first
value
+ val sameType: (DataType, DataType) => Boolean =
+ if
(conf.getConf(SQLConf.LEGACY_UNPIVOT_USE_FIRST_VALUE_NESTED_NULLABILITY)) {
+ DataTypeUtils.sameType
+ } else {
+ _ == _
Review Comment:
You're right, this PR is what sends these inputs into `findTypeForComplex`.
I reworked the fix to avoid that: `valuesTypeCoercioned` uses
`DataTypeUtils.sameType` again, so values that differ only in nullability are
no longer coerced. Instead, `UnpivotTransformer`
merges their nested nullability into the value column type with
`StructType.unionLikeMerge`, the same way `Union` merges the output of its
children. The value column keeps the nested field names and metadata of the
first value. I added a test that covers both value
orders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
