mihailoale-db commented on code in PR #59271:
URL: https://github.com/apache/spark/pull/59271#discussion_r4260509198
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -1759,15 +1759,26 @@ case class Unpivot(
def canBeCoercioned: Boolean = values.exists(_.nonEmpty) &&
values.exists(_.forall(_.forall(_.resolved))) &&
+ // values are coerced by position, so every value group must have one
value per value column,
+ // otherwise CheckAnalysis reports UNPIVOT_VALUE_SIZE_MISMATCH
+ values.exists(_.forall(_.length == valueColumnNames.length)) &&
// when no ids are given, values must be Attributes (column names) to
allow detecting ids
// coercion will add aliases, would disallow detecting ids, so defer
coercion after id detection
ids.exists(_.forall(_.resolved))
- def valuesTypeCoercioned: Boolean = canBeCoercioned &&
- // all inner values at position idx must have the same data type
+ def valuesTypeCoercioned: Boolean = canBeCoercioned && {
+ // all inner values at position idx must have exactly the same data type,
including nested
+ // nullability, because the value column takes the data type of the first
value
+ val sameType: (DataType, DataType) => Boolean =
+ if
(conf.getConf(SQLConf.LEGACY_UNPIVOT_USE_FIRST_VALUE_NESTED_NULLABILITY)) {
+ DataTypeUtils.sameType
+ } else {
+ _ == _
Review Comment:
The same rework addresses this. Values that differ only in nested
nullability no longer get a `Cast`: the merged type is set only on the `Expand`
output attribute, as `Union` does, so `Cast` itself is unchanged. The new test
asserts that the analyzed plan has no
`Cast`, and the `unpivot.sql` analyzer results no longer show one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
