Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18460#discussion_r143908510
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -100,6 +101,17 @@ object TypeCoercion {
case (_: TimestampType, _: DateType) | (_: DateType, _: TimestampType)
=>
Some(TimestampType)
+ case (t1 @ StructType(fields1), t2 @ StructType(fields2)) if
t1.sameType(t2) =>
+ Some(StructType(fields1.zip(fields2).map { case (f1, f2) =>
+ // Since `t1.sameType(t2)` is true, two StructTypes have the same
DataType
+ // except `name` (in case of `spark.sql.caseSensitive=false`) and
`nullable`.
+ // - Different names: use a lower case name because
findTightestCommonType is commutative.
+ // - Different nullabilities: `nullable` is true iff one of them
is nullable.
+ val name = if (f1.name == f2.name) f1.name else
f1.name.toLowerCase(Locale.ROOT)
+ val dataType = findTightestCommonType(f1.dataType, f2.dataType).get
+ StructField(name, dataType, nullable = f1.nullable || f2.nullable)
--- End diff --
This PR works as you want. This function is used to compare the equality
only. BTW, for this function, it should should lower (or upper case) because it
should be commutative.
```
scala> sql("SELECT struct(1 a) UNION ALL (SELECT struct(2 A))").printSchema
root
|-- named_struct(a, 1 AS `a`): struct (nullable = false)
| |-- a: integer (nullable = false)
scala> sql("SELECT struct(1 A) UNION ALL (SELECT struct(2 a))").printSchema
root
|-- named_struct(A, 1 AS `A`): struct (nullable = false)
| |-- A: integer (nullable = false)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
