Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18460#discussion_r144082145
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -100,6 +101,17 @@ object TypeCoercion {
case (_: TimestampType, _: DateType) | (_: DateType, _: TimestampType)
=>
Some(TimestampType)
+ case (t1 @ StructType(fields1), t2 @ StructType(fields2)) if
t1.sameType(t2) =>
+ Some(StructType(fields1.zip(fields2).map { case (f1, f2) =>
+ // Since `t1.sameType(t2)` is true, two StructTypes have the same
DataType
+ // except `name` (in case of `spark.sql.caseSensitive=false`) and
`nullable`.
+ // - Different names: use a lower case name because
findTightestCommonType is commutative.
+ // - Different nullabilities: `nullable` is true iff one of them
is nullable.
+ val name = if (f1.name == f2.name) f1.name else
f1.name.toLowerCase(Locale.ROOT)
+ val dataType = findTightestCommonType(f1.dataType, f2.dataType).get
+ StructField(name, dataType, nullable = f1.nullable || f2.nullable)
--- End diff --
Please see
[TypeCoercionSuite.checkWidenType](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala#L130-L142).
In order to use the first type name, we need to loosen this test helper
function and to break the existing commutative assumption. I'm ok for that if
you want.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
