Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21074#discussion_r182284460
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -176,10 +176,16 @@ object TypeCoercion {
}
private def findWiderCommonType(types: Seq[DataType]): Option[DataType]
= {
- types.foldLeft[Option[DataType]](Some(NullType))((r, c) => r match {
- case Some(d) => findWiderTypeForTwo(d, c)
- case None => None
- })
+ // findWiderTypeForTwo doesn't satisfy the associative law, i.e. (a op
b) op c may not equal
+ // to a op (b op c). This is only a problem for StringType. Excluding
StringType,
+ // findWiderTypeForTwo satisfies the associative law. For instance,
(TimestampType,
+ // IntegerType, StringType) should have StringType as the wider common
type.
+ val (stringTypes, nonStringTypes) = types.partition(_ == StringType)
--- End diff --
Out of curiosity, does this work with array types too (array of string vs
array of non string types)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
