Github user bersprockets commented on a diff in the pull request:
https://github.com/apache/spark/pull/21073#discussion_r199678852
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -551,6 +551,36 @@ object TypeCoercion {
case None => s
}
+ case m @ MapConcat(children) if children.forall(c =>
MapType.acceptsType(c.dataType)) &&
+ !haveSameType(children) =>
+ val keyTypes =
children.map(_.dataType.asInstanceOf[MapType].keyType)
--- End diff --
I don't necessarily have a _good_ reason, but here are two reasons I did
that extra junk:
1) The Concat-like code didn't find a wider type amongst types
map<tinyint,tinyint> and map<int,int>. So, it just fell to <code>case None =>
m</code>
2) If the call to map_concat has the same child types but multiple
valueContainsNull values, the Concat-style code added a Cast to each child
(this is because haveSameType considers expressions with different
valueContainsNull values to have different types). It does no harm, as far as I
can tell, but it seemed wrong.
About issue 1): I will debug. I might have done something wrong there.
Plus, even if it's a real bug in findWiderCommonType, it affects my longer
code, which may be looking for a wider common type amongst the keys or values,
which could themselves be maps.
About issue 2): Maybe not an issue. Or I can create an alternate
haveSameType() function for Maps.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
