HyukjinKwon commented on a change in pull request #26724: return tightest
common type when inferring schema
URL: https://github.com/apache/spark/pull/26724#discussion_r352332088
##########
File path: python/pyspark/sql/types.py
##########
@@ -1095,8 +1095,7 @@ def _merge_type(a, b, name=None):
elif isinstance(b, NullType):
return a
elif type(a) is not type(b):
- # TODO: type cast (such as int -> long)
- raise TypeError(new_msg("Can not merge type %s and %s" % (type(a),
type(b))))
Review comment:
I'm not sure about this for two reasons:
1. We should make a copy of
[TypeCoercion.scala](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala)
and duplicate logics here in Python and Scala. I am not sure if this is really
what we want.
2. Even if we find a common wide type, it might not work due to Pyrolite
type mapping between JVM and Python (see
[EvaluatePython.scala#L89-L202](https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala#L89-L202))
Are you able to make a minimised rule per
[EvaluatePython.scala#L89-L202](https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala#L89-L202)
and add a test for such combinations? Otherwise, I would like just close.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
