Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19389#discussion_r150682875
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -468,14 +460,16 @@ object PartitioningUtils {
}
/**
- * Given a collection of [[Literal]]s, resolves possible type conflicts
by up-casting "lower"
- * types.
+ * Given a collection of [[Literal]]s, resolves possible type conflicts
by
+ * [[TypeCoercion.findWiderCommonType]]. See
[[TypeCoercion.findWiderTypeForTwo]].
*/
private def resolveTypeConflicts(literals: Seq[Literal], timeZone:
TimeZone): Seq[Literal] = {
- val desiredType = {
- val topType =
literals.map(_.dataType).maxBy(upCastingOrder.indexOf(_))
--- End diff --
Both the values and the schemas could be changed. The external applications
might be broken if the schema is different.
The new behaviors are consistent with what we does for the other type
coercion cases. However, implicit type casting and partition discovery are
unstable. The other mature systems have clear/stable rules about it. Below is
an example.
https://docs.oracle.com/cloud/latest/db112/SQLRF/sql_elements002.htm#g195937
If each release introduces new behaviors, it becomes hard to use by the end
users who have such expectation. Thus, my suggestion is to first stabilize our
type coercion rules before addressing this.
https://github.com/apache/spark/pull/18853 is the first PR attempt in this
direction.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
