Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12736#discussion_r61379135
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala
---
@@ -488,14 +488,6 @@ class HiveTypeCoercionSuite extends PlanTest {
assert(r1.right.isInstanceOf[Project])
assert(r2.left.isInstanceOf[Project])
assert(r2.right.isInstanceOf[Project])
-
- val r3 = wt(Except(firstTable, firstTable)).asInstanceOf[Except]
- checkOutput(r3.left, Seq(IntegerType, DecimalType.SYSTEM_DEFAULT,
ByteType, DoubleType))
- checkOutput(r3.right, Seq(IntegerType, DecimalType.SYSTEM_DEFAULT,
ByteType, DoubleType))
-
- // Check if no Project is added
- assert(r3.left.isInstanceOf[LocalRelation])
- assert(r3.right.isInstanceOf[LocalRelation])
--- End diff --
We added a new condition in `resolved`:
https://github.com/apache/spark/pull/12736/files#diff-72917e7b68f0311b2fb42990e0dc616dR180
In this test case, we are trying to `Except` the same table `firstTable`.
Thus, the value of `resolved` becomes `false` now.
When we resolving `WidenSetOperationTypes` in
https://github.com/apache/spark/blob/c417cec067715455c1536d37e0dba97cc8657f7b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala#L230,
the whole `if` condition becomes true. (Before this PR, this condition is
always false if you try to except the same table. Thus, we did nothing in this
case. Thus, no `Project` is added)
However, when `if` condition becomes `true`, we execute the corresponding
logics. We are able to find the common type. Then, we will add `Project`. This
is doing the duplicate checking like what we did in
https://github.com/apache/spark/blob/d280d1da1aec925687a0bfb496f3a6e0979e896f/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercionSuite.scala#L479-L490.
That is why I removed it.
Sorry, it is a little bit complicated to explain the whole logics. Let me
know if anything is not clear
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
