Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19451#discussion_r144934607
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceOperatorSuite.scala
---
@@ -50,6 +52,42 @@ class ReplaceOperatorSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ test("replace Except with Filter while both the nodes are of type
Filter") {
+ val attributeA = 'a.int
+ val attributeB = 'b.int
+
+ val table1 = LocalRelation.fromExternalRows(Seq(attributeA,
attributeB), data = Seq(Row(1, 2)))
+ val table2 = Filter(attributeB === 2, Filter(attributeA === 1, table1))
+ val table3 = Filter(attributeB < 1, Filter(attributeA >= 2, table1))
+
+ val query = Except(table2, table3)
+ val optimized = Optimize.execute(query.analyze)
+
+ val correctAnswer =
+ Aggregate(table1.output, table1.output,
+ Filter(Not(attributeA >= 2 && attributeB < 1),
--- End diff --
This is not a correct answer, right? We also need to consider NULL
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
