[GitHub] spark pull request #16026: [SPARK-18597][SQL] Do push-down predicates to rig...

2016-11-27 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/16026#discussion_r89715298
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
 ---
@@ -514,6 +514,39 @@ class FilterPushdownSuite extends PlanTest {
 comparePlans(optimized, 
analysis.EliminateSubqueryAliases(correctAnswer))
   }
 
+  test("joins: push down where clause into left anti join") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val originalQuery =
+  x.join(y, LeftAnti, Some("x.b".attr === "y.b".attr))
+.where("x.a".attr > 10)
+.analyze
+val optimized = Optimize.execute(originalQuery)
+val correctAnswer =
+  x.where("x.a".attr > 10)
+.join(y, LeftAnti, Some("x.b".attr === "y.b".attr))
+.analyze
+comparePlans(optimized, 
analysis.EliminateSubqueryAliases(correctAnswer))
+  }
+
+  test("joins: only push down to the right of a left anti join") {
--- End diff --

Do we need a JIRA issue number here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16026: [SPARK-18597][SQL] Do push-down predicates to rig...

2016-11-27 Thread gatorsmile 
Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16026#discussion_r89713119
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
 ---
@@ -514,6 +514,39 @@ class FilterPushdownSuite extends PlanTest {
 comparePlans(optimized, 
analysis.EliminateSubqueryAliases(correctAnswer))
   }
 
+  test("joins: push down where clause into left anti join") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val originalQuery =
+  x.join(y, LeftAnti, Some("x.b".attr === "y.b".attr))
+.where("x.a".attr > 10)
+.analyze
+val optimized = Optimize.execute(originalQuery)
+val correctAnswer =
+  x.where("x.a".attr > 10)
+.join(y, LeftAnti, Some("x.b".attr === "y.b".attr))
+.analyze
+comparePlans(optimized, 
analysis.EliminateSubqueryAliases(correctAnswer))
+  }
+
+  test("joins: only push down to the right of a left anti join") {
--- End diff --

How about `joins: only push down join conditions to the right of a left 
anti join`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16026: [SPARK-18597][SQL] Do push-down predicates to rig...

2016-11-27 Thread hvanhovell 
GitHub user hvanhovell opened a pull request:

https://github.com/apache/spark/pull/16026

[SPARK-18597][SQL] Do push-down predicates to right side for a LEFT ANTI 
join

## What changes were proposed in this pull request?
We currently push down join predicates for a Left Anti join to both sides 
of the join if we can. This is similar to Inner, Left Semi and Existence 
(specialized left semi) join. The problem is that this changes the semantics of 
join; a left anti join filters out rows that matches the join condition.

This PR fixes this by only allowing the push down to the left hand side of 
the join. This is similar to the behavior of left outer join.

## How was this patch tested?
Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite 
file for left anti joins with a regression test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hvanhovell/spark SPARK-18597

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16026.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16026


commit 7696cf2cf67ed8ea541c0b8a90901f15ae393cbe
Author: Herman van Hovell 
Date:   2016-11-20T21:42:35Z

Fix 2.10 compilation

commit d51b63583670d274acf0fda2042915a29b174d2f
Author: Herman van Hovell 
Date:   2016-11-21T10:11:56Z

Merge remote-tracking branch 'apache-github/master'

commit 7ccb680ebe8a2939e0b81c31be6703c3927dc7bb
Author: Herman van Hovell 
Date:   2016-11-25T14:35:09Z

Merge remote-tracking branch 'apache-github/master'

commit e386fed62ad992eedc06e63aaeaf3524fff19b54
Author: Herman van Hovell 
Date:   2016-11-27T20:48:40Z

Merge remote-tracking branch 'apache-github/master'

commit 4d6b18361828e56fbf1e18c1b694eb318b0851f2
Author: Herman van Hovell 
Date:   2016-11-28T00:18:16Z

Do not change left anti join semantics by pushing down to the right side of 
the join.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org