[GitHub] spark pull request #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoi...

[koertkuipers]

Github user koertkuipers commented on a diff in the pull request:

https://github.com/apache/spark/pull/17660#discussion_r111842598
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala 
---
@@ -124,7 +125,15 @@ case class EliminateOuterJoin(conf: SQLConf) extends 
Rule[LogicalPlan] with Pred
 val emptyRow = new GenericInternalRow(attributes.length)
 val boundE = BindReferences.bindReference(e, attributes)
 if (boundE.find(_.isInstanceOf[Unevaluable]).isDefined) return false
-val v = boundE.eval(emptyRow)
+val v = try {
+  boundE.eval(emptyRow)
--- End diff --

yeah sure i can do a scan for similar problems


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoi...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17660#discussion_r111834960
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala 
---
@@ -124,7 +125,15 @@ case class EliminateOuterJoin(conf: SQLConf) extends 
Rule[LogicalPlan] with Pred
 val emptyRow = new GenericInternalRow(attributes.length)
 val boundE = BindReferences.bindReference(e, attributes)
 if (boundE.find(_.isInstanceOf[Unevaluable]).isDefined) return false
-val v = boundE.eval(emptyRow)
+val v = try {
+  boundE.eval(emptyRow)
--- End diff --

Could you check whether there exists the other similar cases in the code 
base that could trigger `NullPointerException`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoi...

[koertkuipers]

GitHub user koertkuipers opened a pull request:

https://github.com/apache/spark/pull/17660

[SPARK-20359][SQL] catch NPE in EliminateOuterJoin optimization

catch NPE in EliminateOuterJoin and add test in DataFrameSuite to confirm 
NPE is no longer thrown

## What changes were proposed in this pull request?
catch possible NPE in this line
```val v = boundE.eval(emptyRow)```
and conclude the optimization can not be performed.

## How was this patch tested?

Added test in DataFrameSuite that failed before this fix and now succeeds. 
Note that a test in catalyst project would be better but i am unsure how to do 
this. Will look into it now.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tresata/spark 
feat-catch-npe-in-eliminate-outer-join

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17660.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17660


commit 7a759cca3fb302d55b5758e3e8cb85deca460112
Author: Koert Kuipers 
Date:   2017-04-17T00:11:00Z

catch NPE in EliminateOuterJoin and add test in DataFrameSuite to confirm 
NPE is no longer thrown




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org