Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145948447
@cloud-fan , please confirm my understanding of the code (fairly new to the
codebase..:-)
In the code we go through the entire in list and run evaluate the
expression flagging hasNull. But
we continue with next items and return true if we see a match. If we
haven't seen it then we look at the hasNull flag and return Null or False.
To confirm if there is a issue, i tried to run the following two queries
again. The output looks
ok to me..
select * from inttab where 1 in (1,2,NULL)
var2: org.apache.spark.sql.DataFrame = [c1: int]
+---+
| c1|
+---+
| 1|
| 2|
| 3|
| 4|
| 5|
+---+
== Parsed Logical Plan ==
'Project [unresolvedalias(*)]
'Filter 1 IN (1,2,null)
'UnresolvedRelation [inttab], None
== Analyzed Logical Plan ==
c1: int
Project [c1#0]
Filter 1 IN (cast(1 as int),cast(2 as int),cast(null as int))
Subquery inttab
LogicalRDD [c1#0], MapPartitionsRDD[4] at rddToDataFrameHolder at
<console>:26
== Optimized Logical Plan ==
LogicalRDD [c1#0], MapPartitionsRDD[4] at rddToDataFrameHolder at
<console>:26
== Physical Plan ==
Scan PhysicalRDD[c1#0]
Code Generation: true
select * from inttab where 1 in (NULL,1,2)
var2: org.apache.spark.sql.DataFrame = [c1: int]
+---+
| c1|
+---+
| 1|
| 2|
| 3|
| 4|
| 5|
+---+
== Parsed Logical Plan ==
'Project [unresolvedalias(*)]
'Filter 1 IN (null,1,2)
'UnresolvedRelation [inttab], None
== Analyzed Logical Plan ==
c1: int
Project [c1#0]
Filter 1 IN (cast(null as int),cast(1 as int),cast(2 as int))
Subquery inttab
LogicalRDD [c1#0], MapPartitionsRDD[4] at rddToDataFrameHolder at
<console>:26
== Optimized Logical Plan ==
LogicalRDD [c1#0], MapPartitionsRDD[4] at rddToDataFrameHolder at
<console>:26
== Physical Plan ==
Scan PhysicalRDD[c1#0]
Code Generation: true
Please let me know your thoughts ..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
