Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/8983#issuecomment-145696762
Thanks Wenchen. You are right that not all types can be casted to boolean.
However, in this case, we are not trying to cast the in list types to the
LHS type (null type in our case) as we know that this is a special case
predicate would always evaluate to NULL. That is why we are simply transforming
the in predicate to NULL one and dropping the in list altogether.
== Parsed Logical Plan ==
'Project [unresolvedalias(*)]
'Filter NOT null IN (1,2,3,4) => original one
'UnresolvedRelation [inttab], None
== Analyzed Logical Plan ==
c1: int
Project [c1#0]
Filter NOT null => rewritten one
Subquery inttab
LogicalRDD [c1#0], MapPartitionsRDD[4] at rddToDataFrameHolder at
<console>:26
Please let me know what you think .. If you have a test case in mind that
would exhibit a problem then i would like to try it out.
Thanks a lot for your help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
