Github user davies commented on the issue:
https://github.com/apache/spark/pull/13887
@dongjoon-hyun Thanks for the patch, this optimization sounds reasonable.
I'm thinking of is it possible to make the optimization for IN/INSET more
general. We could have a optimizer to insert a GreaterThanOrEqual and
LessThanOrEqual for IN/INSET (checking the data type to make sure they are
orderable), it will be cheaper to evaluate. For IN with single expression, we
could rewrite as EqualTo. When doing this, we should be careful for null, to
respect the sematics of null in IN/INSET. By doing this, all the data sources
could benefit from it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
