Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11618#issuecomment-195472409
A couple of issues I hit when I try to do it.
1. The first issue is the one I submitted earlier regarding the `IsNotNull`
of compound expressions.
2. The second issue is the duplicate conditions could be added if we do not
change the existing rules, e.g., `PushPredicateThroughJoin`.
3. Another issue is about `a=b` && `a=c`, we can infer `b=c`. However, the
last one is useless and will burn extra costs. Ideally, we should select the
predicates based on the cardinality. However, that is missing.
Maybe more. Still trying to write more test cases. Hopefully, I can find
all of them.
Thank you! If you do not mind it, I will try it this weekend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
