Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/14411
@hvanhovell,
Thank you for your comment. There are quite a few patterns being
blacklisted already, such as correlation under set operators (UNION, EXCEPT,
INTERSECT), correlation outside of WHERE/HAVING context, correlation in the
right table of a LEFT [OUTER] JOIN (and the left table of a RIGHT [OUTER]
JOIN]). I am working on discovering more issues in this area but it looks like
a bigger project to me. I have a general idea that the rewrite of correlation
subquery to join should not happen in the Analysis phase. We should build a
Logical plan to represent the subquery and perform the rewrite at the Optimizer
phase instead.
I am new to the Spark code and this is my first PR. So I'd like to make it
a small, self-contained project to gain my confidence in working with the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
