cloud-fan commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-495070324 I think @dilipbiswal has a good point here. For non-correlated EXISTS/IN, it's a bad idea to collect all the data of a table to the driver side and do the calculation. That said, we should not have a physical version of EXISTS/IN, they always need to be converted to join (sorry for the back and forth!). But we do have a chance to optimize non-correlated EXISTS/IN. More generally, if a left semi/anti join has a condition that only refers to attributes from the right side, we can probably turn this join into a filter operator.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
