shehabgamin commented on issue #27229: URL: https://github.com/apache/datafusion/issues/27229#issuecomment-6722487168
Thanks @alamb, I agree we need a general decorrelation pass, and I don't think these PRs work against that. Rather than adding support for new subquery shapes, they add join operators that a general pass has to lower into. The general decorrelation approach from [Unnesting Arbitrary Queries](https://dl.gi.de/handle/20.500.12116/2418) and [The Complete Story of Joins (in HyPer)](https://dl.gi.de/handle/20.500.12116/657) turns a decorrelated scalar subquery into a single join, and an `IN` into a mark join that tracks `NULL`s. DuckDB and Spark have these joins too. DataFusion doesn't have these joins in the form a general pass needs. It has no single join that errors when an outer row gets two matches. Its null-aware mark join can only build the hash table on the outer side. And its null-aware joins can't run partitioned. A general decorrelation pass would need all three before it could be added. The only piece that touches subquery support is a small change in #27228 that lets `ScalarSubqueryToJoin` use the flag, and it exists only so the single join can be covered by an sqllogictest integration test. A general pass would replace that rule anyway, so if you'd rather not add any new shapes to it, I'm happy to drop that part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
