peter-toth commented on PR #38640: URL: https://github.com/apache/spark/pull/38640#issuecomment-1312731989
Side note: There seem to be many issues with V2 Scan construction in batch `Early Filter and Projection Push-Down` currently. As that is the only place where the scans are constructed the batch seems to be: - too late for `PushDownLeftSemiAntiJoin` (see the issue above with some of the queries) - and too early for `RewritePredicateSubquery` + `ColumnPruning` in batch `RewriteSubquery` as obviuosly the scans should be (re)constructed after the latest `ColumnPruning` or filter modifying rule to not return unecessary data. (This actually causes serious performance degradation in q94, q16 with DSv2 sources.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
