viirya commented on PR #58272: URL: https://github.com/apache/spark/pull/58272#issuecomment-5560877884
Could we first consider moving the statistics-dependent part of `PushDownLeftSemiAntiJoin` after scan pushdown, rather than introducing a new public connector API to accommodate its current position? In particular, the Aggregate branch consults `canPlanAsBroadcastHashJoin`, while the other pushdown cases do not require this statistics-based decision. Could we separate that branch and run it once scan statistics are available? This would also align with the existing optimizer comment that rules relying on statistics should run after the “Early Filter and Projection Push-Down” batch. I am also concerned about whether full-table catalog statistics are suitable for this decision. The broadcast condition was introduced in SPARK-34081 to avoid performance regressions from pushing a join below an Aggregate. Table-level statistics may overestimate a heavily filtered scan and prevent a useful rewrite, or underestimate its size and permit a rewrite that becomes expensive. Later scan statistics may change the physical join strategy, but do not necessarily undo the earlier logical rewrite. Table-level and scan-level statistics can be useful concepts independently. However, the motivation here appears tied to Spark’s internal rule ordering. Before adding another interface for connector authors to implement and maintain, could we evaluate whether splitting or deferring this rule resolves the issue using the existing `SupportsReportStatistics` API? Moving the rule may affect aggregate/join pushdown opportunities, so this would need plan tests and performance validation. If those interactions make early statistics necessary, a concrete example demonstrating that tradeoff would help justify the new API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
