andygrove opened a new issue, #2761: URL: https://github.com/apache/datafusion-ballista/issues/2761
> **This is a research task.** Spark AQE is the reference point for the problem, not for the solution. Ballista has different constraints than Spark (DataFusion operators, stage-at-a-time replanning, no external shuffle service, partition-count-sensitive operators), so we may not want to copy how Spark does this. The first deliverable is a write-up of whether and how to do it here, ideally with a measurement showing the gain, before any implementation. ## Problem Spark's AQE optimizer re-runs a small set of logical rules on the remaining plan once a query stage has materialized, using exact row counts as input. Besides empty-relation propagation, the batches include rules such as `EliminateLimits` (drop a limit above a stage that already has no more rows than the limit) and `OptimizeOneRowPlan` (simplify sorts and aggregates over a plan known to produce at most one row). Ballista's `PropagateEmptyExecRule` covers the zero-row case, and the coalesce, join-selection and skew rules use measured bytes. Nothing uses an exact *small* row count (1 row, or fewer rows than a limit) to simplify downstream operators. Scalar subqueries, global aggregates without a group key and `LIMIT`/top-N stages over a small input are the usual cases. They can leave a final `SortExec`, `GlobalLimitExec` or `AggregateExec` plus its exchange in place for data that is already a single row. ## Questions to answer - Do the exact row counts the stage reports reach the planner reliably enough to base a rewrite on (versus estimates)? See #1960 for the column-statistics side. - Which rewrites pay off in Ballista? Candidates are dropping a limit when stage rows are at most the fetch, collapsing a final aggregate over a one-row input, and skipping a downstream shuffle whose input is a single row. Some may already be handled by DataFusion's own optimizer once it sees exact statistics. - Is this better done as a physical rule over `ExchangeExec` statistics (like `PropagateEmptyExecRule`), or by feeding exact statistics back into DataFusion's existing rules? - How much does it save? Most queries will gain little. The research should find which workloads, if any, benefit before adding rules. ## Related - #1960, #2665 (statistics surfaced to AQE), the empty-propagation rule, #1989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
