andygrove opened a new issue, #2758: URL: https://github.com/apache/datafusion-ballista/issues/2758
> **This is a research task.** Spark AQE is the reference point for the problem, not for the solution. Ballista has different constraints than Spark (DataFusion operators, stage-at-a-time replanning, no external shuffle service, partition-count-sensitive operators), so we may not want to copy how Spark does this. The first deliverable is a write-up of whether and how to do it here, ideally with a measurement showing the gain, before any implementation. ## Problem When a stage completes, AQE replans the remaining plan and adopts the result unconditionally. There is no check that the new plan is no worse than the old one. #2024 shows AQE making join-heavy queries slower than the static planner, and other issues (#2454, #2661) are cases where a replan made a worse or invalid choice from partial or misleading statistics. Spark AQE guards against this with a cost evaluator (`SimpleCostEvaluator`, pluggable through `spark.sql.adaptive.customCostEvaluatorClass`). After re-optimizing, it adopts the new plan only if its cost, by default the number of shuffles, is no higher than the current plan's. The skew-join rule gets an exemption for the extra shuffles it deliberately adds. ## Questions to answer - Is a guard of this kind worth having in Ballista? Which past regressions (#2024, #2454, #2661, #2321) would one have caught, and which would it have wrongly blocked? - What should "cost" be here? Shuffle count is Spark's proxy. Other candidates are estimated shuffled bytes, the number of tasks, or the presence of a single-task stage (such as a `CollectLeft` join pinned to one task). - Rules in Ballista decide locally, for example `SelectJoinRule` per join. Would a plan-level accept/reject be cleaner than making each rule defensive, or just hide bugs? - What does rejecting a replan mean when the old plan is stale because a stage already completed? - Should an embedder be able to supply its own evaluator through `AdaptiveRules`? ## Related - #2024, #2454, #2661, #2321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
