peter-toth opened a new pull request, #57765: URL: https://github.com/apache/spark/pull/57765
### What changes were proposed in this pull request? This PR adds a "Merging Subplans" section to `docs/sql-performance-tuning.md`, between "Join Strategy Hints" and "Adaptive Query Execution". Docs only, no code change. The section covers: - what `MergeSubplans` does and which subplans qualify (non-correlated deterministic scalar subqueries and non-grouping aggregates), with a small example that merges by default, - how a merged subplan looks in `EXPLAIN` (a subquery whose single output column is `mergedValue`, shared via `ReusedSubquery`), - when two subplans are merged, and that a query still carrying a non-inlined `WITH` clause is skipped, - why the one-sided filter case is always beneficial while the symmetric one is not, with the TPC-DS numbers measured in SPARK-40193 and SPARK-56677 and guidance on when to consider enabling it, - the 4 configs under `spark.sql.optimizer.mergeSubplans.filterPropagation.*` in a config table, and how to turn the rule off via `spark.sql.optimizer.excludedRules`. ### Why are the changes needed? The optimization and its configs are not described anywhere in the docs today. The 4 configs are public, so they show up in the generated SQL config table in `configuration.md`, but nothing tells users what the optimization does, which query shapes it applies to, or what the trade-offs of the 3 non-default configs are. The only other mention in `docs/` is the migration-guide note about the rule's package move. This matters most for symmetric filter propagation. It is disabled by default because merging two filtered scans widens the scan filter to `OR(f1, f2)`, which can lose partition or file pruning. But on queries that compute several differently filtered aggregates over the same table it is a big win: q9 and q28 got about 3.5x faster in TPC-DS runs, and together with propagation through joins q88 got about 7x and q90 about 2x faster. Users have no way to discover that today. The section also makes the plan shape discoverable: a merged subplan shows up as a subquery with a `mergedValue` column, which is otherwise hard to trace back to its source. ### Does this PR introduce _any_ user-facing change? No, documentation only. ### How was this patch tested? No tests, docs only. I verified every statement against the implementation on master: the config keys, defaults and versions against `SQLConf`, the merge conditions and the join nullability rules against `PlanMerger`, the skipped-CTE and candidate conditions against `MergeSubplans`, the `SCAN_MERGING` opt-in against `TableCapability` and its (test-only) implementors, and the rule name used in `excludedRules` against `Rule.ruleName` and the non-excludable rule list. The `mergedValue` output and the `ReusedSubquery` shape are cross-checked against the checked-in TPC-DS q9 golden plan, which also confirms that subplans sharing the same filter merge with the default configs. The added table follows the same `spark-config` markup as the other tables on the page. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
