[
https://issues.apache.org/jira/browse/SPARK-58562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth resolved SPARK-58562.
--------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57765
[https://github.com/apache/spark/pull/57765]
> Document subplan merging in the SQL performance tuning guide
> ------------------------------------------------------------
>
> Key: SPARK-58562
> URL: https://issues.apache.org/jira/browse/SPARK-58562
> Project: Spark
> Issue Type: Documentation
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> The MergeSubplans optimizer rule
> (org.apache.spark.sql.execution.planmerging.MergeSubplans) merges
> one-row-result subplans - scalar subqueries and non-grouping aggregates -
> that read the same input, so the input is scanned once instead of once per
> subplan. The rule is on by default and has 4 public configs under
> spark.sql.optimizer.mergeSubplans.filterPropagation.*, 3 of which are
> disabled by default and need a user decision.
> Today none of this is described in the docs. The configs only show up in the
> generated SQL config table in configuration.md, and the single mention
> anywhere else is the 4.3 migration-guide note about the rule's package move.
> In particular users have no way to discover that symmetric filter
> propagation, while off by default because OR-widening the merged filter can
> cost IO pruning, gave multi-fold speedups on TPC-DS query shapes that compute
> several differently-filtered aggregates over the same table.
> Add a section to docs/sql-performance-tuning.md that covers what the rule
> does, the query shapes it applies to, the configs and their tradeoffs, and
> how to disable the rule.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
