[
https://issues.apache.org/jira/browse/SPARK-58950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58950:
-----------------------------------
Labels: pull-request-available (was: )
> Materialize repeated references to the same view as a single CTE
> ----------------------------------------------------------------
>
> Key: SPARK-58950
> URL: https://issues.apache.org/jira/browse/SPARK-58950
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: James Xu
> Priority: Major
> Labels: pull-request-available
>
> Problem:
> In ETL workloads, views are almost always costly: they encapsulate large
> scans, joins, and aggregations over big base tables. When such a view is
> referenced more than once in one query, the current plan inlines the view
> body at every reference site, so the costly body executes once per
> reference. Materializing the view once and sharing the result across all
> references avoids that repeated work.
> SELECT t1.user_id, COUNT(*)
> FROM serving_log_view t1 JOIN serving_log_view t2
> ON t1.user_id = t2.user_id
> WHERE t1.tier = 'priority'
> GROUP BY t1.user_id
> Here the single scan and the aggregation behind serving_log_view run twice,
> even though both references read identical data.
> Root Cause:
> View elimination substitutes each reference site with an independent copy of
> the body in place; optimization then treats the copies as unrelated
> subtrees, so their cost is replicated with no dedup opportunity.
> Solution:
> Add an optimizer rule that runs in FinishAnalysis, right before view
> elimination. It groups occurrences of the same view by catalog identity and
> canonicalized body, then rewrites each qualifying group into a single
> force-materialized CTE definition plus one reference per occurrence. The
> body is computed once, and its shuffle is shared by all references through
> exchange reuse.
> The rule is conservative: only deterministic, non-streaming views convert;
> all occurrences must agree on the view SQL configuration and output schema;
> views nested inside other views convert too, with definitions in topological
> order; bodies with internally correlated subqueries are safe because view
> analysis guarantees those references resolve inside the body. The rule is
> gated behind a new configuration flag, off by default.
> Expected Impact:
> For a view referenced N times, the body is evaluated once instead of N
> times - for a self-joined view whose body dominates the query, scan and
> compute replication drop from N to 1, at the cost of one shuffle boundary
> per reference site, which exchange reuse deduplicates. Estimated; concrete
> numbers to follow in the PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
