James Xu created SPARK-58950:
--------------------------------
Summary: Materialize repeated references to the same view as a
single CTE
Key: SPARK-58950
URL: https://issues.apache.org/jira/browse/SPARK-58950
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: James Xu
Problem:
In ETL workloads, views are almost always costly: they encapsulate large
scans, joins, and aggregations over big base tables. When such a view is
referenced more than once in one query, the current plan inlines the view
body at every reference site, so the costly body executes once per
reference. Materializing the view once and sharing the result across all
references avoids that repeated work.
SELECT t1.user_id, COUNT(*)
FROM serving_log_view t1 JOIN serving_log_view t2
ON t1.user_id = t2.user_id
WHERE t1.tier = 'priority'
GROUP BY t1.user_id
Here the single scan and the aggregation behind serving_log_view run twice,
even though both references read identical data.
Root Cause:
View elimination substitutes each reference site with an independent copy of
the body in place; optimization then treats the copies as unrelated
subtrees, so their cost is replicated with no dedup opportunity.
Solution:
Add an optimizer rule that runs in FinishAnalysis, right before view
elimination. It groups occurrences of the same view by catalog identity and
canonicalized body, then rewrites each qualifying group into a single
force-materialized CTE definition plus one reference per occurrence. The
body is computed once, and its shuffle is shared by all references through
exchange reuse.
The rule is conservative: only deterministic, non-streaming views convert;
all occurrences must agree on the view SQL configuration and output schema;
views nested inside other views convert too, with definitions in topological
order; bodies with internally correlated subqueries are safe because view
analysis guarantees those references resolve inside the body. The rule is
gated behind a new configuration flag, off by default.
Expected Impact:
For a view referenced N times, the body is evaluated once instead of N
times - for a self-joined view whose body dominates the query, scan and
compute replication drop from N to 1, at the cost of one shuffle boundary
per reference site, which exchange reuse deduplicates. Estimated; concrete
numbers to follow in the PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
