Peter Toth created SPARK-58549:
----------------------------------
Summary: Preserve key-grouped partitioning and ordering across a
DSv2 scan merge
Key: SPARK-58549
URL: https://issues.apache.org/jira/browse/SPARK-58549
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
Follow-up to SPARK-40259 (subquery plan merge for DataSource V2 scans).
Today the DSv2 scan merge declines whenever either input scan reports
key-grouped partitioning or ordering: the rebuilt merged scan carries neither,
so fusing the two scans could drop a partitioning/ordering the original plan
relied on and force an extra shuffle or sort. Declining is safe but leaves the
merge on the table for any partitioned/ordered source (e.g. an SPJ join sitting
inside the merged subplan).
This change lets the merge proceed and re-derives the merged scan's own report
instead of declining up front:
- Drop the kGP/ordering conjuncts from the mergeable gate.
- At the leaf, combine the two inputs' reports (remapped into the merged
relation's attribute space) into the single report the merge must preserve: kGP
must be equal, ordering is the stronger of the two. If the inputs are
incompatible (differing non-empty kGP, or neither ordering satisfies the other)
no rebuilt scan can keep both not-worse, so decline right there, before
rebuilding.
- After rebuilding the merged scan, re-derive its partitioning/ordering
(V2ScanPartitioningAndOrdering) and decline only if that degrades the required
report.
Two new opt-in configs gate accepting a degradation (default false, so the
default behavior is a pure improvement: merge when not worse, decline on loss):
-
spark.sql.optimizer.mergeSubplans.dsv2ScanMerge.allowKeyGroupedPartitioningDegradation
- spark.sql.optimizer.mergeSubplans.dsv2ScanMerge.allowOrderingDegradation
Only affects sources that declare the SCAN_MERGING table capability. No
behavior change for existing sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
