Peter Toth created SPARK-60083:
----------------------------------
Summary: Leave sort orders over a partition transform out of the
logical comparisons of a DSv2 scan's reported ordering
Key: SPARK-60083
URL: https://issues.apache.org/jira/browse/SPARK-60083
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Peter Toth
SPARK-59995 drops every sort order that holds a partition transform from a V2
scan's physical output ordering ({{DataSourceV2ScanExecBase.outputOrdering}}).
The logical {{DataSourceV2ScanRelation.ordering}} still keeps them, and two
places compare it:
* {{PlanMerger.combineRequiredOrdering}} and {{mergeDegradesReporting}} decline
to merge two DSv2 scans whose reported orderings do not satisfy each other.
* {{DataSourceV2ScanRelation.doCanonicalize}} keeps the leading run of the
ordering in the canonical form, so it decides whether two scans count as the
same subplan.
Both compare sort orders by {{semanticEquals}}. For a transform, that depends
on {{BoundFunction.equals}}, which is only a SHOULD. So two scans that bind the
same transform function separately, without a semantic {{equals}}, never match.
For example, take a {{SCAN_MERGING}} table that reports {{[years(ts)]}} and
whose {{years}} function does not override {{equals}}. Two scalar subqueries
over it are not merged, unless
{{spark.sql.optimizer.mergeSubplans.dsv2ScanMerge.orderingDegradation.enabled}}
is on, although both physical scans now report {{[]}}. This is reasoned from
the code, not measured.
The sort orders these comparisons protect are no longer used by any physical
plan. Leaving them out of the logical comparisons too would remove the cost,
the way {{doCanonicalize}} already leaves out the sort orders past a pruned
column. A logical cut cannot just remove a transform sort order, though. The
sort orders after it hold only on partition keys, and only when the physical
scan reports a {{KeyedPartitioning}}, which depends on confs the logical rule
does not see. So the logical side would cut at the first transform, or the
comparisons would apply the physical rule.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
