[
https://issues.apache.org/jira/browse/SPARK-60043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60043:
-----------------------------------
Labels: pull-request-available (was: )
> Keep the sort orders after a partition-key transform in a V2 scan's reported
> ordering
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60043
> URL: https://issues.apache.org/jira/browse/SPARK-60043
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> SPARK-59995 drops every sort order that holds a partition transform from a V2
> scan's output ordering. In a reported ordering, such a sort order also ends
> the leading run of sort orders that the scan keeps. A sort order on a
> transform that is itself a partition key is constant within each partition,
> like any partition key, so the sort orders after it still hold. They are
> dropped anyway:
> * keys {{[days(ts)]}}, reported {{[days(ts), id]}}: the scan reports {{[]}}
> instead of {{[id]}};
> * keys {{[years(ts), id]}}, reported {{[years(ts), id, name]}}: {{[id]}}
> instead of {{[id, name]}};
> * keys {{[days(ts), id]}}, reported {{[days(ts)]}}: {{[]}}, while no report
> would derive {{[id]}}.
> Skipping such a sort order instead of ending the run would keep them.
> Matching a transform sort order to a transform key needs {{isSameFunction}}
> plus semantically equal children rather than {{ExpressionSet}}, since the two
> are bound separately and {{BoundFunction.equals}} is only a SHOULD. Falling
> back to the derived ordering when nothing of a non-empty report survives
> would cover the last case.
> This is not a regression, since none of these orderings satisfied a
> requirement before SPARK-59995. Measured on the SPARK-59995 PR: two tables
> that report {{[years(ts), ts]}} over the key {{years(ts)}}, joined on {{ts}}
> with {{preserveOrderingOnCoalesce}} on, get 2 {{SortExec}}s. Skipping the
> key's sort order removes both, with correct rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]