[
https://issues.apache.org/jira/browse/SPARK-61573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61573:
-----------------------------------
Labels: pull-request-available (was: )
> Non-root LIMIT/OFFSET over a sorted input keeps arbitrary rows when a
> generator sits between the sort and the limit
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61573
> URL: https://issues.apache.org/jira/browse/SPARK-61573
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Linhong Liu
> Priority: Major
> Labels: pull-request-available
>
> A LIMIT or OFFSET that is not the root of the query (e.g. inside CTAS,
> INSERT, a UNION ALL branch or a join) over a global Sort is planned as
> GlobalLimit <- Exchange SinglePartition <- LocalLimit <- Generate <- Sort.
> The single-partition shuffle reader concatenates map outputs in fetch order,
> so the limit can keep arbitrary rows. InsertSortForLimitAndOffset
> (SPARK-50124) re-sorts after the shuffle, but
> extractOrderingAndPropagateOrderingColumns does not walk through
> GenerateExec, so the sort is not inserted when a generator (e.g. explode)
> sits between the sort and the limit.
> Fix: let the rule walk through GenerateExec and propagate the ordering
> columns, so the post-shuffle sort is inserted. A new conf
> spark.sql.legacy.orderingAwareLimitOffset.ignoreGenerate (default false)
> restores the previous behavior. Found via correctness analysis.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]