[ 
https://issues.apache.org/jira/browse/SPARK-61573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61573:
-----------------------------------
    Labels: pull-request-available  (was: )

> Non-root LIMIT/OFFSET over a sorted input keeps arbitrary rows when a 
> generator sits between the sort and the limit
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61573
>                 URL: https://issues.apache.org/jira/browse/SPARK-61573
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Linhong Liu
>            Priority: Major
>              Labels: pull-request-available
>
> A LIMIT or OFFSET that is not the root of the query (e.g. inside CTAS, 
> INSERT, a UNION ALL branch or a join) over a global Sort is planned as 
> GlobalLimit <- Exchange SinglePartition <- LocalLimit <- Generate <- Sort. 
> The single-partition shuffle reader concatenates map outputs in fetch order, 
> so the limit can keep arbitrary rows. InsertSortForLimitAndOffset 
> (SPARK-50124) re-sorts after the shuffle, but 
> extractOrderingAndPropagateOrderingColumns does not walk through 
> GenerateExec, so the sort is not inserted when a generator (e.g. explode) 
> sits between the sort and the limit.
> Fix: let the rule walk through GenerateExec and propagate the ordering 
> columns, so the post-shuffle sort is inserted. A new conf 
> spark.sql.legacy.orderingAwareLimitOffset.ignoreGenerate (default false) 
> restores the previous behavior. Found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to