Linhong Liu created SPARK-61573:
-----------------------------------

             Summary: Non-root LIMIT/OFFSET over a sorted input keeps arbitrary 
rows when a generator sits between the sort and the limit
                 Key: SPARK-61573
                 URL: https://issues.apache.org/jira/browse/SPARK-61573
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Linhong Liu


A LIMIT or OFFSET that is not the root of the query (e.g. inside CTAS, INSERT, 
a UNION ALL branch or a join) over a global Sort is planned as GlobalLimit <- 
Exchange SinglePartition <- LocalLimit <- Generate <- Sort. The 
single-partition shuffle reader concatenates map outputs in fetch order, so the 
limit can keep arbitrary rows. InsertSortForLimitAndOffset (SPARK-50124) 
re-sorts after the shuffle, but extractOrderingAndPropagateOrderingColumns does 
not walk through GenerateExec, so the sort is not inserted when a generator 
(e.g. explode) sits between the sort and the limit.

Fix: let the rule walk through GenerateExec and propagate the ordering columns, 
so the post-shuffle sort is inserted. A new conf 
spark.sql.legacy.orderingAwareLimitOffset.ignoreGenerate (default false) 
restores the previous behavior. Found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to