[
https://issues.apache.org/jira/browse/SPARK-61296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Utkarsh Bajaj updated SPARK-61296:
----------------------------------
Description: CollectLimitExec moves all sorted partitions into one
partition/task through a shuffle, and then keeps the rows between OFFSET and
LIMIT. The shuffle hands back the partitions in the order they arrive, not in
partition order. So the rows can come back out of order, and OFFSET can skip
the wrong rows for order by queries. (was: CollectLimitExec moves all sorted
partitions into one partition/task through a shuffle, and then keeps the rows
between OFFSET and LIMIT. The shuffle hands back the partitions in the order
they arrive, not in partition order. So the rows can come back out of order,
and OFFSET can skip the wrong rows.)
> Return CollectLimitExec gather results in partition order
> ---------------------------------------------------------
>
> Key: SPARK-61296
> URL: https://issues.apache.org/jira/browse/SPARK-61296
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
> Affects Versions: 4.0.5
> Reporter: Utkarsh Bajaj
> Priority: Major
>
> CollectLimitExec moves all sorted partitions into one partition/task through
> a shuffle, and then keeps the rows between OFFSET and LIMIT. The shuffle
> hands back the partitions in the order they arrive, not in partition order.
> So the rows can come back out of order, and OFFSET can skip the wrong rows
> for order by queries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]