Utkarsh Bajaj created SPARK-61296:
-------------------------------------
Summary: Return CollectLimitExec gather results in partition order
Key: SPARK-61296
URL: https://issues.apache.org/jira/browse/SPARK-61296
Project: Spark
Issue Type: Bug
Components: Shuffle, Spark Core
Affects Versions: 4.0.5
Reporter: Utkarsh Bajaj
CollectLimitExec moves all sorted partitions into one partition/task through a
shuffle, and then keeps the rows between OFFSET and LIMIT. The shuffle hands
back the partitions in the order they arrive, not in partition order. So the
rows can come back out of order, and OFFSET can skip the wrong rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
