[
https://issues.apache.org/jira/browse/SPARK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14098785#comment-14098785
]
Davies Liu commented on SPARK-3073:
-----------------------------------
This is for PySpark, currently we do not support large data sets in reduce
stage during sortBy() or sortByKey().
This also will be useful for groupByKey() with hot keys. (the memory can not
hold one hot key).
> improve large sort (external sort) for PySpark
> ----------------------------------------------
>
> Key: SPARK-3073
> URL: https://issues.apache.org/jira/browse/SPARK-3073
> Project: Spark
> Issue Type: Improvement
> Reporter: Davies Liu
> Assignee: Davies Liu
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
