Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15041
This takes a heavy sort-based path well before it would cause an OOM it
seems though. This is a simple Utils method and I'm not sure it should be
causing spills to disk. If the problem is that you're running out of memory
taking 10M records to the driver, don't you just generally need more driver
memory? you're going to be in memory eventually on the driver when the result
is produced.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
