Andrew Ash created SPARK-5997:
---------------------------------
Summary: Increase partition count without performing a shuffle
Key: SPARK-5997
URL: https://issues.apache.org/jira/browse/SPARK-5997
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Andrew Ash
When decreasing partition count with rdd.repartition() or rdd.coalesce(), the
user has the ability to choose whether or not to perform a shuffle. However
when increasing partition count there is no option of whether to perform a
shuffle or not.
This Jira is to create a {{rdd.repartition(largeNum, shuffle=false)}} call that
performs a repartition to a higher partition count without a shuffle.
The motivating use case is to decrease the size of an individual partition
enough that the .toLocalIterator has significantly reduced memory pressure on
the driver, as it loads a partition at a time into the driver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
