Hieu Tri Huynh created SPARK-24992:
--------------------------------------
Summary: spark should randomize yarn local dir selection
Key: SPARK-24992
URL: https://issues.apache.org/jira/browse/SPARK-24992
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.3.1
Reporter: Hieu Tri Huynh
Utils.getLocalDir is used to get path of a temporary directory. However, it
always returns the the same directory, which is the first element in the array
_localRootDirs_. When running on YARN, this might causes the case that we
always write to one disk, which makes it busy while other disks are free. We
should randomize the selection to spread out the loads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
