[
https://issues.apache.org/jira/browse/SPARK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davies Liu updated SPARK-3153:
------------------------------
Description:
If we have several disks in SPARK_LOCAL_DIRS, and one of them is much smaller
than others (maybe added in my mistake, or special disk, SSD), them the shuffle
will meet the problem of run out of space with this smaller disk.
PySpark also has this issue during spilling.
was:If we have several disks in SPARK_LOCAL_DIRS, and one of them is much
smaller than others (maybe added in my mistake, or special disk, SSD), them the
shuffle will meet the problem of run out of space with this smaller disk.
> shuffle will run out of space when disks have different free space
> ------------------------------------------------------------------
>
> Key: SPARK-3153
> URL: https://issues.apache.org/jira/browse/SPARK-3153
> Project: Spark
> Issue Type: Bug
> Reporter: Davies Liu
>
> If we have several disks in SPARK_LOCAL_DIRS, and one of them is much smaller
> than others (maybe added in my mistake, or special disk, SSD), them the
> shuffle will meet the problem of run out of space with this smaller disk.
> PySpark also has this issue during spilling.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
