[
https://issues.apache.org/jira/browse/YARN-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-4011.
------------------------------
Resolution: Duplicate
Yes, we ran into this a while ago, see YARN-2473 which was ultimately fixed by
YARN-90. There were a number of other fixes as well such as YARN-3850 and
YARN-3925. Closing this as a duplicate of YARN-90.
As for the ability to limit disk space, this has been discussed many times
before and as far back as MAPREDUCE-1100. An unsophisticated solution is to
have the container monitor that already is looking for too much memory usage
also monitor disk usage and kill the container if its too large. However this
doesn't solve the problem for containers that are writing to locations that
aren't container-specific (e.g.: where maps store their shuffle outputs, /tmp,
etc. I think it could be difficult to enforce tasks that are filling the disk
in arbitrary ways, but it could be straightforward to catch the task that
simply is logging too much.
> Jobs fail since nm-local-dir not cleaned up when rogue job fills up disk
> ------------------------------------------------------------------------
>
> Key: YARN-4011
> URL: https://issues.apache.org/jira/browse/YARN-4011
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.4.0
> Reporter: Ashwin Shankar
>
> We observed jobs failed since tasks couldn't launch on nodes due to
> "java.io.IOException No space left on device".
> On digging in further, we found a rogue job which filled up disk.
> Specifically it was wrote a lot of map spills(like
> attempt_1432082376223_461647_m_000421_0_spill_10000.out) to nm-local-dir
> causing disk to fill up, and it failed/got killed, but didn't clean up these
> files in nm-local-dir.
> So the disk remained full, causing subsequent jobs to fail.
> This jira is created to address why files under nm-local-dir doesn't get
> cleaned up when job fails after filling up disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
