[jira] [Commented] (YARN-2473) YARN never cleans up container directories from a full disk
[ https://issues.apache.org/jira/browse/YARN-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140798#comment-14140798 ] Varun Vasudev commented on YARN-2473: - [~jlowe], can you please review the latest patch for YARN-90([apache-yarn-90.4.patch|https://issues.apache.org/jira/secure/attachment/12669998/apache-yarn-90.4.patch])? It should handle the case you pointed out here as well. Thanks! YARN never cleans up container directories from a full disk --- Key: YARN-2473 URL: https://issues.apache.org/jira/browse/YARN-2473 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Assignee: Varun Vasudev Priority: Blocker After YARN-1781 when a container ends up filling a local disk the nodemanager will mark it as a bad disk and remove it from the list of good local dirs. When the container eventually completes the files that filled the disk will not be removed because the NM thinks the directory is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2473) YARN never cleans up container directories from a full disk
[ https://issues.apache.org/jira/browse/YARN-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115540#comment-14115540 ] Varun Vasudev commented on YARN-2473: - [~jlowe] are you going to work on this? I can take it up if it's fine by you. YARN never cleans up container directories from a full disk --- Key: YARN-2473 URL: https://issues.apache.org/jira/browse/YARN-2473 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Priority: Blocker After YARN-1781 when a container ends up filling a local disk the nodemanager will mark it as a bad disk and remove it from the list of good local dirs. When the container eventually completes the files that filled the disk will not be removed because the NM thinks the directory is bad. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2473) YARN never cleans up container directories from a full disk
[ https://issues.apache.org/jira/browse/YARN-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115556#comment-14115556 ] Varun Vasudev commented on YARN-2473: - My apologies for missing this when I put up the patch for YARN-1781 YARN never cleans up container directories from a full disk --- Key: YARN-2473 URL: https://issues.apache.org/jira/browse/YARN-2473 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Priority: Blocker After YARN-1781 when a container ends up filling a local disk the nodemanager will mark it as a bad disk and remove it from the list of good local dirs. When the container eventually completes the files that filled the disk will not be removed because the NM thinks the directory is bad. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2473) YARN never cleans up container directories from a full disk
[ https://issues.apache.org/jira/browse/YARN-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115580#comment-14115580 ] Jason Lowe commented on YARN-2473: -- No worries, Varun, we all missed it. ;-) We may need to track full disks separately from bad disks so we can know whether or not it's OK to try to delete a container directory from a particular disk that isn't a known good disk. I'm hesitant to have the NM try to remove container directories even from bad disks since touching them can cause a very long pause for the thread that did it. YARN never cleans up container directories from a full disk --- Key: YARN-2473 URL: https://issues.apache.org/jira/browse/YARN-2473 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.4.0 Reporter: Jason Lowe Priority: Blocker After YARN-1781 when a container ends up filling a local disk the nodemanager will mark it as a bad disk and remove it from the list of good local dirs. When the container eventually completes the files that filled the disk will not be removed because the NM thinks the directory is bad. -- This message was sent by Atlassian JIRA (v6.2#6252)
