[
https://issues.apache.org/jira/browse/YARN-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920735#comment-13920735
]
Sunil G commented on YARN-1781:
-------------------------------
Thank you for the reply. I have another point in addition to what I mentioned
as point 2.
3). To find the percentage of free space, below code is used.
float freePerc = testDir.getUsableSpace()/(float)testDir.getTotalSpace();
Percenatage will very much depend on total size avilable in different directory
partitions.
And if multiple directories are configured with different disk size, this value
may not come in same uniform manner.
So sizeRemaining in terms of few GB's or MB's may be a good measure. Like for
example, minimum 1GB free is needed to assign a directory.
Adding more information abount point 2).
Currently the check in LocalDirAllocator is to see whether "size" can meet the
"capacity" in "lastAccessed" directory.
In a scenario where Dir1 is assigned for a task to write 200 MB and may be
because of heavy usage another task also will be given with Dir1 for 100MB
immediately.
Now both these tasks will write 300MB. But when task 2 is given, it never
checked for what was assigned to task1. Only free space at that point is
chceked.
Last allotted space is not considered while assiging for next task. It is not
possible to consider as it is hard to predict disk write speed here.
So a free space check here also can help (can avoid disk full within 2 sec
healthcheck monitor) to avoid a disk full.
> NM should allow users to specify max disk utilization for local disks
> ---------------------------------------------------------------------
>
> Key: YARN-1781
> URL: https://issues.apache.org/jira/browse/YARN-1781
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Reporter: Varun Vasudev
> Assignee: Varun Vasudev
> Attachments: apache-yarn-1781.0.patch
>
>
> This is related to YARN-257(it's probably a sub task?). Currently, the NM
> does not detect full disks and allows full disks to be used by containers
> leading to repeated failures. YARN-257 deals with graceful handling of full
> disks. This ticket is only about detection of full disks by the disk health
> checkers.
> The NM should allow users to set a maximum disk utilization for local disks
> and mark disks as bad once they exceed that utilization. At the very least,
> the NM should at least detect full disks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
