[
https://issues.apache.org/jira/browse/TEZ-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228365#comment-17228365
]
Rajesh Balamohan commented on TEZ-4246:
---------------------------------------
Can you share more details on this [~okumin] ? This should flip between 0 & 1
disks, it is quite possible that all data files are sent to a disk due to
flipping. But why would it create issues w.r.t to soft limit? (as it should
have same number of file.out and file.out.index).
> Avoid uneven local disk usage for spills
> ----------------------------------------
>
> Key: TEZ-4246
> URL: https://issues.apache.org/jira/browse/TEZ-4246
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: okumin
> Priority: Major
>
> This ticket would help a task attempt avoid overusing a specific disk.
>
> I have observed PipelinedSorter repeat spilling a large amount of data to one
> of two disks.
> In case that NodeManager has just two disks, they are basically selected in a
> round-robin fashion completely.
> [https://github.com/apache/hadoop/blob/rel/release-3.1.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java#L422-L439]
> Each iteration of a spill tries to create its data file and the index file,
> meaning that Tez is likely to put all data files on the same disk in such
> cases.
>
> This unfair usage is inconvenient especially when we use features with a soft
> limit like this.
> * https://issues.apache.org/jira/browse/TEZ-4112
>
> Index files are relatively small, and I'd say we can put a data file and its
> index file in the same directory so that the round-robin doesn't skip any
> disks for such small usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
