[
https://issues.apache.org/jira/browse/TEZ-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200619#comment-15200619
]
Bikas Saha commented on TEZ-3168:
---------------------------------
For all of the problems with queue capacity, IMO cluster capacity is a more
stable metric to look at.
Logically, the data is distributed across the cluster and so accounting for
that dispersion while calculating splits. This also solves the current
immediate problem of creating too small splits. Essentially the job wants to
run tasks across all cluster nodes. The queue capacity determines how the job
gets waves/windows of tasks that move around the cluster to read that data
locally.
> Provide a more predictable approach for total resource guidance for
> wave/split calculation
> -------------------------------------------------------------------------------------------
>
> Key: TEZ-3168
> URL: https://issues.apache.org/jira/browse/TEZ-3168
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Hitesh Shah
> Attachments: TEZ-3168.wip.2.patch, TEZ-3168.wip.patch
>
>
> Currently, Tez uses headroom for checking total available resources. This is
> flaky as it ends up causing the split count to be determined by a point in
> time lookup at what is available in the cluster. A better approach would be
> either the queue size or even cluster size to get a more predictable count.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
