[
https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626092#action_12626092
]
Hemanth Yamijala commented on HADOOP-4035:
------------------------------------------
A reasonable assumption to make while computing used capacity is to assume that
for all TTs in a cluster, the amount of memory per slot is configured to be the
same value. Note that this can be done even if the TTs themselves have
different hardware configurations (RAM, CPU slots, etc). This assumption is
also reasonable from the perspective that it helps users to easily verify if
they need to specify a higher limit for their jobs.
Based on this assumption, the total number of slots each task is virtually
taking is (number of running tasks for the job * the number of slots each task
of the job is taking). The latter is a something like (amount of per task
memory the job has requested / the memory per slot for the cluster).
Makes sense ?
> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory
> requirements and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4035
> URL: https://issues.apache.org/jira/browse/HADOOP-4035
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Vivek Ratan
> Fix For: 0.19.0
>
>
> HADOOP-3749 introduced configuration variables that can be used to specify
> memory requirements for jobs, and also modified the tasktrackers to report
> their free memory. The capacity scheduler in HADOOP-3445 should schedule
> tasks based on these parameters. A task that is scheduled on a TT that uses
> more than the default amount of memory per slot can be viewed as effectively
> using more than one slot, as it would decrease the amount of free memory on
> the TT by more than the default amount while it runs. The scheduler should
> make the used capacity account for this additional usage while enforcing
> limits, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
