[
https://issues.apache.org/jira/browse/MAPREDUCE-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783702#action_12783702
]
rahul k singh commented on MAPREDUCE-763:
-----------------------------------------
Approach :
For every job with higher memory requirement we reserve a TT , now if a task
from the job is assigned to a unreserved tasktracker, we
simply remove a tasktracker from the existing reserved TT list. This would
make sure that we do not reserve more than required.
JobInProgress maintains the list of TTs reserved for map and reduce, we can
use the same , TT removed will just be first element in the list.
The above solution has a small starvation issue incase of speculative
tasks.That is because speculative tasks cannot run on certain kind of TTs. For
example:
There is a job with 3 tips namely tip1 , tip2 , tip3.
for attempt tip1_1 we have reserved TT1.
for attempt tip2_1 we have reserved TT2.
for attempt tip3_1 we have reserved TT3.
In the above case if tip1_1 is assigned to TT4 then we simply unreserve TT1 as
it is at the top of list. Now if there is second attempt
tip1_2 for tip1 , and if it cannot run on TT2 or TT3, there can be slight
starvation for tip1_2 as it has to wait till it gets a TT where it could
run.But this is kind of ok , as this is comparatively remote case and list of
TTs is a dynamic structure. In order to make above work correctly we would need
attempt id , reserved tt mapping(Any other suggestions?) . This would require
some significant changes.
The above approach would also make sure that code changes are simple and
straight forward . It definitly alleviate the current situation, where chances
of going overboard with reservation is relatively higher.
> Capacity scheduler should clean up reservations if it runs tasks on nodes
> other than where it has made reservations
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-763
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-763
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
> Fix For: 0.21.0
>
>
> Currently capacity scheduler makes a reservation on nodes for high memory
> jobs that cannot currently run at the time. It could happen that in the
> meantime other tasktrackers become free to run the tasks of this job. Ideally
> in the next heartbeat from the reserved TTs the reservation should be
> removed. Otherwise it could unnecessarily block capacity for a while (until
> the TT has enough slots free to run a task of this job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
