[
https://issues.apache.org/jira/browse/MAPREDUCE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073550#comment-13073550
]
Robert Joseph Evans commented on MAPREDUCE-2324:
------------------------------------------------
I did initially look at trying to fix reduce.input.limit. Currently it is
something that someone has to manually guess what the value should be. What is
more this value is likely to need to change as the cluster fills up with data,
and as data is deleted off of the cluster. If it is wrong then either too many
jobs fail that would have succeeded or some jobs, probably a very small number,
will starve and never finish.
To fix it Hadoop would have to automatically set reduce.input.limit dynamically
and the only way I can think of to do that would be to gather statistics about
all the nodes in the cluster and try to predict how likely this particular
reduce will ever find the space it needs on a node. I believe that we can
compute the mean and X% confidence interval for disk space on the cluster
without too much difficulty but I have my doubts that this will apply to a
small cluster. From what I read anything under 40 samples tends to be suspect,
so it might not work for a cluster under 40 nodes. Also I am not sure how the
statistics would apply to this particular situation. Would we want to compute
this based off of a recent history of cluster of just a snapshot of its current
state? If there is history how far back would we want to go, and how would we
handle some nodes heart-beating in more regularly then others. I am not a
statistician and I could not find one to look over my work, so instead I
decided to take a bit more of a brute force approach that I know would work.
If you know a statistician that could provide a robust solution to this problem
or at least tell me what if anything I am doing wrong then I am very happy to
implement it.
> Job should fail if a reduce task can't be scheduled anywhere
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2324
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.20.205.0
> Reporter: Todd Lipcon
> Assignee: Robert Joseph Evans
> Fix For: 0.20.205.0
>
> Attachments: MR-2324-security-v1.txt, MR-2324-security-v2.txt,
> MR-2324-security-v3.patch, MR-2324-secutiry-just-log-v1.patch
>
>
> If there's a reduce task that needs more disk space than is available on any
> mapred.local.dir in the cluster, that task will stay pending forever. For
> example, we produced this in a QA cluster by accidentally running terasort
> with one reducer - since no mapred.local.dir had 1T free, the job remained in
> pending state for several days. The reason for the "stuck" task wasn't clear
> from a user perspective until we looked at the JT logs.
> Probably better to just fail the job if a reduce task goes through all TTs
> and finds that there isn't enough space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
