[
https://issues.apache.org/jira/browse/MAPREDUCE-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092179#comment-16092179
]
Jason Lowe commented on MAPREDUCE-6916:
---------------------------------------
Slow or temporarily unavailable namenodes and slow datanodes are examples of
things that can cause one or both of these tasks to execute more slowly than it
should. We want to run these tasks at a fixed rate, but we don't want to
"catch up" any missing task executions if we fall behind since there's no point
to that. A better design would be to have the tasks that are actually
scheduled with the executor to not do the full job but simply notify another
thread that is responsible for doing the task. That way the executor tasks are
quick and should never back up in practice, and we won't end up
double-executing if we notify more than once while the task is still running.
> History server scheduling tasks at fixed rate can be problematic when those
> tasks are slow
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6916
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6916
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 2.7.4
> Reporter: Jason Lowe
>
> The job history server currently schedules both the task of moving jobs from
> intermediate to done and the task of cleaning jobs at a fixed rate. If those
> tasks take longer than the rate period to execute then a backlog of
> to-be-scheduled tasks can build up and cause a long storm of them to execute
> later when the blockage clears.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
