[
https://issues.apache.org/jira/browse/MAPREDUCE-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065445#comment-14065445
]
Allen Wittenauer commented on MAPREDUCE-302:
--------------------------------------------
I think YARN sort of makes this JIRA obsolete, but I'd like some verification
before closing it.
> Maintaining cluster information across multiple job submissions
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-302
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-302
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Lohit Vijayarenu
> Assignee: dhruba borthakur
>
> Could we have a way to maintain cluster state across multiple job submissions.
> Consider a scenario where we run multiple jobs in iteration on a cluster back
> to back. The nature of the job is same, but input/output might differ.
> Now, if a node is blacklisted in one iteration of job run, it would be useful
> to maintain this information and blacklist this node for next iteration of
> job as well.
> Another situation which we saw is, if there are failures less than
> mapred.map.max.attempts in each iterations few nodes are never marked for
> blacklisting. But in we consider two or three iterations, these nodes fail
> all jobs and should be taken out of cluster. This hampers overall performance
> of the job.
> Could have have config variables something which matches a job type (provided
> by user) and maintains the cluster status for that job type alone?
--
This message was sent by Atlassian JIRA
(v6.2#6252)