[
https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-1943:
-------------------------------------
Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
this patch is an addendum to the last patch. This fixes a bug wherein counters
arent counted across tasks before the job completes. This patch updates the
numer of counters on every heartbeat for the job and kill the job in case it
exceeds the limit.
> Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1943
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1943-0.20-yahoo.patch,
> MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch,
> MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
>
>
> We have come across issues in production clusters wherein users abuse
> counters, statusreport messages and split sizes. One such case was when one
> of the users had 100 million counters. This leads to jobtracker going out of
> memory and being unresponsive. In this jira I am proposing to put sane limits
> on the status report length, the number of counters and the size of block
> locations returned by the input split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
