[
https://issues.apache.org/jira/browse/HADOOP-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616853#action_12616853
]
sharadag edited comment on HADOOP-153 at 7/25/08 4:45 AM:
----------------------------------------------------------------
We can have this Jira to address the base skipping functionality. I have
created two dependent Jiras HADOOP-3828 and HADOOP-3829 which are the
incremental features over this one.
was (Author: sharadag):
We can have this Jira to address the core skipping functionality. I have
created two dependent Jiras HADOOP-3828 and HADOOP-3829 which are the
incremental features over the core.
> skip records that fail Task
> ---------------------------
>
> Key: HADOOP-153
> URL: https://issues.apache.org/jira/browse/HADOOP-153
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.2.0
> Reporter: Doug Cutting
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 153_1.patch, 153_2.patch, 153_3.patch,
> skipRecords_wip1.patch
>
>
> MapReduce should skip records that throw exceptions.
> If the exception is thrown under RecordReader.next() then RecordReader
> implementations should automatically skip to the start of a subsequent record.
> Exceptions in map and reduce implementations can simply be logged, unless
> they happen under RecordWriter.write(). Cancelling partial output could be
> hard. So such output errors will still result in task failure.
> This behaviour should be optional, but enabled by default. A count of errors
> per task and job should be maintained and displayed in the web ui. Perhaps
> if some percentage of records (>50%?) result in exceptions then the task
> should fail. This would stop jobs early that are misconfigured or have buggy
> code.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
