[
https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v2.patch
fairly tested patch. few additional things:
- moved counters from Counters.java to SkipBadRecords.java as these are
specific to skip feature.
- fixed hasNext in SortedRanges.SkipRangeIterator.
- skipped records are not written to HDFS, if it is a test attempt (test
attempt is to figure out whether a range is good or bad. In this only records
in test range are passed to the mapper/reducer, others are skipped).
- recordreader.next not called beyond the test range during test attempt.
- renamed failedRanges to skipRanges in Task.java
- added config params to hadoop-default
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not
> acceptable to the user, then narrow down the skipped range to the user
> acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
