Jakub Stransky created MAPREDUCE-6016:
-----------------------------------------
Summary: hadoop yarn mapreduce skip failed records doesn't work
Key: MAPREDUCE-6016
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6016
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.2.0
Reporter: Jakub Stransky
Priority: Minor
I am trying to use "skip failed records" map-reduce functionality during the
map phase. I created special testing file with 8 corrupted records. I am using
TextInputFormat and during the processing (of the record) map function fails
with unhandled exception (parsing the record into expected structure). Job is
using the old mapred api.
My job settings for enabling "skip failed records feature":
<property>
<name>mapred.skip.mode.enabled</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.maxattempts</name>
<value>10</value>
</property>
<property>
<name>mapreduce.task.skip.start.attempts</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.skip.maxrecords</name>
<value>1</value>
</property>
I verified that those properties are propagated via verification in job.xml.
I am using hadoop 2.2.0 (HDP 2.0). Job is still failing after 10 attempts.
UPDATE:
- obviously job is not entering skip record mode
Q: Does this feature works on RecordReader level only? Hadoop definite guide
(which is for v.1) descibes thais feature at the level of map/reduce funciton
--
This message was sent by Atlassian JIRA
(v6.2#6252)
