[
https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472972
]
Riccardo commented on HADOOP-1014:
----------------------------------
Same problem here. I have a bunch of M/R jobs that process some 100Gb of text
data and collect different types of stats. Since Hadoop 0.10.0 I started
noticing that some of the reduce outputs were missing data (some of them were
empty). I modified the TestMapRed() JUnit to run the following test. I create
two separate sequence files by generating N random integers and storing them in
one file as IntWritable() values and as Text() writables in the other one.
Matching integers have the same key. Then I ran the M/R job with an identity
mapper (I use GenericWritable() to wrap the different value types) and in the
reducer I check if both the integer and the string representation are received
for a given key. The job runs just fine until up about 50M keys, then it starts
failing for 100M keys or more. The assertion exception is caused by the reducer
receiving only one value not both. One problem is that the job does not fail
consistently and it always throws the assertion exceptions for different keys
and within different reduce tasks. Our M/R reduce cluster is composed of 10
high-end dual-core linux boxes with about 8Tb of aggregate HD capacity. I
cannot get this JUnit (or any of my jobs) to fail with Hadoop 0.9.2. They
actually run on it like a breeze.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the
> reduce. This looks to be a blocker until it is resolved. There were two
> relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
