Partioning Failing in certain scenarios for Hadoop Streaming using Ruby
-----------------------------------------------------------------------
Key: MAPREDUCE-3924
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3924
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2
Environment: Linux OS
44 Node cluster
Hadoop version 0.20.3-cdhu3 is the version
Reporter: Subir S
We noticed a wierd scenario. The partitioning option fails in scenario below.
The map reduce just aggregates values like a word count. However the key is
having 2 parts.
Eg:
Input Records: (<key1|key2><tab><qty1|amt1|qty2|amt2|qty3|amt3>
1201|420 1|24.0|2|26.0|0|0.0
1200|420 1|24.0|2|26.0|0|0.0
1200|420 1|25.0|3|52.0|1|55.0
1403|400 1|25.0|3|52.0|1|55.0
1201|420 1|24.0|2|26.0|0|0.0
1201|420 1|24.0|2|26.0|0|0.0
1403|400 1|25.0|3|52.0|1|55.0
Partioning option: -k1,1 and Comparator option -k1,1n -k2,2n works fine and the
output we get is expected output
1200|420 2|49.0|5|78.0|1|55.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0
However if we use
Partioning option: -k1,1 -k2,2 and Comparator option -k1,1n -k2,2n then the
output is faulty. Seems the same key does not go to same reducer in output we
see duplicate records, however this does not happen to all records. A
hypothetical faulty output based on above input is:
1200|420 1|24.0|2|26.0|0|0.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0
1200|420 1|25.0|3|52.0|1|55.0
See that records with the key (1200|420) is not aggregated. This can happen
only if the record does not go to the same reducer after partioning.
Any clue why this is happening? I could not understanding what is going wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
