[
https://issues.apache.org/jira/browse/HADOOP-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676373#comment-13676373
]
Jason Lowe edited comment on HADOOP-9622 at 6/5/13 9:33 PM:
------------------------------------------------------------
Attaching a testcase for the problem, which requires the same input file as
used in one of Pig's bzip2 unit tests,
[blockEndingInCR.txt.bz2|https://issues.apache.org/jira/secure/attachment/12438881/blockEndingInCR.txt.bz2].
Drop the file into hadoop-common-project/hadoop-common/src/test/resources/
and the unit test can be run via Maven.
I've also manually verified the loss of records via a simple streaming job:
{noformat}
hadoop jar hadoop-*/share/hadoop/tools/lib/hadoop-streaming*.jar
-Dmapreduce.input.fileinputformat.split.minsize=136498 -Dmapreduce.job.maps=3
-input blockEndingInCR.txt.bz2 -output streamout -mapper cat -reducer NONE
{noformat}
Examining the output of the job shows that the record after the carriage return
in the file was dropped. Running a similar streaming job with only one map (no
splits) shows the record is not dropped.
was (Author: jlowe):
Attaching a testcase for the problem, which requires the same input file as
used in one of Pig's bzip2 unit tests,
[blockEndingInCR.txt.bz2|https://issues.apache.org/jira/secure/attachment/12438881/blockEndingInCR.txt.bz2].
Drop the file into hadoop-common-project/hadoop-common/src/test/resources/
and the unit test can be run via Maven.
I've also manually verified the loss of records via a simple streaming job:
hadoop jar hadoop-*/share/hadoop/tools/lib/hadoop-streaming*.jar
-Dmapreduce.input.fileinputformat.split.minsize=136498 -Dmapreduce.job.maps=3
-input blockEndingInCR.txt.bz2 -output streamout -mapper cat -reducer NONE
Examining the output of the job shows that the record after the carriage return
in the file was dropped. Running a similar streaming job with only one map (no
splits) shows the record is not dropped.
> bzip2 codec can drop records when reading data in splits
> --------------------------------------------------------
>
> Key: HADOOP-9622
> URL: https://issues.apache.org/jira/browse/HADOOP-9622
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 2.0.4-alpha, 0.23.8
> Reporter: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-9622-testcase.patch
>
>
> Bzip2Codec.BZip2CompressionInputStream can cause records to be dropped when
> reading them in splits based on where record delimiters occur relative to
> compression block boundaries.
> Thanks to [~knoguchi] for discovering this problem while working on PIG-3251.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
