Jason Lowe created PIG-3352:
-------------------------------
Summary: Bzip2TextInputFormat can duplicate records across splits
Key: PIG-3352
URL: https://issues.apache.org/jira/browse/PIG-3352
Project: Pig
Issue Type: Bug
Affects Versions: 0.10.1
Reporter: Jason Lowe
If a bz2 block boundary occurs in the middle of a record that is terminated by
a carriage-return then the next record will be duplicated. The compressed
stream position is updated at the same time a carriage-return character is seen
without a subsequent line-feed character. Based on the method of reporting
position within the compression stream, it incorrectly believes it has read
only the carriage-return character into the next compression block and ends up
processing the next record which will also be processed by the consumer of the
next split.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
