Ashutosh Gupta created HADOOP-18321:
---------------------------------------
Summary: Fix when to read an additional record from a BZip2 text
file split
Key: HADOOP-18321
URL: https://issues.apache.org/jira/browse/HADOOP-18321
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.3
Reporter: Ashutosh Gupta
Assignee: Ashutosh Gupta
Fix data correctness issue with TextInputFormat that can occur when reading
BZip2 compressed text files. When triggered this bug would cause a split to
return the first record of the succeeding split that reads the next BZip2
block, thereby duplicating that record.
*When the bug is triggered?*
The condition for the bug to occur requires that the flag
"needAdditionalRecord" in CompressedSplitLineReader to be set to true by
#fillBuffer at an inappropriate time: when we haven't read the remaining bytes
of split. This can happen when the inDelimiter parameter is true while
#fillBuffer is invoked while reading the next line. The inDelimiter parameter
is true when either 1) the last byte of the buffer is a CR character ('\r') if
using the default delimiters, or 2) the last bytes of the buffer are a common
prefix of the delimiter if using a custom delimiter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
