Jason Lowe commented on HADOOP-15206:

Thanks for updating the patch!

I believe the latest patch will break CONTINUOUS mode since it will no longer 
strip the bzip2 file header in that case.

I don't think it will be OK to remove calling readStreamHeader when reset() is 
called.  We're resetting the codec state to start afresh, and that means 
potentially reading a new file header (e.g.: concatenated bzip2 files).  My 
thinking is that we need to read the header, but we should not report the byte 
position being updated when doing so while we're in BLOCK mode (i.e.: split 

I think we need to revert the stream header reading logic to the original 
behavior.  Instead we can put a small change in the BZip2InputStream 
constructor to handle the special case of small splits that can start at or 
before the first bz2 block.  If the read mode is BLOCK and 0 < startingPos <= 
HEADER_LEN + SUB_HEADER_LEN then we skip bytes until we get to the HEADER_LEN + 
SUB_HEADER_LEN + 1 offset in the stream.  The bufferedIn.skip method will be 
useful here, but it needs to be called in a loop in case the skip fails to skip 
everything in one call (per the javadoc).

> BZip2 drops and duplicates records when input split size is small
> -----------------------------------------------------------------
>                 Key: HADOOP-15206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15206
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.3, 3.0.0
>            Reporter: Aki Tanaka
>            Priority: Major
>         Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch, 
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch, 
> HADOOP-15206.005.patch
> BZip2 can drop and duplicate record when input split file is small. I 
> confirmed that this issue happens when the input split size is between 1byte 
> and 4bytes.
> I am seeing the following 2 problem behaviors.
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is 
> small
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO  [Thread-17] mapred.TestTextInputFormat 
> (TestTextInputFormat.java:verifyPartitions(317)) - 
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
>  count=99{code}
> > The input format read only 99 records but not 100 records
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat 
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file 
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
>  count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat 
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4 
> at position 8
> {code}
> I experienced this error when I execute Spark (SparkSQL) job under the 
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to