Jason Lowe commented on MAPREDUCE-7052:

Thanks for the report and the patch!  The unit test is unrelated, I filed 
MAPREDUCE-7053 to track the failure which was introduced by MAPREDUCE-5124.  
You may be interested in following that.

I think the fixup is appropriate given the record length adjustment is tied to 
what the 19th test is doing.  However I'd like to see the fixup occur near the 
point where they are needing the value be > 1.  An issue now is that if someone 
updates the code to divide by 3 instead of 2 they have to remember to find the 
fixup code 20+ lines above it and fix that, too.  It'd be nice if the logic for 
handling test 19 was all in one place.

> TestFixedLengthInputFormat#testFormatCompressedIn is flaky
> ----------------------------------------------------------
>                 Key: MAPREDUCE-7052
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7052
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client, test
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>            Priority: Major
>         Attachments: MAPREDUCE-7052-001.patch
> Sometimes the test case TestFixedLengthInputFormat#testFormatCompressedIn can 
> fail with the following error:
> {noformat}
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>       at 
> org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:322)
>       at 
> org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90)
> {noformat}
> *Root cause:* under special circumstances, the following line can return a 
> huge number:
> {noformat}
>           // Test a split size that is less than record len
>           numSplits = (int)(fileSize/Math.floor(recordLength/2));
> {noformat}
> For example, let {{seed}} be 2026428718. This causes {{recordLength}} to be 1 
> at iteration 19. {{Math.floor()}} returns negative Infinity, which becomes 
> positve infinity after the divison. Casting it to {{int}} yields 
> {{Integer.MAX_VALUE}}. Eventually we get an OOME because the test wants to 
> create a huge {{InputSplit}} array.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

Reply via email to