[
https://issues.apache.org/jira/browse/HADOOP-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573423#comment-13573423
]
Tom White commented on HADOOP-9230:
-----------------------------------
I guess that the legacy split check was included as a sanity check. Since the
split calculation is slightly different, the occasional difference should not
be too surprising. Nor should it be a problem for correctness, since the aim of
the calculation is to produce roughly equal-sized splits for distcp. Therefore
I think we can remove the legacy check in order to ensure that the test always
passes. +1 for the patch.
bq. I would expect the math discrepancy to lead to more than 10% failure rate
though.
Although the difference between floor and ceiling will almost always produce
different target split sizes differing by 1 (whenever the number of splits
doesn't exactly divide the total size), only very rarely will this affect
whether a file is or is not included in any particular split. This is why the
failures are relatively rare.
E.g. on one failure I got the file sizes were 1482, 2012, 1860, ... (making a
total size of 31439) and for 9 maps the legacy target size works out at 3493,
while the new target size is 3494. The first legacy split only included 1482
(since 1482 + 2012 > 3493) while the first split for the new code includes both
1482 and 2012.
> TestUniformSizeInputFormat fails intermittently
> -----------------------------------------------
>
> Key: HADOOP-9230
> URL: https://issues.apache.org/jira/browse/HADOOP-9230
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.2-alpha
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Labels: distcp
> Attachments: hadoop-9230.patch
>
>
> TestUniformSizeFileInputFormat fails intermittently. I ran the test 50 times
> and noticed 5 failures.
> Haven't noticed any particular pattern to which runs fail.
> A sample stack trace is as follows:
> {noformat}
> java.lang.AssertionError: expected:<1944> but was:<1820>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.junit.Assert.assertEquals(Assert.java:454)
> at
> org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.checkAgainstLegacy(TestUniformSizeInputFormat.java:244)
> at
> org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:126)
> at
> org.apache.hadoop.tools.mapred.TestUniformSizeInputFormat.testGetSplits(TestUniformSizeInputFormat.java:252)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
