[
https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051195#comment-16051195
]
Aaron Fabbri commented on HADOOP-14527:
---------------------------------------
After my changes, runtime goes from ~8 minutes to 2:30. Added a clear()
function for wiping inconsistency timers, and also reduced the amount of work
some of the test cases were doing. Nested loops with many permutations and
actual I/O to dynamo/S3 were the dominating factors, so on a higher-latency
network connection your runtime could have been over 10 minutes, easily.
{noformat}
2017-06-15 15:49:37,558 [JUnit-testConsistentRenameAfterDelete] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentRenameAfterDelete(245)) - took
6818 msec
2017-06-15 15:49:44,879 [JUnit-testConsistentListAfterDelete] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterDelete(195)) - took
5250 msec
2017-06-15 15:50:38,259 [JUnit-testConsistentListFiles] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListFiles(374)) - took 51463
msec
2017-06-15 15:51:07,688 [JUnit-testConsistentListLocatedStatusAfterPut] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListLocatedStatusAfterPut(306))
- took 21273 msec
2017-06-15 15:51:16,423 [JUnit-testConsistentListAfterRename] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterRename(122)) - took
5386 msec
2017-06-15 15:51:20,738 [JUnit-testConsistentListStatusAfterPut] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListStatusAfterPut(280)) - took
2660 msec
2017-06-15 15:51:49,477 [JUnit-testRollingRenames] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testRollingRenames(154)) - took 26420 msec
{noformat}
Will attach v1 patch that gave me these results..
> ITestS3GuardListConsistency is too slow
> ---------------------------------------
>
> Key: HADOOP-14527
> URL: https://issues.apache.org/jira/browse/HADOOP-14527
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Priority: Minor
>
> I'm really glad to see folks adopting the inconsistency injection stuff and
> adding test cases to ITestS3GuardListConsistency. That test class has become
> very slow, however, mostly due to {{Thread.sleep()}} calls that wait for the
> inconsistency injection timers to expire.
> I will take a stab at speeding up this test class. As is it takes about 8
> minutes to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
