[
https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051053#comment-16051053
]
Aaron Fabbri commented on HADOOP-14527:
---------------------------------------
Added a method to clear any delayed keys from the inconsistent client, instead
of doing Thread.sleep() to wait for timeout. This only gave small speedup.
Pasting runtimes of main test cases here:
{noformat}
2017-06-15 11:29:24,976 [JUnit-testConsistentRenameAfterDelete] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentRenameAfterDelete(244)) - took
17939 msec
2017-06-15 11:29:43,360 [JUnit-testConsistentListAfterDelete] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterDelete(194)) - took
16404 msec
2017-06-15 11:33:15,135 [JUnit-testConsistentListFiles] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListFiles(373)) - took 209143
msec
2017-06-15 11:34:57,714 [JUnit-testConsistentListLocatedStatusAfterPut] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListLocatedStatusAfterPut(305))
- took 73391 msec
2017-06-15 11:35:13,635 [JUnit-testConsistentListAfterRename] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterRename(121)) - took
9961 msec
2017-06-15 11:35:18,434 [JUnit-testConsistentListStatusAfterPut] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListStatusAfterPut(279)) - took
3212 msec
2017-06-15 11:35:56,313 [JUnit-testRollingRenames] INFO
s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testRollingRenames(153)) - took 35460 msec
{noformat}
Will look at reducing the big ones when I get a chance.
> ITestS3GuardListConsistency is too slow
> ---------------------------------------
>
> Key: HADOOP-14527
> URL: https://issues.apache.org/jira/browse/HADOOP-14527
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Priority: Minor
>
> I'm really glad to see folks adopting the inconsistency injection stuff and
> adding test cases to ITestS3GuardListConsistency. That test class has become
> very slow, however, mostly due to {{Thread.sleep()}} calls that wait for the
> inconsistency injection timers to expire.
> I will take a stab at speeding up this test class. As is it takes about 8
> minutes to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
