[
https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881473#comment-16881473
]
Steve Loughran commented on HADOOP-16415:
-----------------------------------------
and of course we can parallelise existing tests better by splitting up single
large test suites into smaller ones; with the FcContext one being the obvious
target. But: if we improve recycling of s3a FS instances across a single test
suite, we'd actually get more benefit from the larger suites
See: HADOOP-13330 for what we can do w.r.t delete speedup. We can do the
s3guard updates incrementally and internally, parallel issuing of batch
updates. For deleting 1000 files, DynamoDB, and hence S3guard, becomes the
bottleneck.
> Speed up S3A test runs
> ----------------------
>
> Key: HADOOP-16415
> URL: https://issues.apache.org/jira/browse/HADOOP-16415
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Major
>
> S3A Test runs are way too slow.
> Speed them by
> * reducing test setup/teardown costs
> * eliminating obsolete test cases
> * merge small tests into larger ones.
> One thing i see is that the main S3A test cases create and destroy new FS
> instances; There's both a setup and teardown cost there, but it does
> guarantee better isolation.
> Maybe if we know all test cases in a specific suite need the same options, we
> can manage that better; demand create the FS but only delete it in an
> @Afterclass method. That'd give us the OO-inheritance based setup of tests,
> but mean only one instance is done per suite
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
