[
https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881996#comment-16881996
]
Steve Loughran commented on HADOOP-16415:
-----------------------------------------
All the tests of S3A committers are slow because they spin up a yarn cluster
each (slow) then the actual MR jobs
Proposal: rework so that the miniyarn cluster comes up once, with each job
running as its own test suite. This is complex for the Terasort tests (only
run on scale) as they are implemented as ordered set of test cases; we'd need
to copy the base suite and rework to run terasort for dir and magic committers
in sequence
The ITest*CommitProtocol suites are slow too; 300-400+ seconds each
* do we need the staging one?
* what can we do for better parallelism here? Even if its just faster creation
of temp files, deletion etc
> Speed up S3A test runs
> ----------------------
>
> Key: HADOOP-16415
> URL: https://issues.apache.org/jira/browse/HADOOP-16415
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Major
>
> S3A Test runs are way too slow.
> Speed them by
> * reducing test setup/teardown costs
> * eliminating obsolete test cases
> * merge small tests into larger ones.
> One thing i see is that the main S3A test cases create and destroy new FS
> instances; There's both a setup and teardown cost there, but it does
> guarantee better isolation.
> Maybe if we know all test cases in a specific suite need the same options, we
> can manage that better; demand create the FS but only delete it in an
> @Afterclass method. That'd give us the OO-inheritance based setup of tests,
> but mean only one instance is done per suite
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
