[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13113:
-----------------------------------
Attachment: HADOOP-13113.001.patch
I'm attaching patch v001. This uses the same parallel test execution
techniques that have been effective in hadoop-common and hadoop-hdfs. I
updated pom.xml to add the {{parallel-tests}} profile and changed some test
code to generate testing paths based on the Surefire fork number, so that
they're guaranteed to be isolated from each other. HADOOP-12801 is a
pre-requisite for this patch.
There are some limitations to this approach. Tests that cover root directory
access, multi-part upload, or YARN usage would run into conflicts on shared
resources. To handle this, I've moved those tests to a separate Surefire
execution step, which runs sequentially instead of parallelized. A future
improvement could be to refactor tests related to root out of
{{FileSystemContractBaseTest}}, so that the non-root tests in that suite can
run in parallel. {{TestS3AFileSystemContract}} takes ~90 seconds to run, so it
would be nice to parallelize that with other suites. I chose not to step
outside hadoop-aws for the scope of the current patch though.
While I was editing pom.xml, I also removed an unused dependency on Jackson.
This change is backwards-compatible, because clients who had been relying on
receiving Jackson via transitive dependency through hadoop-aws will continue to
receive it through the dependency chain hadoop-aws -> hadoop-common -> Jackson.
In my environment, plain {{mvn test}} runs in ~30 minutes, and {{mvn
-Pparallel-tests test}} runs in ~13 minutes.
[[email protected]], would you please review when you get a chance?
> Enable parallel test execution for hadoop-aws.
> ----------------------------------------------
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13113.001.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
