[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Klochkov updated MAPREDUCE-4980:
---------------------------------------
Attachment: MAPREDUCE-4980.patch
Attaching a patch which does fixes similar to the ones introduced in
HADOOP-9287 and HDFS-4491 and it depends on these jiras. The patch introduces
multi-fork execution for the hadoop-mapreduce-client-jobclient module only.
This patch, together with HADOOP-9287 and HDFS-4491 introduces multi-fork
execution for 3 Hadoop Core modules which take majority of time to test:
hadoop-common, hadoop-hdfs and hadoop-mapreduce-client-jobclient.
Overview of changes:
1. A new profile "parallel-tests" is introduced in
hadoop-mapreduce-client-jobclient/pom.xml
2. Tests are refactored to use PathUtils.getTestDir/getTestPath methods to get
a directory/path to be used for test data. Earlier, the refactored tests
implemented it in a similar but slightly different way.
3. MiniMRClientClusterFactory is replaced with MiniMRClientClusterBuilder which
makes it more flexible when configuring clusters, in a way similar to
MiniDFSCluster.Builder
4. All usages of deprecated class MiniMRCluster are replaced using
MiniMRClientClusterBuilder. This is required to avoid FS contention when
running mini mr clusters in parallel.
5. All usages of MiniDFSCluster constructors are replaced with
MiniDFSCluster.Builder usage, same purpose as prev item.
The changes are tested comparing stableness of tests to trunk running both in
serial mode and parallel mode. No additional flakiness is noticed.
> Parallel test execution of hadoop-mapreduce-client-core
> -------------------------------------------------------
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: test
> Affects Versions: 3.0.0
> Reporter: Tsuyoshi OZAWA
> Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the
> tests can be run more faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
