[
https://issues.apache.org/jira/browse/HADOOP-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565964#comment-17565964
]
Steve Loughran commented on HADOOP-18335:
-----------------------------------------
parallel testing is great, does take effort to make robust. paths and network
ports are the risks
s3a does suites in parallel. abfs tests actually run separate test cases,
that's a bolder move there!
> Improve Pre-Commit Time
> -----------------------
>
> Key: HADOOP-18335
> URL: https://issues.apache.org/jira/browse/HADOOP-18335
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Ayush Saxena
> Priority: Critical
> Labels: buid
>
> As of now the complete build time has reached ~24 hours, which makes tracking
> Jira with root level changes very tough.
> Even at module level, it is on the higher side, kind of 6 hours or more,
> which is typically equivalent to 1 working day hours:
> Some areas to explore:
> * Enable Parallel-Test profile for the modules which don't have it.
> * Explore improvements in the area of our Test setup, increase memory,
> number of threads, or some parallel executions?
> * Remove the modules or atleast disable the test suites of modules which are
> no longer being used or maintained (Separate Mail Chain is there for this)
> * Spin Two Pre-Commit jobs rather than 1 and split some task between them,
> kind of one for Java-11 build & Javadoc, Checkstyle and related stuff.
> * Figure out & remove irrelevant or similar tests.
> * Improve existing tests, Like reusing MiniDfsClusters or so, rather than a
> bunch of test spinning one for themselves.
> * If possible, if there are multiple modules to be tested, run those modules
> test in parallel (Exploratory: may lead to OOM)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
