[
https://issues.apache.org/jira/browse/HADOOP-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574499#comment-17574499
]
Ayush Saxena commented on HADOOP-18335:
---------------------------------------
Makes sense [~svaughan] , Will merge the MAPREDUCE-7386 if we get a green build
and no further comments, I remember checking that once...
{quote}With this change you can build the entire distribution
{quote}
We do it for both trunk & with patch, so should be a good save.
{quote}I'm currently doing a full distribution including native libraries in
under 10 minutes using Maven's "-T 2C"
{quote}
Can you create a subtask under this Jira, to make this change in our Pre-Commit
jobs and we can chase this post MAPREDUCE-7386. {{hadoop.sh}} will give you an
option to add this flag. Most probably adding here should do(need to check)
[https://github.com/apache/hadoop/blob/123d1aa8846a2099b04fd8e8ed7f2bd6db4f36b5/dev-support/bin/hadoop.sh#L347]
{quote}For testing I'm focused on the tests for hadoop-hdfs and
hadoop-hdfs-client (and dependent modules) which run an additional 13 minutes
when distributed across a Kuberentes cluster.
{quote}
13 minutes sound too good, I am happy with 31 mins also, if we can achieve it
somehow here. If you think we have a possibility to do so here and you plan to
chase that, feel free to create subtasks here & I can help review, or try to
involve folks who are more experienced in that area.
> Improve Pre-Commit Time
> -----------------------
>
> Key: HADOOP-18335
> URL: https://issues.apache.org/jira/browse/HADOOP-18335
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Ayush Saxena
> Priority: Critical
> Labels: buid
>
> As of now the complete build time has reached ~24 hours, which makes tracking
> Jira with root level changes very tough.
> Even at module level, it is on the higher side, kind of 6 hours or more,
> which is typically equivalent to 1 working day hours:
> Some areas to explore:
> * Enable Parallel-Test profile for the modules which don't have it.
> * Explore improvements in the area of our Test setup, increase memory,
> number of threads, or some parallel executions?
> * Remove the modules or atleast disable the test suites of modules which are
> no longer being used or maintained (Separate Mail Chain is there for this)
> * Spin Two Pre-Commit jobs rather than 1 and split some task between them,
> kind of one for Java-11 build & Javadoc, Checkstyle and related stuff.
> * Figure out & remove irrelevant or similar tests.
> * Improve existing tests, Like reusing MiniDfsClusters or so, rather than a
> bunch of test spinning one for themselves.
> * If possible, if there are multiple modules to be tested, run those modules
> test in parallel (Exploratory: may lead to OOM)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
