[
https://issues.apache.org/jira/browse/CAMEL-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aurélien Pupier updated CAMEL-24318:
------------------------------------
Description:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
* without, it is still running after 1h40 and it has built 484 modules so far
Currently, the full compilation can fail when launched with -T1C
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
and we also have a lot of test failures. To improve that, i think we can:
* try to launch with the option locally first to not overcrowd the CI and
upload to develocity with a specific tag to be able to analyze easily the tests
which are not yet good for //. In fact, we could be able to first provide an
@Isolated on all of these ones in a first time, this should accelerate anyway
the global time.
* Then when able to have green build locally, try with a specific jenkins job,
separated from the others to avoid bringing too much instability
* Then enjoy it on the "normal" flow
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available memory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exhaust.
Maybe can use lower in general and tweak with higher value only on modules
which requires it
was:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
* without, it is still running after 1h40 and it has built 484 modules so far
Currently, the full compilation can fail when launched with -T1C
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
and we also have a lot of test failures.
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available merory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exaust.
Maybe can use lower in general and tweak with highe rvalue only on modules
whiich requires it
> Support Maven build and test using multiple cores
> -------------------------------------------------
>
> Key: CAMEL-24318
> URL: https://issues.apache.org/jira/browse/CAMEL-24318
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Currently, the build on CI is launched using a single Core. Using multiple
> cores should allow to drastically reduce the time.
> Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
> * -T1C for 642 modules, it took 40'
> * without, it is still running after 1h40 and it has built 484 modules so far
> Currently, the full compilation can fail when launched with -T1C
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process
> (default) on project bundles-pom: Error loading supplemental data models:
> Could not find resource 'target/legal/legal.xml'. -> [Help 1]
> and we also have a lot of test failures. To improve that, i think we can:
> * try to launch with the option locally first to not overcrowd the CI and
> upload to develocity with a specific tag to be able to analyze easily the
> tests which are not yet good for //. In fact, we could be able to first
> provide an @Isolated on all of these ones in a first time, this should
> accelerate anyway the global time.
> * Then when able to have green build locally, try with a specific jenkins
> job, separated from the others to avoid bringing too much instability
> * Then enjoy it on the "normal" flow
> Some ideas:
> * to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on
> blarge jenkins machines)
> * see if it works and improve things for sonar analysis (40' on CI), this is
> a separate command than the others
> * on GitHub there are already 2 phases but it is using mvnd; [it is taking
> around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
> * evaluate with -T1C and more specific values, T1C might overload the
> Jenkins nodes. To check.
> * evaluate available memory too, most surefire tests are launch in forked
> process with 2G of memory, if too many are launched in // it might exhaust.
> Maybe can use lower in general and tweak with higher value only on modules
> which requires it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
