[
https://issues.apache.org/jira/browse/CAMEL-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aurélien Pupier updated CAMEL-24318:
------------------------------------
Description:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
* without, it is still running after 1h40 and it has built 484 modules so far
Currently, the full compilation can fail when launched with -T1C:
* [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
* [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile (default-compile)
on project camel-jooq: Compilation failure: Compilation failure:
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
package org.springframework.beans.factory.annotation does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
package org.springframework.stereotype does not exist
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
cannot find symbol
[ERROR] symbol: class Repository
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.AuthorDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
[ERROR]
/home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
cannot find symbol
[ERROR] symbol: class Autowired
[ERROR] location: class
org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
[ERROR] -> [Help 2]
and we also have a lot of test failures. To improve that, i think we can:
* try to launch with the option locally first to not overcrowd the CI and
upload to develocity with a specific tag to be able to analyze easily the tests
which are not yet good for //. In fact, we could be able to first provide an
@Isolated on all of these ones in a first time, this should accelerate anyway
the global time.
* Then when able to have green build locally, try with a specific jenkins job,
separated from the others to avoid bringing too much instability
* Then enjoy it on the "normal" flow
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available memory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exhaust.
Maybe can use lower in general and tweak with higher value only on modules
which requires it
was:
Currently, the build on CI is launched using a single Core. Using multiple
cores should allow to drastically reduce the time.
Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
* -T1C for 642 modules, it took 40'
* without, it is still running after 1h40 and it has built 484 modules so far
Currently, the full compilation can fail when launched with -T1C
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process (default)
on project bundles-pom: Error loading supplemental data models: Could not find
resource 'target/legal/legal.xml'. -> [Help 1]
and we also have a lot of test failures. To improve that, i think we can:
* try to launch with the option locally first to not overcrowd the CI and
upload to develocity with a specific tag to be able to analyze easily the tests
which are not yet good for //. In fact, we could be able to first provide an
@Isolated on all of these ones in a first time, this should accelerate anyway
the global time.
* Then when able to have green build locally, try with a specific jenkins job,
separated from the others to avoid bringing too much instability
* Then enjoy it on the "normal" flow
Some ideas:
* to check also if we can do the pure compile in // with all threads and then
launch tests as of now (1 threads used although 22 are available on blarge
jenkins machines)
* see if it works and improve things for sonar analysis (40' on CI), this is a
separate command than the others
* on GitHub there are already 2 phases but it is using mvnd; [it is taking
around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
* evaluate with -T1C and more specific values, T1C might overload the Jenkins
nodes. To check.
* evaluate available memory too, most surefire tests are launch in forked
process with 2G of memory, if too many are launched in // it might exhaust.
Maybe can use lower in general and tweak with higher value only on modules
which requires it
> Support Maven build and test using multiple cores
> -------------------------------------------------
>
> Key: CAMEL-24318
> URL: https://issues.apache.org/jira/browse/CAMEL-24318
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Priority: Major
>
> Currently, the build on CI is launched using a single Core. Using multiple
> cores should allow to drastically reduce the time.
> Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
> * -T1C for 642 modules, it took 40'
> * without, it is still running after 1h40 and it has built 484 modules so far
>
> Currently, the full compilation can fail when launched with -T1C:
> * [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process
> (default) on project bundles-pom: Error loading supplemental data models:
> Could not find resource 'target/legal/legal.xml'. -> [Help 1]
> * [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile
> (default-compile) on project camel-jooq: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.AuthorDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
> [ERROR] -> [Help 2]
> and we also have a lot of test failures. To improve that, i think we can:
> * try to launch with the option locally first to not overcrowd the CI and
> upload to develocity with a specific tag to be able to analyze easily the
> tests which are not yet good for //. In fact, we could be able to first
> provide an @Isolated on all of these ones in a first time, this should
> accelerate anyway the global time.
> * Then when able to have green build locally, try with a specific jenkins
> job, separated from the others to avoid bringing too much instability
> * Then enjoy it on the "normal" flow
> Some ideas:
> * to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on
> blarge jenkins machines)
> * see if it works and improve things for sonar analysis (40' on CI), this is
> a separate command than the others
> * on GitHub there are already 2 phases but it is using mvnd; [it is taking
> around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
> * evaluate with -T1C and more specific values, T1C might overload the
> Jenkins nodes. To check.
> * evaluate available memory too, most surefire tests are launch in forked
> process with 2G of memory, if too many are launched in // it might exhaust.
> Maybe can use lower in general and tweak with higher value only on modules
> which requires it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
