[
https://issues.apache.org/jira/browse/CAMEL-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130902#comment-18130902
]
Aurélien Pupier commented on CAMEL-24318:
-----------------------------------------
on GitHub workflows, mvnd is used so there is already parallelism in place.
Given that less CPUs and less modules are built at the same time, it was
causing less stability issues
> Support Maven build and test using multiple cores
> -------------------------------------------------
>
> Key: CAMEL-24318
> URL: https://issues.apache.org/jira/browse/CAMEL-24318
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.21.0
> Reporter: Aurélien Pupier
> Assignee: Aurélien Pupier
> Priority: Major
>
> Currently, the build on CI is launched using a single Core. Using multiple
> cores should allow to drastically reduce the time.
> Locally tests (22 cores, 32Gb RAM, similar to Jenkins large machines):
> * -T1C for 642 modules, it took 40'
> ** command used: ./mvnw clean install -T1C -Dci.env.name=fakeCI-local -B -e
> -fae -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
> -Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
> -Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
> -Dcamel.threads.virtual.enabled=false -Pcoverage
> * without, it took 2h48minutes for the same 642 modules
> ** command used: ./mvnw clean install -Dci.env.name=fakeCI-local -B -e -fae
> -V -Dnoassembly -Dsurefire.rerunFailingTestsCount=2
> -Dfailsafe.rerunFailingTestsCount=1 -Dchetype.test.skip
> -Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true install
> -Dcamel.threads.virtual.enabled=false -Pcoverage
>
> [FIXED] Currently, the full compilation can fail when launched with -T1C
> * [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:3.3.0:process
> (default) on project bundles-pom: Error loading supplemental data models:
> Could not find resource 'target/legal/legal.xml'. -> [Help 1]
> * this one also with single thread locally... [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.15.0:compile
> (default-compile) on project camel-jooq: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[15,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[16,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[23,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[14,52]
> package org.springframework.beans.factory.annotation does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[15,38]
> package org.springframework.stereotype does not exist
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[22,2]
> cannot find symbol
> [ERROR] symbol: class Repository
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/AuthorDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.AuthorDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookDao.java:[36,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class org.apache.camel.component.jooq.db.tables.daos.BookDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookStoreDao
> [ERROR]
> /home/apupier/git/camel/components/camel-jooq/target/generated-test-sources/jooq/org/apache/camel/component/jooq/db/tables/daos/BookToBookStoreDao.java:[35,6]
> cannot find symbol
> [ERROR] symbol: class Autowired
> [ERROR] location: class
> org.apache.camel.component.jooq.db.tables.daos.BookToBookStoreDao
> [ERROR] -> [Help 2]
> and we also have a lot of test failures. To improve that, i think we can:
> * try to launch with the option locally first to not overcrowd the CI and
> upload to develocity with a specific tag to be able to analyze easily the
> tests which are not yet good for //. In fact, we could be able to first
> provide an @Isolated on all of these ones in a first time, this should
> accelerate anyway the global time.
> * Then when able to have green build locally, try with a specific jenkins
> job, separated from the others to avoid bringing too much instability
> * Then enjoy it on the "normal" flow
> Some ideas:
> * to check also if we can do the pure compile in // with all threads and
> then launch tests as of now (1 threads used although 22 are available on
> blarge jenkins machines)
> * see if it works and improve things for sonar analysis (40' on CI), this is
> a separate command than the others
> ** no gain
> [https://community.sonarsource.com/t/feature-suggestion-parallel-analysis-in-maven-sonarscanner/19876/11]
> ** tried locally with a local instance and it took 8'15" in both cases
> ** sonar scanner is unable to leverage multiple threads
> * on GitHub there are already 2 phases but it is using mvnd; [it is taking
> around 18 minutes|https://github.com/apache/camel/actions/runs/30412528538/
> * evaluate with -T1C and more specific values, T1C might overload the
> Jenkins nodes. To check.
> * evaluate available memory too, most surefire tests are launch in forked
> process with 2G of memory, if too many are launched in // it might exhaust.
> Maybe can use lower in general and tweak with higher value only on modules
> which requires it
> * [DONE] For deploy jobs, given that they are not running tests, see if it
> is speeding things up, currently taking around 3 hours for Camel Core but
> bottleneck might be the network more than the build itself
> ** locally with local repo:
> *** 8'32" ( ./mvnw -T1C -U -B -e -fae -V -Dnoassembly
> -Pdeploy,apache-snapshots -Dquickly clean deploy
> -DaltDeploymentRepository=your_repo_id::default::[file:/tmp/alt-repo-2|file:///tmp/alt-repo-2]
> )
> *** 18'33" (./mvnw -U -B -e -fae -V -Dnoassembly -Pdeploy,apache-snapshots
> -Dquickly clean deploy
> -DaltDeploymentRepository=your_repo_id::default::[file:/tmp/alt-repo-2|file:///tmp/alt-repo-2])
> ** on jenkins for Camel core:
> *** 25min instead of usually 3 hours, previous record was 2h35 , see
> [https://ci-builds.apache.org/job/Camel/job/Camel%20Daily%20Snapshot%20Deploy/job/main/1494/]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
