[ 
https://issues.apache.org/jira/browse/CAMEL-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145530#comment-18145530
 ] 

Aurélien Pupier edited comment on CAMEL-24290 at 11/16/26, 10:43 AM:
---------------------------------------------------------------------

_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier. 
Figures are derived from the Develocity build scan timeline data and should be 
verified before acting on them._

h3. Critical path analysis of build scan intlvjfp7sc2o

Scans compared:
* eb33cs73xhkq4: https://develocity.apache.org/s/eb33cs73xhkq4/timeline (2303.4 
s)
* vq6u4jcg4egze: https://develocity.apache.org/s/vq6u4jcg4egze/timeline (2278.6 
s)
* elj7p4rvx65gi: https://develocity.apache.org/s/elj7p4rvx65gi/timeline (2227.6 
s)
* *intlvjfp7sc2o*: https://develocity.apache.org/s/intlvjfp7sc2o/timeline 
(*2183.4 s*)

h4. Method
* Per-module span = first goal start to last goal end, taken from 
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and 
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back 
to the dependency that finished last before the module started. The path has 58 
modules, and the gap between consecutive steps is at most 4.5 s.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s 
sampling) over each module's span. "% time <=60% CPU" is the share of the 
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
* Module names link to the open "Improve test time" JIRA issue when there is 
one.

h4. Critical path modules of at least 10 s (scan intlvjfp7sc2o, path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Maven Plugins :: Camel Maven Package|24|12.1 s|26|100|9|surefire:test 5 s|
|Util|37|11.7 s|27|100|9|surefire:test 6 s|
|Support|58|15.2 s|33|87|10|compiler:compile 3 s|
|Core Model|93|10.1 s|12|100|9|compiler:compile 2 s|
|XML IO|103|23.3 s|60|48|9|jaxb2:schemagen 6 s|
|[*Core*|https://issues.apache.org/jira/browse/CAMEL-27007]|131|*207.6 
s*|20|89|*165*|surefire:test 172 s|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|339|85.2 
s|21|100|68|surefire:test 77 s|
|Test :: JUnit5|424|18.7 s|51|68|9|surefire:test 17 s|
|Hugging Face to SAP NetWeaver (18 modules, see note)|443 to 1206|11 to 95 s 
each|87-99|0-9|0-3|CyberArk Vault 95 s, MINA 92 s|
|Spring XML|814|69.7 s|77|23|16|surefire:test 44 s|
|Test :: Spring :: JUnit6|884|32.0 s|56|53|14|surefire:test 30 s|
|Jetty|1206|76.9 s|96|0|3|surefire:test 62 s|
|*Netty HTTP*|1283|*300.0 s*|86|18|43|surefire:test 288 s|
|[*Chaos*|https://issues.apache.org/jira/browse/CAMEL-27026]|1583|*138.9 
s*|43|71|*80*|surefire:test 134 s|
|Catalog :: Camel Catalog|1724|25.4 s|31|96|17|surefire:test 19 s|
|YAML DSL|1758|19.0 s|57|53|8|surefire:test 14 s|
|YAML DSL :: Validator|1777|21.0 s|71|19|6|surefire:test 18 s|
|[*JBang :: 
Core*|https://issues.apache.org/jira/browse/CAMEL-26818]|1800|*264.4 
s*|30|92|*184*|surefire:test serial-test 159 s|
|JBang :: Plugin :: TUI|2065|46.4 s|54|53|22|surefire:test 18 s|
|[Launcher|https://issues.apache.org/jira/browse/CAMEL-27029]|2111|67.3 
s|22|97|53|surefire:test 53 s|

Note on the grouped row: it covers the modules of 10 s or more between 443 s 
and 1206 s, excluding Spring XML and Test :: Spring :: JUnit6, which are listed 
separately. All run at 87-99% CPU.

h4. Long modules that also have CPU available (ranked by idle-seconds)
# [*JBang :: Core*|https://issues.apache.org/jira/browse/CAMEL-26818]: 264 s at 
30% CPU, 184 idle-s. The {{serial-test}} execution is 159 s of it.
# [*Core*|https://issues.apache.org/jira/browse/CAMEL-27007]: 208 s at 20% CPU, 
165 idle-s.
# *Chaos*: 139 s at 43% CPU, 80 idle-s.
# [*Management*|https://issues.apache.org/jira/browse/CAMEL-25965]: 85 s at 21% 
CPU, 68 idle-s.
# *Launcher*: 67 s at 22% CPU, 53 idle-s.

h4. Long modules where the machine is saturated
* *Netty HTTP* (300 s, 18% of the time at or below 60% CPU), Jetty (77 s) and 
the grouped stretch from 443 s to 1206 s. These are CPU-bound, so running their 
tests in parallel is unlikely to help. Only lighter tests would.
* Netty (291 s, 99% CPU) is also saturated but is not on the path in this scan. 
It runs from 916 s to 1207 s, and Netty HTTP starts right after Jetty ends.

h4. Module spans across the four scans (seconds, then mean machine CPU %)
||Module||eb33cs73xhkq4||vq6u4jcg4egze||elj7p4rvx65gi||intlvjfp7sc2o||
|HTTP|314, 98|315, 98|91, 99|96, 99|
|Netty|292, 98|279, 98|277, 98|291, 99|
|Netty HTTP|282, 93|309, 89|278, 92|300, 86|
|Jetty|57, 91|54, 92|71, 96|77, 96|
|[Core|https://issues.apache.org/jira/browse/CAMEL-27007]|266, 31|243, 23|251, 
16|208, 20|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|85, 43|77, 
29|82, 21|85, 20|
|[JBang :: Core|https://issues.apache.org/jira/browse/CAMEL-26818]|281, 20|304, 
27|272, 17|264, 30|
|Spring XML|117, 69|68, 74|65, 75|70, 77|
|[AI :: MCP Server|https://issues.apache.org/jira/browse/CAMEL-26897]|60, 
46|64, 41|101, 26|48, 50|
|[Chaos|https://issues.apache.org/jira/browse/CAMEL-27026]|115, 42|122, 44|140, 
34|139, 43|
|[Launcher|https://issues.apache.org/jira/browse/CAMEL-27029]|69, 23|70, 32|69, 
22|67, 22|

h4. Observations
* Total build time is down about 120 s since the first scan (2303.4 s to 2183.4 
s).
* Core is the biggest gain in this scan: {{surefire:test}} went from 213 s to 
172 s, and the module span from 251 s to 208 s.
* AI :: MCP Server is back to 48 s. The 101 s in the previous scan was probably 
noise.
* Netty HTTP is 22 s slower than in the previous scan and still spends 288 s in 
{{surefire:test}}.
* Chaos is now on the critical path (139 s, 43% CPU, 80 idle-s, 
{{surefire:test}} 134 s).
* HTTP stays at about 96 s ({{default-test}} 60 s, {{isolated-test}} 25 s).
* The project count grew from 722 to 726 between the earlier scans and this one.

h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The 
Core -> Management spine and the Jetty -> Netty HTTP -> ... -> JBang :: Core 
tail are the stable parts. The middle of the path depends on Maven {{-T1C}} 
scheduling and differs between scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled 
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or 
timeouts). That needs a look at the surefire configuration and per-test 
durations.


was (Author: apupier):
_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier. 
Figures are derived from the Develocity build scan timeline data and should be 
verified before acting on them._

h3. Critical path analysis of build scan intlvjfp7sc2o

Scans compared:
* eb33cs73xhkq4: https://develocity.apache.org/s/eb33cs73xhkq4/timeline (2303.4 
s)
* vq6u4jcg4egze: https://develocity.apache.org/s/vq6u4jcg4egze/timeline (2278.6 
s)
* elj7p4rvx65gi: https://develocity.apache.org/s/elj7p4rvx65gi/timeline (2227.6 
s)
* *intlvjfp7sc2o*: https://develocity.apache.org/s/intlvjfp7sc2o/timeline 
(*2183.4 s*)

h4. Method
* Per-module span = first goal start to last goal end, taken from 
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and 
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back 
to the dependency that finished last before the module started. The path has 58 
modules, and the gap between consecutive steps is at most 4.5 s.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s 
sampling) over each module's span. "% time <=60% CPU" is the share of the 
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
* Module names link to the open "Improve test time" JIRA issue when there is 
one.

h4. Critical path modules of at least 10 s (scan intlvjfp7sc2o, path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Maven Plugins :: Camel Maven Package|24|12.1 s|26|100|9|surefire:test 5 s|
|Util|37|11.7 s|27|100|9|surefire:test 6 s|
|Support|58|15.2 s|33|87|10|compiler:compile 3 s|
|Core Model|93|10.1 s|12|100|9|compiler:compile 2 s|
|XML IO|103|23.3 s|60|48|9|jaxb2:schemagen 6 s|
|[*Core*|https://issues.apache.org/jira/browse/CAMEL-27007]|131|*207.6 
s*|20|89|*165*|surefire:test 172 s|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|339|85.2 
s|21|100|68|surefire:test 77 s|
|Test :: JUnit5|424|18.7 s|51|68|9|surefire:test 17 s|
|Hugging Face to SAP NetWeaver (18 modules, see note)|443 to 1206|11 to 95 s 
each|87-99|0-9|0-3|CyberArk Vault 95 s, MINA 92 s|
|Spring XML|814|69.7 s|77|23|16|surefire:test 44 s|
|Test :: Spring :: JUnit6|884|32.0 s|56|53|14|surefire:test 30 s|
|Jetty|1206|76.9 s|96|0|3|surefire:test 62 s|
|*Netty HTTP*|1283|*300.0 s*|86|18|43|surefire:test 288 s|
|[*Chaos*|https://issues.apache.org/jira/browse/CAMEL-27026]|1583|*138.9 
s*|43|71|*80*|surefire:test 134 s|
|Catalog :: Camel Catalog|1724|25.4 s|31|96|17|surefire:test 19 s|
|YAML DSL|1758|19.0 s|57|53|8|surefire:test 14 s|
|YAML DSL :: Validator|1777|21.0 s|71|19|6|surefire:test 18 s|
|[*JBang :: 
Core*|https://issues.apache.org/jira/browse/CAMEL-26818]|1800|*264.4 
s*|30|92|*184*|surefire:test serial-test 159 s|
|JBang :: Plugin :: TUI|2065|46.4 s|54|53|22|surefire:test 18 s|
|Launcher|2111|67.3 s|22|97|53|surefire:test 53 s|

Note on the grouped row: it covers the modules of 10 s or more between 443 s 
and 1206 s, excluding Spring XML and Test :: Spring :: JUnit6, which are listed 
separately. All run at 87-99% CPU.

h4. Long modules that also have CPU available (ranked by idle-seconds)
# [*JBang :: Core*|https://issues.apache.org/jira/browse/CAMEL-26818]: 264 s at 
30% CPU, 184 idle-s. The {{serial-test}} execution is 159 s of it.
# [*Core*|https://issues.apache.org/jira/browse/CAMEL-27007]: 208 s at 20% CPU, 
165 idle-s.
# *Chaos*: 139 s at 43% CPU, 80 idle-s.
# [*Management*|https://issues.apache.org/jira/browse/CAMEL-25965]: 85 s at 21% 
CPU, 68 idle-s.
# *Launcher*: 67 s at 22% CPU, 53 idle-s.

h4. Long modules where the machine is saturated
* *Netty HTTP* (300 s, 18% of the time at or below 60% CPU), Jetty (77 s) and 
the grouped stretch from 443 s to 1206 s. These are CPU-bound, so running their 
tests in parallel is unlikely to help. Only lighter tests would.
* Netty (291 s, 99% CPU) is also saturated but is not on the path in this scan. 
It runs from 916 s to 1207 s, and Netty HTTP starts right after Jetty ends.

h4. Module spans across the four scans (seconds, then mean machine CPU %)
||Module||eb33cs73xhkq4||vq6u4jcg4egze||elj7p4rvx65gi||intlvjfp7sc2o||
|HTTP|314, 98|315, 98|91, 99|96, 99|
|Netty|292, 98|279, 98|277, 98|291, 99|
|Netty HTTP|282, 93|309, 89|278, 92|300, 86|
|Jetty|57, 91|54, 92|71, 96|77, 96|
|[Core|https://issues.apache.org/jira/browse/CAMEL-27007]|266, 31|243, 23|251, 
16|208, 20|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|85, 43|77, 
29|82, 21|85, 20|
|[JBang :: Core|https://issues.apache.org/jira/browse/CAMEL-26818]|281, 20|304, 
27|272, 17|264, 30|
|Spring XML|117, 69|68, 74|65, 75|70, 77|
|[AI :: MCP Server|https://issues.apache.org/jira/browse/CAMEL-26897]|60, 
46|64, 41|101, 26|48, 50|
|Chaos|115, 42|122, 44|140, 34|139, 43|
|Launcher|69, 23|70, 32|69, 22|67, 22|

h4. Observations
* Total build time is down about 120 s since the first scan (2303.4 s to 2183.4 
s).
* Core is the biggest gain in this scan: {{surefire:test}} went from 213 s to 
172 s, and the module span from 251 s to 208 s.
* AI :: MCP Server is back to 48 s. The 101 s in the previous scan was probably 
noise.
* Netty HTTP is 22 s slower than in the previous scan and still spends 288 s in 
{{surefire:test}}.
* Chaos is now on the critical path (139 s, 43% CPU, 80 idle-s, 
{{surefire:test}} 134 s).
* HTTP stays at about 96 s ({{default-test}} 60 s, {{isolated-test}} 25 s).
* The project count grew from 722 to 726 between the earlier scans and this one.

h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The 
Core -> Management spine and the Jetty -> Netty HTTP -> ... -> JBang :: Core 
tail are the stable parts. The middle of the path depends on Maven {{-T1C}} 
scheduling and differs between scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled 
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or 
timeouts). That needs a look at the surefire configuration and per-test 
durations.

> Improve Continuous Integration to support larger workload
> ---------------------------------------------------------
>
>                 Key: CAMEL-24290
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24290
>             Project: Camel
>          Issue Type: Task
>          Components: build system
>    Affects Versions: 4.21.0
>            Reporter: Aurélien Pupier
>            Priority: Major
>
> Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads 
> and the planned future will surely increase the load and the need for a more 
> stable CI.
> Currently Jenkins is taking between 4 and 5 hours when things are going well. 
> Can go up to 20 hours when all streams in //
> On GitHub, it happens regularly that the jobs are queued for several hours.
> We have few full green build per week on Jenkins. (We are at one or two per 
> year last year, so already a lot better but not enough to really sustain new 
> workloads efficiently).
> Having a CI failing only for real issues will also help all committers to 
> react faster
>  
> There are several factors for increased, and potential new increase:
>  * Increased number PRs with more review, so more cycle
>  * Increased number of backports
>  * Increased cadence of dependencies release
>  * Longer LTS ? , more streams to maintain?
>  * Shared GitHub runners with all Apache projects (and several of them are 
> facing the same workload increase)
> Several areas identified which worth at least some investigations:
>  * stability of Workflows run: it gives more confidence to the build, avoids 
> the need to rerun to check that it is "just" flaky or an infra hiccups
>  ** develocity help tracking flaky tests (especially flaky tests causing 
> failure)
>  ** -one last test is failing very often: SjmsConnectionRecoveryTest- 
> -[https://issues.apache.org/jira/browse/CAMEL-24284-] DONE no failure since 
> 1st September
>  ** several failing less often (every months)
>  *** useful Develocity dashboard 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.relativeStartTime=P28D]
>  **** beware of tests which failed for good reasons (it caught regressions)
>  **** beware of all the tests which were already fixed (there was a huge 
> session of fix, so maybe need to wait a month to have a clearer view and/or 
> play with the the timeframe)
>  **** I filtered on main and excluded JDK 26 for now. Because i have JDK 26 
> has tests which are failing for an identified bug and a lot of stability 
> improvements were done on main only
>  **** based on previous warning, the current curated list is:
>  ***** MasterEndpointIT 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.zookeepermaster.MasterEndpointIT&tests.test=testEndpoint]
>  ***** MongoDB: 
> [MongoDbBulkWriteOperationsIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbBulkWriteOperationIT],
>  
> [MongoDbCredentialsFromUriConnectionIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbCredentialsFromUriConnectionIT]
>  and 
> [MongoDbFindOneAndReplaceOperationIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbFindOneAndReplaceOperationIT]
>  ***** 
> [SpringFileAntPathMatcherRemoteFileFilterTest|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.itest.ftp.SpringFileAntPathMatcherRemoteFileFilterTest]
>  CAMEL-24618
>  ***** 
> [CassandraIdempotentRespotoryIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.processor.idempotent.cassandra.CassandraIdempotentRepositoryIT]
>  ***** and certainly a few others currently buried in the list of already 
> fixed ones
>  ** and also several flaky during the run but passing in less than 3 
> attempts, slowing down the build
>  *** 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main&search.relativeStartTime=P28D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY]
>  ***  
>  ** -one Jenkins workspace leak to fix due to some jbang tests- CAMEL-23795 
> DONE
>  * modification of test matrixes
>  ** testing a single JDK/OS permanently and push to weekly/nightly others on 
> jenkins?
>  ** -Reduce test matrix on maintenance stream?- DONE
>  * Optimize usage of available resources
>  ** see if we can build on smaller instances of Jenkins, for Linux it will 
> allow to move up from 4 to ~20 instances available.
>  ** use several threads (but requires a far better stability and robustness 
> of tests and ensure no port conflict) 
> [https://issues.apache.org/jira/browse/CAMEL-24318]
>  *** to check also if we can do the pure compile in // with all threads and 
> then launch tests as of now (1 threads used although 22 are available on big 
> machines)
>  **** on GitHub there are already 2 phases but it is using mvnd; [it is 
> taking around 18 
> minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
>  **** on jenkins it means to split in different phase
>  **** to test for sonar scanner ([currently 1 thread is 
> used|https://develocity.apache.org/s/fjldcrmqc7bds] and it is taking between 
> around 30 minutes)
>  *** on Jenkins, I think we could attempt to provide a specific job for this 
> experiment with specific develocity tag so that can filter out/in the results 
> easily
>  ** also seems 3.5G of Memory is used for the build, maybe we could use more 
> when in //
>  *** 32G in total on large jenkins machine
>  *** but [develocity seems to report only 1.7 
> max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used 
> currently
>  *** maybe an indication that we could now run on smaller Jenkins node as 
> this Memory constraint seems to date from 2018 and it has not changed, good 
> chance that jenkins nodes size has been improved since that time
>  * several s390x and ppc64le in //, currently [3 
> ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]  
> and [6 
> s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/] nodes 
> are referenced as shared node but a single one is available at a time
>  ** started dicussion here 
> [https://the-asf.slack.com/archives/CF6PY5M4N/p1787908008691489]
>  ** Bug opened in Jenkins [https://github.com/jenkinsci/jenkins/issues/27318]
>  * improve performance of the product/test code itself maybe the trickier one 
> but also the most beneficial for our end users
>  * use of Develocity build cache (not a big fan of not rebuilding but could 
> be something to reuse in some configurations)
>  * use of Develocity test selection
>  * improve scalpel to reduce even more the scope of PR tests
>  *  [DONE] improve [fetch based branch for scalpel change detection on GitHub 
> which is taking almost 5 
> minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
>  * gating PR builds?
>  ** have a specific account for AI agents which does not trigger build 
> automatically
>  ** only a subset triggered automatically then full test suite launched on 
> demand for all? for specific accounts?
>  * see if budget to provide more github runners and/or jenkins nodes
>  * check for performance bottleneck in the build and test process (develocity 
> build scan performance can give some ideas)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to