[ 
https://issues.apache.org/jira/browse/CAMEL-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146111#comment-18146111
 ] 

Aurélien Pupier commented on CAMEL-24290:
-----------------------------------------

_Analysis generated by Bob (IBM Bob, AI assistant) on behalf of apupier. 
Figures are derived from the Develocity build scan timeline data and should be 
verified before acting on them._

h3. Critical path analysis of build scan ez7j3mcy5apte

Scans compared:
* eb33cs73xhkq4: https://develocity.apache.org/s/eb33cs73xhkq4/timeline (2303.4 
s)
* vq6u4jcg4egze: https://develocity.apache.org/s/vq6u4jcg4egze/timeline (2278.6 
s)
* elj7p4rvx65gi: https://develocity.apache.org/s/elj7p4rvx65gi/timeline (2227.6 
s)
* intlvjfp7sc2o: https://develocity.apache.org/s/intlvjfp7sc2o/timeline (2183.4 
s)
* *ez7j3mcy5apte*: https://develocity.apache.org/s/ez7j3mcy5apte/timeline 
(*2082.0 s*)

h4. Method
* Per-module span = first goal start to last goal end, taken from 
{{/scan-data/maven/<scan>/timeline}}.
* Dependency edges inferred from {{pom.xml}} files (parent, dependencies and 
plugins, with parent inheritance).
* Critical path = start at the last-finishing module and repeatedly step back 
to the dependency that finished last before the module started. The path has 54 
modules, and the gap between consecutive steps is at most 2.2 s.
* CPU = mean of the machine-wide {{allProcessesCpu}} metric (about 1 s 
sampling) over each module's span. "% time <=60% CPU" is the share of the 
module's time at or below 60% load. Idle-seconds = duration x (100 - mean CPU%).
* Module names link to the open "Improve test time" JIRA issue when there is 
one.

h4. Where the 101 s gain over the previous scan came from
||Module||Previous scan||This scan||Change||
|[Chaos|https://issues.apache.org/jira/browse/CAMEL-27026]|139 s|29 s|-110 s 
(no longer on the path)|
|[Launcher|https://issues.apache.org/jira/browse/CAMEL-27029]|67 s|23 s|-44 s|
|[Core|https://issues.apache.org/jira/browse/CAMEL-27095]|208 s|176 s|-32 s|
|[JBang :: Core|https://issues.apache.org/jira/browse/CAMEL-26818]|264 s|203 
s|-61 s|

* Core: {{surefire:test}} went from 172 s to 140 s.
* JBang :: Core: {{default-test}} is 163 s and {{serial-test}} is now 10 s (it 
was 159 s). The tests appear to have moved between executions rather than 
disappeared, which should be confirmed in the pom diff.
* Launcher: {{surefire:test}} went from 53 s to 11 s.
* Chaos: {{surefire:test}} went from 134 s to 20 s.
* It has not been checked which commits are included in each scan.

h4. Critical path modules of at least 10 s (scan ez7j3mcy5apte, path order)
||Module||Start (s)||Duration||CPU %||% time <=60% CPU||Idle-s||Longest goal||
|Maven Plugins :: Camel Maven Package|26|11.9 s|29|100|9|surefire:test 5 s|
|Util|39|11.8 s|25|100|9|surefire:test 7 s|
|Support|59|14.4 s|32|80|10|surefire:test virtual-threads 3 s|
|Core Model|85|11.1 s|13|100|10|compiler:compile 2 s|
|XML IO|96|23.2 s|61|52|9|jaxb2:schemagen 7 s|
|[*Core*|https://issues.apache.org/jira/browse/CAMEL-27095]|124|*176.3 
s*|22|88|*137*|surefire:test 140 s|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|301|82.2 
s|19|99|66|surefire:test 76 s|
|Test :: JUnit6|383|18.6 s|51|67|9|surefire:test 17 s|
|AI :: Tool to Zendesk (12 modules, see note)|401 to 793|11 to 86 s 
each|96-99|0-5|0-2|OPA 86 s (failsafe 50 s), Paho 54 s|
|Spring XML|794|72.3 s|77|22|16|surefire:test 46 s|
|Test :: Spring :: JUnit6|866|32.0 s|56|53|14|surefire:test 31 s|
|*Netty*|898|*298.6 s*|99|0|4|surefire:test 293 s|
|Jetty|1197|76.5 s|96|0|3|surefire:test 67 s|
|*Netty HTTP*|1273|*318.9 s*|88|14|38|surefire:test 313 s|
|REST OpenApi|1592|17.1 s|49|72|9|surefire:test 15 s|
|Platform HTTP :: Vert.x|1609|23.9 s|26|100|18|surefire:test 22 s|
|Platform HTTP :: Main|1633|11.2 s|72|18|3|surefire:test 9 s|
|OAuth|1645|33.1 s|44|70|19|surefire:test 31 s|
|AI :: MCP Server|1678|48.2 s|45|69|27|failsafe:integration-test 39 s|
|Catalog :: Camel Catalog|1728|26.7 s|18|100|22|surefire:test 21 s|
|YAML DSL|1762|19.7 s|47|74|11|surefire:test 14 s|
|YAML DSL :: Validator|1782|21.4 s|63|36|8|surefire:test 18 s|
|[*JBang :: 
Core*|https://issues.apache.org/jira/browse/CAMEL-26818]|1806|*202.6 
s*|14|100|*174*|surefire:test default-test 163 s|
|JBang :: Plugin :: TUI|2008|46.2 s|53|49|22|surefire:test 19 s|
|[Launcher|https://issues.apache.org/jira/browse/CAMEL-27029]|2055|23.3 
s|27|83|17|surefire:test 11 s|

Note on the grouped row: it covers the modules of 10 s or more between 401 s 
and 793 s (AI :: Tool, Alibaba Cloud :: Function Compute, AWS2 :: Transcribe, 
Dapr, Google :: Functions, IBM :: Secrets Manager, Jolt, Java Template Engine, 
MVEL, OPA, Paho, Zendesk). All run at 96-99% CPU.

h4. Long modules that also have CPU available (ranked by idle-seconds)
# [*JBang :: Core*|https://issues.apache.org/jira/browse/CAMEL-26818]: 203 s at 
14% CPU, 174 idle-s. {{default-test}} takes 163 s and the machine is at or 
below 60% CPU for 100% of that time.
# [*Core*|https://issues.apache.org/jira/browse/CAMEL-27095]: 176 s at 22% CPU, 
137 idle-s.
# [*Management*|https://issues.apache.org/jira/browse/CAMEL-25965]: 82 s at 19% 
CPU, 66 idle-s.
# *Netty HTTP*: 319 s at 88% CPU, 38 idle-s. It is the longest module on the 
path, but the machine is mostly busy while it runs.
# AI :: MCP Server, Catalog and OAuth: 19-27 idle-s each, in the 30-50 s range.
# Platform HTTP :: Vert.x: 18 idle-s over 24 s.

h4. Long modules where the machine is saturated
* *Netty* (299 s) and *Netty HTTP* (319 s, 14% of the time at or below 60% 
CPU), plus Jetty (77 s) and the grouped AI :: Tool to Zendesk stretch. These 
are CPU-bound, so running their tests in parallel is unlikely to help. Only 
lighter tests would.
* Netty -> Netty HTTP is now the longest uninterrupted CPU-bound stretch on the 
path (about 700 s together with Jetty). Netty HTTP is also slower than in any 
earlier scan (282 s, 309 s, 278 s, 300 s, now 319 s).
* HTTP (105 s) is no longer on the path.

h4. Module spans across the five scans (seconds @ start, then mean machine CPU 
%)
||Module||eb33cs73xhkq4||vq6u4jcg4egze||elj7p4rvx65gi||intlvjfp7sc2o||*ez7j3mcy5apte*||
|Build|2303|2279|2228|2183|*2082*|
|[Core|https://issues.apache.org/jira/browse/CAMEL-27095]|266 @119, 31|243 
@117, 23|251 @120, 16|208 @131, 20|*176 @124, 22*|
|[Management|https://issues.apache.org/jira/browse/CAMEL-25965]|85 @385, 43|77 
@360, 29|82 @371, 21|85 @339, 20|*82 @301, 19*|
|HTTP|314 @1018, 98|315 @929, 98|91 @924, 99|96 @916, 99|*105 @898, 99*|
|Netty|292 @1018, 98|279 @929, 98|277 @924, 98|291 @916, 99|*299 @898, 99*|
|Jetty|57 @1332, 91|54 @1244, 92|71 @1201, 96|77 @1206, 96|*77 @1197, 96*|
|Netty HTTP|282 @1389, 93|309 @1299, 89|278 @1272, 92|300 @1283, 86|*319 @1273, 
88*|
|[JBang :: Core|https://issues.apache.org/jira/browse/CAMEL-26818]|281 @1879, 
20|304 @1830, 27|272 @1815, 17|264 @1800, 30|*203 @1806, 14*|
|JBang :: Plugin :: TUI|68 @2160, 33|69 @2134, 37|67 @2087, 31|46 @2065, 54|*46 
@2008, 53*|
|[Launcher|https://issues.apache.org/jira/browse/CAMEL-27029]|69 @2228, 23|70 
@2203, 32|69 @2154, 22|67 @2111, 22|*23 @2055, 27*|
|[Chaos|https://issues.apache.org/jira/browse/CAMEL-27026]|115 @1671, 42|122 
@1608, 44|140 @1550, 34|139 @1583, 43|*29 @1592, 41*|
|Spring XML|117 @871, 69|68 @828, 74|65 @823, 75|70 @814, 77|*72 @794, 77*|
|AI :: MCP Server|60 @1743, 46|64 @1680, 41|101 @1633, 26|48 @1665, 50|*48 
@1678, 45*|

h4. Observations
* Total build time is down about 221 s since the first scan (2303.4 s to 2082.0 
s), and about 101 s since the previous one.
* The longest remaining CPU-available blocks are JBang :: Core, Core and 
Management (about 377 idle-s between them).
* The project count grew from 726 to 727 since the previous scan.

h4. Caveats
* Dependency edges are inferred from pom files, not reported by Develocity. The 
Core -> Management spine and the Netty -> Netty HTTP -> ... -> JBang :: Core 
tail are the stable parts. The middle of the path depends on Maven {{-T1C}} 
scheduling and differs between scans.
* CPU is machine-wide (all processes on the agent), not per module, and sampled 
at about 1 s.
* The data does not show why a module leaves CPU idle (serial tests, sleeps or 
timeouts). That needs a look at the surefire configuration and per-test 
durations.

> Improve Continuous Integration to support larger workload
> ---------------------------------------------------------
>
>                 Key: CAMEL-24290
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24290
>             Project: Camel
>          Issue Type: Task
>          Components: build system
>    Affects Versions: 4.21.0
>            Reporter: Aurélien Pupier
>            Priority: Major
>
> Our 2 CI systems (Jenkins and GitHub actions) are already under heavy loads 
> and the planned future will surely increase the load and the need for a more 
> stable CI.
> Currently Jenkins is taking between 4 and 5 hours when things are going well. 
> Can go up to 20 hours when all streams in //
> On GitHub, it happens regularly that the jobs are queued for several hours.
> We have few full green build per week on Jenkins. (We are at one or two per 
> year last year, so already a lot better but not enough to really sustain new 
> workloads efficiently).
> Having a CI failing only for real issues will also help all committers to 
> react faster
>  
> There are several factors for increased, and potential new increase:
>  * Increased number PRs with more review, so more cycle
>  * Increased number of backports
>  * Increased cadence of dependencies release
>  * Longer LTS ? , more streams to maintain?
>  * Shared GitHub runners with all Apache projects (and several of them are 
> facing the same workload increase)
> Several areas identified which worth at least some investigations:
>  * stability of Workflows run: it gives more confidence to the build, avoids 
> the need to rerun to check that it is "just" flaky or an infra hiccups
>  ** develocity help tracking flaky tests (especially flaky tests causing 
> failure)
>  ** -one last test is failing very often: SjmsConnectionRecoveryTest- 
> -[https://issues.apache.org/jira/browse/CAMEL-24284-] DONE no failure since 
> 1st September
>  ** several failing less often (every months)
>  *** useful Develocity dashboard 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.relativeStartTime=P28D]
>  **** beware of tests which failed for good reasons (it caught regressions)
>  **** beware of all the tests which were already fixed (there was a huge 
> session of fix, so maybe need to wait a month to have a clearer view and/or 
> play with the the timeframe)
>  **** I filtered on main and excluded JDK 26 for now. Because i have JDK 26 
> has tests which are failing for an identified bug and a lot of stability 
> improvements were done on main only
>  **** based on previous warning, the current curated list is:
>  ***** MasterEndpointIT 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.zookeepermaster.MasterEndpointIT&tests.test=testEndpoint]
>  ***** MongoDB: 
> [MongoDbBulkWriteOperationsIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbBulkWriteOperationIT],
>  
> [MongoDbCredentialsFromUriConnectionIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbCredentialsFromUriConnectionIT]
>  and 
> [MongoDbFindOneAndReplaceOperationIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.mongodb.integration.MongoDbFindOneAndReplaceOperationIT]
>  ***** 
> [SpringFileAntPathMatcherRemoteFileFilterTest|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.itest.ftp.SpringFileAntPathMatcherRemoteFileFilterTest]
>  CAMEL-24618
>  ***** 
> [CassandraIdempotentRespotoryIT|https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main%20-maven.environment.jvmVersion:*26*&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.processor.idempotent.cassandra.CassandraIdempotentRepositoryIT]
>  ***** and certainly a few others currently buried in the list of already 
> fixed ones
>  ** and also several flaky during the run but passing in less than 3 
> attempts, slowing down the build
>  *** 
> [https://develocity.apache.org/scans/tests?search.query=project:camel%20tag:main&search.relativeStartTime=P28D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY]
>  ***  
>  ** -one Jenkins workspace leak to fix due to some jbang tests- CAMEL-23795 
> DONE
>  * modification of test matrixes
>  ** testing a single JDK/OS permanently and push to weekly/nightly others on 
> jenkins?
>  ** -Reduce test matrix on maintenance stream?- DONE
>  * Optimize usage of available resources
>  ** see if we can build on smaller instances of Jenkins, for Linux it will 
> allow to move up from 4 to ~20 instances available.
>  ** use several threads (but requires a far better stability and robustness 
> of tests and ensure no port conflict) 
> [https://issues.apache.org/jira/browse/CAMEL-24318]
>  *** to check also if we can do the pure compile in // with all threads and 
> then launch tests as of now (1 threads used although 22 are available on big 
> machines)
>  **** on GitHub there are already 2 phases but it is using mvnd; [it is 
> taking around 18 
> minutes|https://github.com/apache/camel/actions/runs/30412528538/job/90451766586#step:9:1]
>  **** on jenkins it means to split in different phase
>  **** to test for sonar scanner ([currently 1 thread is 
> used|https://develocity.apache.org/s/fjldcrmqc7bds] and it is taking between 
> around 30 minutes)
>  *** on Jenkins, I think we could attempt to provide a specific job for this 
> experiment with specific develocity tag so that can filter out/in the results 
> easily
>  ** also seems 3.5G of Memory is used for the build, maybe we could use more 
> when in //
>  *** 32G in total on large jenkins machine
>  *** but [develocity seems to report only 1.7 
> max|https://develocity.apache.org/s/kuigqxgdpicsy/performance/build] is used 
> currently
>  *** maybe an indication that we could now run on smaller Jenkins node as 
> this Memory constraint seems to date from 2018 and it has not changed, good 
> chance that jenkins nodes size has been improved since that time
>  * several s390x and ppc64le in //, currently [3 
> ppc64le|https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/]  
> and [6 
> s390x|https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/] nodes 
> are referenced as shared node but a single one is available at a time
>  ** started dicussion here 
> [https://the-asf.slack.com/archives/CF6PY5M4N/p1787908008691489]
>  ** Bug opened in Jenkins [https://github.com/jenkinsci/jenkins/issues/27318]
>  * improve performance of the product/test code itself maybe the trickier one 
> but also the most beneficial for our end users
>  * use of Develocity build cache (not a big fan of not rebuilding but could 
> be something to reuse in some configurations)
>  * use of Develocity test selection
>  * improve scalpel to reduce even more the scope of PR tests
>  *  [DONE] improve [fetch based branch for scalpel change detection on GitHub 
> which is taking almost 5 
> minutes|https://github.com/apache/camel/actions/runs/30429114691/job/90501998254#step:3:1]
>  * gating PR builds?
>  ** have a specific account for AI agents which does not trigger build 
> automatically
>  ** only a subset triggered automatically then full test suite launched on 
> demand for all? for specific accounts?
>  * see if budget to provide more github runners and/or jenkins nodes
>  * check for performance bottleneck in the build and test process (develocity 
> build scan performance can give some ideas)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to