apupier commented on code in PR #1831: URL: https://github.com/apache/camel-website/pull/1831#discussion_r4398618723
########## content/blog/2026/10/camel-core-ci-improvements/index.md: ########## @@ -0,0 +1,253 @@ +--- +title: "Scaling the Apache Camel Core CI: From a Few Green Builds a Year to Reliable Faster Feedback" +date: 2026-10-23 +draft: false +authors: [apupier] +categories: ["Community"] +keywords: ["apache camel", "continuous integration", "CI", "jenkins", "github actions", "develocity", "flaky tests", "maven", "test stability", "backports"] +preview: "A year ago the Camel Core CI on Jenkins had between one and four green builds per year. Here is how we made it stable and faster to support more branches, more pull requests and more changes." +--- + +## Where we were a year ago + +A year ago, the [Camel Core](https://github.com/apache/camel) CI on Jenkins was producing between **one and four fully green builds per year**, i.e. in success for the whole test matrix at the same time. The build was taking almost 6 hours when things were going fine. + +The number of flaky tests was very important causing committers to stop really looking to failures and just awaiting following build results hoping for a flaky test. And when it wasn't, it means searching through a larger set of commits given that for resource constraints we didn't keep the whole build history. + +At the same time, the workload kept growing: + +- more **backports to the LTS branches** + +| Year | Backport PRs merged | Per branch | +| ---------------- | ------------------: | -------------------------------------------------------------------------------- | +| 2024 | 160 | 4.4.x: 58, 4.8.x: 57, 4.0.x: 18, 3.22.x: 14, 3.21.x: 7, 3.x: 5, 4.3.x: 1 | +| 2025 | 164 | 4.10.x: 82, 4.14.x: 48, 4.8.x: 30, 4.4.x: 2, 4.16.x: 1, 3.22.x: 1 | +| 2026 (to 23 Oct) | 820 | 4.18.x: 415, 4.22.x: 259, 4.14.x: 141, 4.10.x: 3, 4.17.x: 1, 3.22.x: 1 | + +- more **pull requests** and more **changes** in general, several of them with **reviews mostly automated** + +| Year | PRs merged on `main` | +| ---------------- | -------------------: | +| 2024 | 3,610 | +| 2025 | 3,326 | +| 2026 (to 23 Oct) | 6,097 | + +On top of that, we started hitting the **GitHub Actions limits**, which are **shared among the whole Apache GitHub organization**. The queue time can go up to dozens of hours on worst days, regularly dozens of minutes. On a 30 minutes job that's quite consequent and slowing down the development process. + +It was time to give a bit more love to our CIs. The focus was first on Jenkins, but several actions benefitted to both of them. This is an on-going work but let's share some of the improvements results today as we reached an important milestone last week on Jenkins CI. + +## High-level figures: before and after + +Let's start with a high-level comparison of the improvements + +| Metric | A year ago | Today | +|--------|-----------|-------| +| Green Jenkins builds on main | 1 to 4 per year | several per days | +| Duration of the Jenkins build for main | 5h45| 1h24 | Review Comment: this top-level includes several improvements, not only the -T1C, which explains the difference with the other specific -T1C table -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
