[
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18020644#comment-18020644
]
Jean Helou commented on JAMES-3978:
-----------------------------------
https://issues.apache.org/jira/browse/INFRA-25461 has been handled providing a
remote build cache node for james
I worked on adding cache support to scala compilation and more importantly on
test execution
The results are both good and disappointing :
compilation stage time went from ~7 minutes to 2:45 min mean
!2025-09-16_13-03-james-build.png!
test stage time went from ~4h45min to ~1h45min
!2025-09-16_13-05-james-tests.png!
This is a great result of course reducing the build time by 66%
I don't think ewe can achieve much more from external changes, there are still
a lot of uncacheable tasks but they account for a very small part of the build
time ( ~2 minutes )
I said the results were disappointing because 1h45 is still very long and the
CI build remains a slow feedback loop. unfortunately further improvements
require an indepth analysis of the depedency chains and actual tasks that keep
running for all the builds.
an empirical study of the various build scans since the build cache was merged
shows that this is mostly a recurring series of long tasks that takes most of
the time ( long being >10s )
!2025-09-16_13-16-long tasks.png!
at first glance I distinguish 3 categories
* Server assembly "unit" tests ( Server :: Memory :: App , Server :: Binaries
:: pulsar based smtp processing with scaling capabilities, Server::
Distributed:: App )
* Various integration tests (Web AdminServer integration tests :: Memory, Web
AdminServer integration tests :: Distributed, Web AdminServer integration tests
:: Postgres App, JMAP-RFC8621 :: * Integration)
* weird outliers ( Third Party :: SpamAssassin, Third Party :: Crowdsec)
> Investigate using develocity to improve james build
> ---------------------------------------------------
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
> Issue Type: Improvement
> Components: Build System
> Reporter: Jean Helou
> Priority: Major
> Attachments: 2025-09-16_13-03-james-build.png,
> 2025-09-16_13-05-james-tests.png, 2025-09-16_13-16-long tasks.png,
> image-2024-02-02-16-30-20-026.png, image-2024-02-02-16-31-24-883.png
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Hello,
> A while ago I noticed [this
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
> gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity
> service to the ASF. It so happens that I investigated develocity as part of
> my day job, and while we can't afford it there it is definitely something we
> might leverage here !
> The tool is available at [https://ge.apache.org|https://ge.apache.org/]
> I want to investigate the following leads
>
> # local caching for builds enable reuse of tasks output to speed up repeated
> tasks
> # build scans to provide detailed build reports and individual build metrics
> # build metrics are collected across all build scans and allow to target
> objectively bad contributors to overall build time
> # enabling the remote build cache
> ** additionally improves both local and CI build times
> ** requires some attention to read/write permissions to avoid cache
> poisoning and supply chain attacks
> ** most likely requires a dedicated build cache node request to infra
> ** several projects already use it so it is a known unknown ( we know it can
> be done we don't yet know how)
> # leverage advanced test features of develocity unknown unknowns at least
> for me:D
> ## the tool collects tests metrics to pinpoint flaky tests enabling targeted
> fix efforts
> ## the tool claims to be able to do test avoidance and not rerun tests that
> are irrelevant to the changes
> ## the tool claims to be able to distribute test excecution accross
> dedicated test workers
> Since we use maven, we won't get the full benefits of the build cache (
> gradle has been pushing plugin authors to write build cache friendly code for
> a long time ) but it should improve the current situation quite a bit.
>
> Apache pulsar is also using develocity with maven:
> - develocity configuration
> [https://github.com/apache/pulsar/blob/master/.mvn/gradle-enterprise.xml]
> - infra wiki has infos on how to onboard
> https://cwiki.apache.org/confluence/display/INFRA/Project+Onboarding+Instructions+for+Develocity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
