Just one comment. This might also be interesting for both Clay and the Beam PMC.
We already have a shared ASF action that uses Artifacts as a cache. It uses the same semantics as regular cache action, and offers several benefits: https://github.com/apache/infrastructure-actions/tree/main/stash. Jacob Wujciak created it. Several PMCs use it successfully. But perhaps there are some differences? I would be interested to see - or maybe we can also improve the actions? J. On Mon, Nov 9, 2026 at 6:59 AM Clay Johnson via dev <[email protected]> wrote: > Hi Beam devs, > > I'm Clay Johnson, and I support the Develocity instance at > develocity.apache.org. I think Beam's CI could get a lot more out of its > caching, and I'd like to propose a change. > > The GitHub Actions cache for apache/beam is over its 10 GB limit (about 13 > GB today; one pip entry alone is 6.4 GB), so entries keep getting evicted > and jobs often start cold. In the last two weeks of October, 59% of Beam's > ~31,000 Gradle CI builds still downloaded dependencies, about 6.4 TB in > total, and configuration took about 43 seconds per build. > > All of that downloading also ties CI to the health of upstream > repositories. Since October 13, 53 Beam CI builds hit connection timeouts > or 502/503 errors from Maven Central's GCS mirror, repository.apache.org, > or the Gradle Plugin Portal, and on October 30 two runs of > :sdks:java:io:iceberg:dataflowIntegrationTest failed outright when > repository.apache.org timed out. > > Two Develocity features address this: > > > - Artifact Cache stores Gradle, pip, and Docker dependencies, with no > 10 GB limit. That covers the pip cache and Docker images too, which Beam > doesn't cache effectively today. Because the dependencies are served from > right next to the runners, builds also make far fewer trips to upstream > repositories. > - Setup Cache reuses Gradle's configuration phase across builds, which > goes after those ~43 seconds. > > > What it would take: Beam would run a Develocity Edge node in GCP > us-central1, next to the self-hosted runners in apache-beam-testing. Both > caches live on the Edge, and it has to be close to the runners to pay off, > so it can't sit with develocity.apache.org. We'd help with the Edge setup > and sizing, and send the PR for the workflow change. > > We recommend starting with one workflow, such as beam_PreCommit_Java, and > expanding to the rest from there. We'll share the before-and-after numbers > here as we go. > > Would folks be open to that? Happy to answer questions any questions you > might have about it. > > Thanks, > Clay Johnson > > > *CONFIDENTIALITY NOTICE*: The contents of this email message, and any > attachments, are intended solely for the addressee(s) and may contain > confidential, proprietary and/or privileged information legally protected > from disclosure. If you are not the intended recipient of this > communication, or if you received this communication by mistake, please > notify the sender immediately and delete this message and any attachments. > If you are not the intended recipient, you are hereby notified that any > use, retransmission, dissemination, copying or storage of this message or > its attachments is strictly prohibited. >
