Hi Beam devs, I'm Clay Johnson, and I support the Develocity instance at develocity.apache.org. I think Beam's CI could get a lot more out of its caching, and I'd like to propose a change.
The GitHub Actions cache for apache/beam is over its 10 GB limit (about 13 GB today; one pip entry alone is 6.4 GB), so entries keep getting evicted and jobs often start cold. In the last two weeks of October, 59% of Beam's ~31,000 Gradle CI builds still downloaded dependencies, about 6.4 TB in total, and configuration took about 43 seconds per build. All of that downloading also ties CI to the health of upstream repositories. Since October 13, 53 Beam CI builds hit connection timeouts or 502/503 errors from Maven Central's GCS mirror, repository.apache.org, or the Gradle Plugin Portal, and on October 30 two runs of :sdks:java:io:iceberg:dataflowIntegrationTest failed outright when repository.apache.org timed out. Two Develocity features address this: - Artifact Cache stores Gradle, pip, and Docker dependencies, with no 10 GB limit. That covers the pip cache and Docker images too, which Beam doesn't cache effectively today. Because the dependencies are served from right next to the runners, builds also make far fewer trips to upstream repositories. - Setup Cache reuses Gradle's configuration phase across builds, which goes after those ~43 seconds. What it would take: Beam would run a Develocity Edge node in GCP us-central1, next to the self-hosted runners in apache-beam-testing. Both caches live on the Edge, and it has to be close to the runners to pay off, so it can't sit with develocity.apache.org. We'd help with the Edge setup and sizing, and send the PR for the workflow change. We recommend starting with one workflow, such as beam_PreCommit_Java, and expanding to the rest from there. We'll share the before-and-after numbers here as we go. Would folks be open to that? Happy to answer questions any questions you might have about it. Thanks, Clay Johnson -- * CONFIDENTIALITY NOTICE*: The contents of this email message, and any attachments, are intended solely for the addressee(s) and may contain confidential, proprietary and/or privileged information legally protected from disclosure. If you are not the intended recipient of this communication, or if you received this communication by mistake, please notify the sender immediately and delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, retransmission, dissemination, copying or storage of this message or its attachments is strictly prohibited.
