Situation: We currently serve all unauthenticated git read traffic through the following URL: https://sourceware.org/git/glibc.git
Similarly for binutils/gdb: https://sourceware.org/git/binutils-gdb.git This includes all git clone and fetch traffic. We also handle all write traffic e.g. `git push` to the same host. This has resulted in a bottleneck at the host, that at times resulted in developers being unable to access sources due to IP deny lists, or the disabling of HTTPS [1] for secure access. Background: Unauthenticated, or anonymous traffic, usually takes many forms including developers doing git pull (in aggregate), CI/CD systems in downstream testing, downstream mirrors (like the gitlab/github ones we have), and LLMs pulling the latest sources. In such a git pull operation the majority of the CPU and memory time can be dominated by the CPU and IO required by the git-upload-pack process. There are some things that can be done on the server side to minimize the cost, likewise on the client side e.g. shallow clones, maintenance and tuning etc. However, the read traffic is anonymous and untrusted, but has a right to access the sources of the project; we don't discriminate against use [2]. We also want that reader to have confidence they are getting sources that they can trust e.g. HTTPS connected pull (following the glibc SSDLC [3]). A single host can be scaled "vertically" e.g. more CPUs, more memory etc. It is not possible to scale the single host read side in a horizontal fashion e.g. geographically close. A vertically scaled host absorbs read traffic spikes up to the limit of the current network which might not be possible to scale before hitting physical limits e.g. data center limits, cage network limits etc. Assessment: It should be possible to safely serve anonymous reads from a mirror. The requirements of this reader are likely sufficiently served by the eventual consistency that a replica can provide. A read-only mirror is a replica of the canonical repository and serves only anonymous git pulls. The server never accepts a git push and has no credentials with which to push. A compromise in the read-only mirror can be detected by comparison to the canonical read-write server. We can therefore scale horizontally by adding N mirror nodes behind a single hostname and distribute with geodns, round-robin, or anycast. Nodes can be added, routed around, replaced, all without impacting the read-write host, and can be operated by the same or different FOSS service providers. There are many robust FOSS solutions that exist to provide such services, but they require scaling horizontally and paying the cost for that scale in admin and hardware. If a read-only mirror is compromised it has no access to any other system and cannot alter history. The damage is limited (but serious) to serving stale or incorrect read data to anonymous clients, which while not done today is detectable via signed content and ref verification (which can be done continuously as a separated and distinct process). The best solution remains to get all commits signed, along with a transparency log, but a defense-in-depth approach can start slowly. The canonical repository remains the single source of truth and the only thing that accepts writes. Note that for now cgit/gitweb reads are not addressed in this RFC but remain a relevant amount of load and traffic that can be discussed in another RFC. Recommendation: The goal is to scale horizontally and ensure pulls succeed from anywhere. The recommended mechanism after discussion between the CTI TAC and LF IT, who deploys similar infrastructure, is grokmirror: https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/ This is the same tooling kernel.org uses to run its mirror network, and is similar to the 4-node mirror network the GNU Project is standing up on savannah, referenced in the prior DNS end points RFC[4]: https://lists.gnu.org/archive/html/savannah-hackers-public/2026-05/msg00034.html The recommendation is to stand up a read-only git mirror network, replicated from the canonical host with grokmirror, under a project service specific end point: git.glibc.coretoolchain.dev -> read-only git mirror network As in the DNS RFC [4], this dovetails into a conversation with the GNU Project and FSF over matching DNS service endpoints which we can document. This introduces the requirement to trust the mirror network operator, but provides horizontal scaling for service robustness. The best ways to mitigate trust risks are to make progress on signing all commits, setting up a transparency log, and generally adopting a defense-in-depth approach. Keep all authenticated push and write traffic on the canonical write end point from the DNS end points RFC: git.glibc.sourceware.org -> read/write git access This works to collapse all anonymous reads to a fixed set of trusted mirrors. While we don't control where the anonymous reads come from or where they go, providing for the availability of moving traffic to N trusted readers versus all anonymous traffic should be measurable. The mirror nodes would run on distinct VM systems and split across US, Canada, and the EU to provide local mirrors for developers. There will need to be some anti-abuse control at the nodes, but the scaling can be reviewed against cost to alleviate that during operation. This has a secure supply chain consequence in that grokmirror must write a manifest on update of the write side of git server in order to support optimal remote mirror pull updates e.g. fingerprint change. This is a much smaller attack surface than the AdaCore git hooks. Only "grok-manifest" has to run in the trusted path, uses only local data (configurable), and arguably manipulates data that should not block it e.g. sha1 (git show-ref output). There is going to be replication lag, and if a developer needs strong read-after-write consistency they should pull from the canonical server. We will monitor replication lag to determine how small we want to make that lag since it has costs. The CTI services are currently provided by LF IT. The service provider commits to GNU Ethical Repository hosting criteria Grade B for services. DNS end points can be moved in the future if service providers change. This should be scaled up to handle gcc, and binutils and gdb too. -- Cheers, Carlos. Notes: - It is a hard requirement that fetch_objstore be disabled in grokmirror. - coretoolchain.dev, being a .dev domain is HSTS-preloaded, no HTTP. [1] https://inbox.sourceware.org/overseers/[email protected]/ [2] https://www.gnu.org/software/repo-criteria.en.html "Does not discriminate against classes of users" [3] https://sourceware.org/glibc/wiki/SSDLC/Policy/glibc [4] https://inbox.sourceware.org/libc-alpha/[email protected]/
