bneradt opened a new pull request, #485: URL: https://github.com/apache/trafficserver-ci/pull/485
Each docker agent has its own local ccache. Jenkins usually sends a job back to the host it last ran on, so that works for some jobs. But URTest's shards spread across hosts, and any job landing on a host it hasn't visited lately compiles cold. That takes 17 to 21 minutes on these 4-core hosts. For example, the first URTest builds ([urtest/1](https://ci.trafficserver.apache.org/job/Github_Builds/job/urtest/1/), [/2](https://ci.trafficserver.apache.org/job/Github_Builds/job/urtest/2/), [/3](https://ci.trafficserver.apache.org/job/Github_Builds/job/urtest/3/)) spent 18.6 to 21.3 minutes in Build. A shared cache lets an object compiled on any agent hit on all of them. ## Measurements Measured on docker11 with the `ci-fedora-cxx20` preset (1,426 compiles). The 1.5 ms delay is simulated and matches the RTT measured between docker5 and docker11. | Scenario | Compile time | |---|---| | Cold, no cache | 1031 s | | Cold, writing to the remote cache | 1005 s | | Warm local cache | 69 s | | Warm remote cache, 1.5 ms RTT | 71 s | | Remote cache from master 3 commits later (two header changes) | 446 s, 80% hits | | Remote cache from master 40 commits later | 584 s, 71% hits | The network adds about 2 s to a whole build. ## Changes - `ccache-remote/`: a self-contained package, in the same style as `github-mirror/`, for a [bazel-remote](https://github.com/buchgr/bazel-remote) server on docker11. docker11 is out of Jenkins rotation; its Java 11 can't run the current agent, which needs Java 17. The package has: - a compose file; - a systemd unit; - an idempotent `bin/install.sh`; - `config/allowed-clients.txt`, the agent addresses; - a README covering client requirements, operations, and rollback. - The server listens only on HTTP `9090` and uses host networking, so ufw controls access. A published Docker port would bypass ufw. bazel-remote has no authentication, so the ufw allow list is the access control. - Disk use is bounded by bazel-remote's `--max_size`, 60 GiB by default, which evicts least recently used entries. docker11 has about 108 GB free. ccache's own `max_size` doesn't apply to remote storage. - `urtest.pipeline`, `fedora.pipeline` and `debian.pipeline` set `CCACHE_REMOTE_STORAGE`. Their images ship ccache 4.12 (`fedora:44`) and 4.7.5 (`debian:12`). Rocky 8, CentOS 7 and Ubuntu 20.04 ship ccache 3.7, which has no remote storage support, so those pipelines are unchanged. ## Notes - ATS builds use `-g`, so ccache's `hash_dir` (on by default) puts the working directory in the hash. Cross-agent hits work because Jenkins gives a job the same workspace path on every agent. Each agent has one executor, so no `@2` workspaces appear; none exist on the agents I checked. This PR doesn't change `hash_dir`. - If the server is down or blocked, ccache counts a miss and compiles locally. I confirmed this on docker11 with ufw blocking the port. ## Testing - Ran the compose file on docker11 under its docker-compose 1.25. Only 9090 was listening, with the gRPC and profiling endpoints disabled. - Both `fedora:44` and `debian:12` clients wrote to the cache and then hit on a second run. - Tested the ufw sync in a throwaway container with its own network namespace. It added missing addresses, removed ones taken off the list, and did nothing on a rerun. ## Deployment The pipeline change only helps once the server is running. After this merges, run `sudo ccache-remote/bin/install.sh` on docker11. That adds the ufw rules and starts the service. Builds until then fall back to their local caches. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
