[
https://issues.apache.org/jira/browse/CASSANDRA-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101043#comment-18101043
]
Michael Semb Wever edited comment on CASSANDRA-21541 at 8/2/26 11:17 AM:
-------------------------------------------------------------------------
the docker-cache pod isn't serving faster (over LAN inside k8s cluster) than
the pulls from apache.frog.io over WAN…
a single docker-cache pod can't handle our thundering herd like frog's cdn can…
the docker-cache pod approach introduces three problems:
- a cold cache is ~6x slower than the existing "just use apache.jfrog.io"
- a warmed cache is ~2.5x slower then the existing "just use apache.jfrog.io"
- the solution introduces fragility: docker-cache becomes an additional thing
to operate and fix (corruptions in images already witnessed)
the benefit of the LAN over WAN is lost for us because our the way our pipeline
thundering herds the images. jfrog's CDN is better suited to handle this than a
single docker-cache pod.
this was based on what was observed in a few test runs:
1. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/11/stages/
dind pods are not configured with registry, pay attention to the timings of
what matters (there was be natural variance in other places). pulls against
jfrog.
2. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/12/stages/
all configured but the registry was cold and so nearly all
cassandra-ubuntu-test image (the big 35G one) pulls were missed and in parallel
hitting dockerhub.
3. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/13/stages/
the docker-cache is now warm (i guess) and pulls are fast now compared to (2),
but only as fast as (1).
4. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/14/stages/
image corruption and pre-condition failures
{noformat}
┌────────────────────────────────────────┬──────────────────────┬─────────────────────┬─────────────────────┐
│ Image class │ Run 11 — jfrog (WAN) │ Run 12 —
cache COLD │ Run 13 — cache WARM │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test (~35Gi) median │ 374s (~6.2 min) │ 2220s (~37
min) │ 960s (~16 min) │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test mean │ 370s │ 2209s
│ 938s │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test n (real pulls) │ 288 │ 261
│ 243 │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ build-img cells (redhat/lint/…) median │ 65s │ 123s
│ 134s │
└────────────────────────────────────────┴──────────────────────┴─────────────────────┴─────────────────────┘
{noformat}
My conclusion is that this ticket is not worth the effort and complication it
adds, and that focus on reducing the cassandra-ubuntu-test image size would be
productive in both results and simplification.
was (Author: michaelsembwever):
the docker-cache pod isn't serving faster (over LAN inside k8s cluster) than
the pulls from apache.frog.io over WAN…
a single docker-cache pod can't handle our thundering herd like frog's cdn can…
the docker-cache pod approach introduces three problems:
- a cold cache is ~6x slower than the existing "just use apache.jfrog.io"
- a warmed cache is ~2.5x slower then the existing "just use apache.jfrog.io"
- the solution introduces fragility: docker-cache becomes an additional thing
to operate and fix (corruptions in images already witnessed)
the benefit of the LAN over WAN is lost for us because our the way our pipeline
thundering herds the images. jfrog's CDN is better suited to handle this than a
single docker-cache pod.
this was based on what was observed in a few test runs:
1. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/11/stages/
dind pods are not configured with registry, pay attention to the timings of
what matters (there was be natural variance in other places). pulls against
jfrog.
2. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/12/stages/
all configured but the registry was cold and so nearly all
cassandra-ubuntu-test image (the big 35G one) pulls were missed and in parallel
hitting dockerhub.
3. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/13/stages/
the docker-cache is now warm (i guess) and pulls are fast now compared to (2),
but only as fast as (1).
4. https://pre-ci.cassandra.apache.org/job/cassandra-5.0-21541-mck/14/stages/
image corruption and pre-condition failures
{noformat}
┌────────────────────────────────────────┬──────────────────────┬─────────────────────┬─────────────────────┐
│ Image class │ Run 11 — jfrog (WAN) │ Run 12 —
cache COLD │ Run 13 — cache WARM │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test (~35Gi) median │ 374s (~6.2 min) │ 2220s (~37
min) │ 960s (~16 min) │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test mean │ 370s │ 2209s
│ 938s │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ ubuntu-test n (real pulls) │ 288 │ 261
│ 243 │
├────────────────────────────────────────┼──────────────────────┼─────────────────────┼─────────────────────┤
│ build-img cells (redhat/lint/…) median │ 65s │ 123s
│ 134s │
└────────────────────────────────────────┴──────────────────────┴─────────────────────┴─────────────────────┘
{noformat}
My conclusion is that this ticket is worth the effort and complication it adds,
and that focus on reducing the cassandra-ubuntu-test image size would be
productive in both results and simplification.
> CI: cache Docker Hub image pulls in-cluster for the k8s Jenkins agents
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-21541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21541
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: CI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
>
> In a Helm deployed CI, each CI agent runs its own docker-in-docker daemon
> on an emptyDir, blank to begin with. Every agent therefore pulls the same
> build and test images afresh; the largest, `cassandra-ubuntu-test`, is ~35Gi.
> At hundreds of concurrent agents this is slow, wastes bandwidth, and trips
> pull-rate limits.
> Add an in-cluster Docker Hub pull-through cache (a registry:2 proxy) and
> point the agents' `dind` at it via `--registry-mirror`. Each image is then
> fetched from Docker Hub once and served to every agent over the cluster LAN;
> `dind` falls back to Docker Hub directly if the cache is unreachable.
> Docker's `--registry-mirror` can only mirror Docker Hub, not an arbitrary
> registry such as apache.jfrog.io. So the prefetch in the Jenkinsfile is
> repointed from the apache.jfrog.io names to those same Docker Hub names.
> Patch changes:
> - `.jenkins/k8s/docker-cache.yaml`: the pull-through cache (Deployment,
> Service, PVC), pinned to the always-on controller node.
> - `.jenkins/k8s/jenkins-deployment.yaml`: `dind` `--registry-mirror` in all
> three agent templates.
> - `.build/run-ci`: `--setup/--only-setup` applies the cache; teardown
> removes it.
> - `.jenkins/Jenkinsfile`: prefetch the Docker Hub image names.
> The cache runs a single replica for now. It converts repeated WAN pulls
> into one origin fetch plus LAN reads; if the pod's egress becomes the
> bottleneck this can be scaled with replicas or an object-storage backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
