[
https://issues.apache.org/jira/browse/CASSANDRA-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094301#comment-18094301
]
Stefan Miklosovic commented on CASSANDRA-21489:
-----------------------------------------------
Between #77's base and #80's base, the CI Dockerfile changed:
git diff abd7dea..e1e56e5 -- .build/docker/ubuntu-test.docker
.build/docker/ubuntu-test.docker | 49
+++++++++++++++++++++++++++++++++++++---
1 file changed, 46 insertions(+), 3 deletions(-)
via commit 464b2e54f45 "Support Python 3.12 and 3.13 in cqlsh" (this is
exactly the virtualenv --python=python3.12 / env3.12 block you saw in the log,
steps ~#48–#55).
Why that breaks it: the test image is layer-cached, and its cache key is the
Dockerfile. Build #77 ran on a base where the image was already built and
cached → the ccm
version-precache steps (the git clone of apache/cassandra.git + the binary:
archive downloads, steps #57–#62) were cache hits → skipped → fast. Commit
464b2e54f45 modified the
Dockerfile, so from #78 onward the cache is invalidated at that layer and
everything after it rebuilds from scratch — which re-runs the full git clone
--bare of
apache/cassandra.git and the ~35 slow archive.apache.org tarball downloads.
That from-scratch rebuild is what now blows the timeout.
So the sequence is:
1. #77 base (Jun 24): cached image → no clone/download → passes.
2. 464b2e54f45 (Python 3.12/3.13) edits ubuntu-test.docker → image cache
busted.
3. #78+ (Jul 1+): every run rebuilds the image → slow ccm clone + archive
downloads → timeout/abort.
> Builds are timing out on downloading Cassandra from archives
> ------------------------------------------------------------
>
> Key: CASSANDRA-21489
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21489
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Stefan Miklosovic
> Priority: Normal
>
> I see in
> {code}
> curl -sk
> "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText"; >
> /tmp/output.txt
> {code}
> first column is a line number
> {code}
> 1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
> 1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
> /tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
> 1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
> 1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
> 1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
> 1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
> /tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
> 1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
> 1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
> 1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
> 1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
> /tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
> 1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
> 1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
> 1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
> 1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
> git:cassandra-4.1 failed. Attempted to delete
> /home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
> will need to be manually deleted
> 1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call
> last):
> 1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/bin/ccm", line 141, in <module>
> 1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
> 1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
> line 144, in run
> 1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
> cluster_class(self.path, self.name, install_dir=self.options.install_dir,
> version=self.options.version, verbose=self.optio ns.verbose,
> options=self.options)
> 1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 100, in __init__
> 1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
> self.load_from_repository(version, verbose)
> 1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 119, in load_from_repository
> 1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
> repository.setup(version, verbose)
> 1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 58, in setup
> 1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
> __setup(version, verbose)
> 1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 91, in __setup
> 1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
> version, verbose=verbose)
> 1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 277, in clone_development
> 1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
> 1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 176, in clone_development
> 1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not
> do a git clone"
> 1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a
> git clone
> {code}
> My diagnosis is that "archive" is super slow to download from
> ● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
> Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
> {code}
> latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
> '4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
> for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
> test && ccm remove test ; done
> {code}
> (and the identical pattern for 4.0.$i and 5.0.$i).
> Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
> It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
> release from .1 up to the latest. The latest_4_1 step finds the newest (say
> 11), then seq 1 $latest_4_1
> loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them:
> 4.1.1, 4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and
> 5.0.x. The intent is to pre-bake
> every historical release into the image so upgrade dtests can spin up any
> old version offline.
> Why that's slow (your archive point)
> ccm downloads binary releases from archive.apache.org, and that host is
> deliberately rate-limited/throttled by ASF infra (it's meant for occasional
> historical access, not bulk/CI pulls). Only the latest release of each line
> lives on the fast downloads.apache.org CDN; every older point release
> (4.1.1–4.1.10, all the old 4.0.x, etc.) has been rotated off the mirror and
> exists only on the slow archive. So the loop pulls ~49 MB each for roughly:
> - 4.0.x: ~18 releases
> - 4.1.x: ~11 releases
> - 5.0.x: ~9 releases
> ≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled
> archive — one after another. That's easily 10+ minutes (you saw step 57/62
> sitting there), which is exactly
> what pushes the image build over its timeout.
> And this is why it "suddenly" started failing
> How to actually fix it
> - Stop pre-caching every point release. For upgrade tests you almost never
> need all of them — cache just the latest of each line (drop the seq 1 $latest
> loop; use
> binary:4.1.$latest_4_1 only), or an explicit small allow-list of the
> versions the dtests actually use.
> - Pull from a faster source: point ccm at a local artifact mirror /
> internal cache, or use downloads.apache.org where the version exists and hit
> archive only for the
> genuinely-needed old ones.
> - Make it resilient: wrap the loop in retries and/or bump the step timeout
> so a slow archive day doesn't abort the whole run.
> - Short term: just re-run — but it'll keep degrading as more releases ship
> until the loop is trimmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
