[
https://issues.apache.org/jira/browse/CASSANDRA-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094298#comment-18094298
]
Stefan Miklosovic commented on CASSANDRA-21489:
-----------------------------------------------
What I am not completely sure about is that this seems to be close enough but
not completely true, it is true that stuff is downloaded from "archive", but it
comes from CCM. The failure is cloning failed, not that downloading was
unsuccessful.
> Builds are timing out on downloading Cassandra from archives
> ------------------------------------------------------------
>
> Key: CASSANDRA-21489
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21489
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Stefan Miklosovic
> Priority: Normal
>
> I see in
> {code}
> curl -sk
> "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText"; >
> /tmp/output.txt
> {code}
> first column is a line number
> {code}
> 1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
> 1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
> /tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
> 1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
> 1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
> 1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
> 1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
> /tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
> 1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
> 1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
> 1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
> 1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
> /tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
> 1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
> 1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
> 1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
> 1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
> git:cassandra-4.1 failed. Attempted to delete
> /home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
> will need to be manually deleted
> 1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call
> last):
> 1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/bin/ccm", line 141, in <module>
> 1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
> 1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
> line 144, in run
> 1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
> cluster_class(self.path, self.name, install_dir=self.options.install_dir,
> version=self.options.version, verbose=self.optio ns.verbose,
> options=self.options)
> 1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 100, in __init__
> 1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
> self.load_from_repository(version, verbose)
> 1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 119, in load_from_repository
> 1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
> repository.setup(version, verbose)
> 1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 58, in setup
> 1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
> __setup(version, verbose)
> 1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 91, in __setup
> 1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
> version, verbose=verbose)
> 1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 277, in clone_development
> 1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
> 1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 176, in clone_development
> 1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not
> do a git clone"
> 1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a
> git clone
> {code}
> My diagnosis is that "archive" is super slow to download from
> ● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
> Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
> {code}
> latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
> '4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
> for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
> test && ccm remove test ; done
> {code}
> (and the identical pattern for 4.0.$i and 5.0.$i).
> Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
> It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
> release from .1 up to the latest. The latest_4_1 step finds the newest (say
> 11), then seq 1 $latest_4_1
> loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them:
> 4.1.1, 4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and
> 5.0.x. The intent is to pre-bake
> every historical release into the image so upgrade dtests can spin up any
> old version offline.
> Why that's slow (your archive point)
> ccm downloads binary releases from archive.apache.org, and that host is
> deliberately rate-limited/throttled by ASF infra (it's meant for occasional
> historical access, not bulk/CI pulls). Only the latest release of each line
> lives on the fast downloads.apache.org CDN; every older point release
> (4.1.1–4.1.10, all the old 4.0.x, etc.) has been rotated off the mirror and
> exists only on the slow archive. So the loop pulls ~49 MB each for roughly:
> - 4.0.x: ~18 releases
> - 4.1.x: ~11 releases
> - 5.0.x: ~9 releases
> ≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled
> archive — one after another. That's easily 10+ minutes (you saw step 57/62
> sitting there), which is exactly
> what pushes the image build over its timeout.
> And this is why it "suddenly" started failing
> How to actually fix it
> - Stop pre-caching every point release. For upgrade tests you almost never
> need all of them — cache just the latest of each line (drop the seq 1 $latest
> loop; use
> binary:4.1.$latest_4_1 only), or an explicit small allow-list of the
> versions the dtests actually use.
> - Pull from a faster source: point ccm at a local artifact mirror /
> internal cache, or use downloads.apache.org where the version exists and hit
> archive only for the
> genuinely-needed old ones.
> - Make it resilient: wrap the loop in retries and/or bump the step timeout
> so a slow archive day doesn't abort the whole run.
> - Short term: just re-run — but it'll keep degrading as more releases ship
> until the loop is trimmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
