Stefan Miklosovic created CASSANDRA-21489:
---------------------------------------------
Summary: Builds are timing out on downloading Cassandra from
archives
Key: CASSANDRA-21489
URL: https://issues.apache.org/jira/browse/CASSANDRA-21489
Project: Apache Cassandra
Issue Type: Bug
Reporter: Stefan Miklosovic
I see in
{code}
curl -sk "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText";
> /tmp/output.txt
{code}
first column is a line number
{code}
1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
/tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
balanced tokens for non-vnode cluster
1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
/tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
balanced tokens for non-vnode cluster
1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
/tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
balanced tokens for non-vnode cluster
1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
git:cassandra-4.1 failed. Attempted to delete
/home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
will need to be manually deleted
1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call last):
1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/bin/ccm", line 141, in <module>
1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
line 144, in run
1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
cluster_class(self.path, self.name, install_dir=self.options.install_dir,
version=self.options.version, verbose=self.optio ns.verbose,
options=self.options)
1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
100, in __init__
1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
self.load_from_repository(version, verbose)
1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
119, in load_from_repository
1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
repository.setup(version, verbose)
1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
58, in setup
1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
__setup(version, verbose)
1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
91, in __setup
1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
version, verbose=verbose)
1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
277, in clone_development
1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
176, in clone_development
1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not do
a git clone"
1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a git
clone
{code}
My diagnosis is that "archive" is super slow to download from
● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
'4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
test && ccm remove test ; done
(and the identical pattern for 4.0.$i and 5.0.$i).
Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
release from .1 up to the latest. The latest_4_1 step finds the newest (say
11), then seq 1 $latest_4_1
loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them: 4.1.1,
4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and 5.0.x. The
intent is to pre-bake
every historical release into the image so upgrade dtests can spin up any old
version offline.
Why that's slow (your archive point)
ccm downloads binary releases from archive.apache.org, and that host is
deliberately rate-limited/throttled by ASF infra (it's meant for occasional
historical access, not bulk/CI
pulls). Only the latest release of each line lives on the fast
downloads.apache.org CDN; every older point release (4.1.1–4.1.10, all the old
4.0.x, etc.) has been rotated off the
mirror and exists only on the slow archive. So the loop pulls ~49 MB each for
roughly:
- 4.0.x: ~18 releases
- 4.1.x: ~11 releases
- 5.0.x: ~9 releases
≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled archive
— one after another. That's easily 10+ minutes (you saw step 57/62 sitting
there), which is exactly
what pushes the image build over its timeout.
And this is why it "suddenly" started failing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
