[
https://issues.apache.org/jira/browse/CASSANDRA-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094300#comment-18094300
]
Stefan Miklosovic commented on CASSANDRA-21489:
-----------------------------------------------
1. It was the first git: create in the image build → _git_cache_apache didn't
exist → ccm ran git clone --bare https://github.com/apache/cassandra.git — a
full bare mirror of the entire Cassandra history (~1 GB packed).
2. That clone ran ~606 s (~10 min) and returned a non-zero exit code → out !=
0 → assert out == 0, "Could not do a git clone" raised at line 176.
3. The except handler tried to rmdirs(target_dir) (gitCOLONcassandra-4.1) —
which didn't cleanly exist/delete (the failure happened at the cache clone,
before the target was made), so it printed the "Building C version … failed.
Attempted to delete … but failed"* line (1302517).
4. finally: raise e re-raised the AssertionError (line 277) → ccm exits
non-zero → Docker RUN → exit code: 1 → failed to solve.
Why you can't see the real git error
Look at out, _, _ = log_info(process, logger). log_info runs the subprocess
and streams git's stdout/stderr into ccm's logfile (~/.ccm/…/*.log via
lastlogfilename()), returning
only the exit code as out. So the actual fatal: … / RPC failed / early EOF
went to a ccm logfile that Jenkins doesn't print — the console only ever shows
the generic assert out ==
0, "Could not do a git clone". That's a ccm limitation, and it's why the
failure looks opaque. (To get the real cause you'd need ~/.ccm logs from that
agent.)
> Builds are timing out on downloading Cassandra from archives
> ------------------------------------------------------------
>
> Key: CASSANDRA-21489
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21489
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Stefan Miklosovic
> Priority: Normal
>
> I see in
> {code}
> curl -sk
> "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText"; >
> /tmp/output.txt
> {code}
> first column is a line number
> {code}
> 1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
> 1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
> /tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
> 1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
> 1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
> 1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
> 1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
> /tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
> 1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
> 1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
> 1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
> 1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
> /tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
> 1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
> 1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
> 1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
> 1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
> git:cassandra-4.1 failed. Attempted to delete
> /home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
> will need to be manually deleted
> 1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call
> last):
> 1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/bin/ccm", line 141, in <module>
> 1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
> 1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
> line 144, in run
> 1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
> cluster_class(self.path, self.name, install_dir=self.options.install_dir,
> version=self.options.version, verbose=self.optio ns.verbose,
> options=self.options)
> 1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 100, in __init__
> 1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
> self.load_from_repository(version, verbose)
> 1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 119, in load_from_repository
> 1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
> repository.setup(version, verbose)
> 1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 58, in setup
> 1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
> __setup(version, verbose)
> 1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 91, in __setup
> 1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
> version, verbose=verbose)
> 1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 277, in clone_development
> 1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
> 1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 176, in clone_development
> 1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not
> do a git clone"
> 1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a
> git clone
> {code}
> My diagnosis is that "archive" is super slow to download from
> ● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
> Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
> {code}
> latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
> '4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
> for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
> test && ccm remove test ; done
> {code}
> (and the identical pattern for 4.0.$i and 5.0.$i).
> Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
> It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
> release from .1 up to the latest. The latest_4_1 step finds the newest (say
> 11), then seq 1 $latest_4_1
> loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them:
> 4.1.1, 4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and
> 5.0.x. The intent is to pre-bake
> every historical release into the image so upgrade dtests can spin up any
> old version offline.
> Why that's slow (your archive point)
> ccm downloads binary releases from archive.apache.org, and that host is
> deliberately rate-limited/throttled by ASF infra (it's meant for occasional
> historical access, not bulk/CI pulls). Only the latest release of each line
> lives on the fast downloads.apache.org CDN; every older point release
> (4.1.1–4.1.10, all the old 4.0.x, etc.) has been rotated off the mirror and
> exists only on the slow archive. So the loop pulls ~49 MB each for roughly:
> - 4.0.x: ~18 releases
> - 4.1.x: ~11 releases
> - 5.0.x: ~9 releases
> ≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled
> archive — one after another. That's easily 10+ minutes (you saw step 57/62
> sitting there), which is exactly
> what pushes the image build over its timeout.
> And this is why it "suddenly" started failing
> How to actually fix it
> - Stop pre-caching every point release. For upgrade tests you almost never
> need all of them — cache just the latest of each line (drop the seq 1 $latest
> loop; use
> binary:4.1.$latest_4_1 only), or an explicit small allow-list of the
> versions the dtests actually use.
> - Pull from a faster source: point ccm at a local artifact mirror /
> internal cache, or use downloads.apache.org where the version exists and hit
> archive only for the
> genuinely-needed old ones.
> - Make it resilient: wrap the loop in retries and/or bump the step timeout
> so a slow archive day doesn't abort the whole run.
> - Short term: just re-run — but it'll keep degrading as more releases ship
> until the loop is trimmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
