[
https://issues.apache.org/jira/browse/CASSANDRA-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-21489:
------------------------------------------
Description:
I see in
{code}
curl -sk "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText";
> /tmp/output.txt
{code}
first column is a line number
{code}
1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
/tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
balanced tokens for non-vnode cluster
1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
/tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
balanced tokens for non-vnode cluster
1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
/tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
balanced tokens for non-vnode cluster
1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
git:cassandra-4.1 failed. Attempted to delete
/home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
will need to be manually deleted
1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call last):
1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/bin/ccm", line 141, in <module>
1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
line 144, in run
1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
cluster_class(self.path, self.name, install_dir=self.options.install_dir,
version=self.options.version, verbose=self.optio ns.verbose,
options=self.options)
1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
100, in __init__
1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
self.load_from_repository(version, verbose)
1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
119, in load_from_repository
1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
repository.setup(version, verbose)
1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
58, in setup
1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
__setup(version, verbose)
1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
91, in __setup
1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
version, verbose=verbose)
1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
277, in clone_development
1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
176, in clone_development
1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not do
a git clone"
1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a git
clone
{code}
My diagnosis is that "archive" is super slow to download from
● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
'4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
test && ccm remove test ; done
(and the identical pattern for 4.0.$i and 5.0.$i).
Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
release from .1 up to the latest. The latest_4_1 step finds the newest (say
11), then seq 1 $latest_4_1
loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them: 4.1.1,
4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and 5.0.x. The
intent is to pre-bake
every historical release into the image so upgrade dtests can spin up any old
version offline.
Why that's slow (your archive point)
ccm downloads binary releases from archive.apache.org, and that host is
deliberately rate-limited/throttled by ASF infra (it's meant for occasional
historical access, not bulk/CI
pulls). Only the latest release of each line lives on the fast
downloads.apache.org CDN; every older point release (4.1.1–4.1.10, all the old
4.0.x, etc.) has been rotated off the
mirror and exists only on the slow archive. So the loop pulls ~49 MB each for
roughly:
- 4.0.x: ~18 releases
- 4.1.x: ~11 releases
- 5.0.x: ~9 releases
≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled archive
— one after another. That's easily 10+ minutes (you saw step 57/62 sitting
there), which is exactly
what pushes the image build over its timeout.
And this is why it "suddenly" started failing
How to actually fix it
- Stop pre-caching every point release. For upgrade tests you almost never
need all of them — cache just the latest of each line (drop the seq 1 $latest
loop; use
binary:4.1.$latest_4_1 only), or an explicit small allow-list of the versions
the dtests actually use.
- Pull from a faster source: point ccm at a local artifact mirror / internal
cache, or use downloads.apache.org where the version exists and hit archive
only for the
genuinely-needed old ones.
- Make it resilient: wrap the loop in retries and/or bump the step timeout so
a slow archive day doesn't abort the whole run.
- Short term: just re-run — but it'll keep degrading as more releases ship
until the loop is trimmed.
was:
I see in
{code}
curl -sk "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText";
> /tmp/output.txt
{code}
first column is a line number
{code}
1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
/tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
balanced tokens for non-vnode cluster
1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
/tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
balanced tokens for non-vnode cluster
1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
/tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
balanced tokens for non-vnode cluster
1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO Downloading
http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
git:cassandra-4.1 failed. Attempted to delete
/home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
will need to be manually deleted
1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call last):
1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/bin/ccm", line 141, in <module>
1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
line 144, in run
1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
cluster_class(self.path, self.name, install_dir=self.options.install_dir,
version=self.options.version, verbose=self.optio ns.verbose,
options=self.options)
1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
100, in __init__
1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
self.load_from_repository(version, verbose)
1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
119, in load_from_repository
1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
repository.setup(version, verbose)
1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
58, in setup
1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
__setup(version, verbose)
1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
91, in __setup
1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
version, verbose=verbose)
1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
277, in clone_development
1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
"/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py", line
176, in clone_development
1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not do
a git clone"
1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a git
clone
{code}
My diagnosis is that "archive" is super slow to download from
● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
'4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
test && ccm remove test ; done
(and the identical pattern for 4.0.$i and 5.0.$i).
Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
release from .1 up to the latest. The latest_4_1 step finds the newest (say
11), then seq 1 $latest_4_1
loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them: 4.1.1,
4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and 5.0.x. The
intent is to pre-bake
every historical release into the image so upgrade dtests can spin up any old
version offline.
Why that's slow (your archive point)
ccm downloads binary releases from archive.apache.org, and that host is
deliberately rate-limited/throttled by ASF infra (it's meant for occasional
historical access, not bulk/CI
pulls). Only the latest release of each line lives on the fast
downloads.apache.org CDN; every older point release (4.1.1–4.1.10, all the old
4.0.x, etc.) has been rotated off the
mirror and exists only on the slow archive. So the loop pulls ~49 MB each for
roughly:
- 4.0.x: ~18 releases
- 4.1.x: ~11 releases
- 5.0.x: ~9 releases
≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled archive
— one after another. That's easily 10+ minutes (you saw step 57/62 sitting
there), which is exactly
what pushes the image build over its timeout.
And this is why it "suddenly" started failing
> Builds are timing out on downloading Cassandra from archives
> ------------------------------------------------------------
>
> Key: CASSANDRA-21489
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21489
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Stefan Miklosovic
> Priority: Normal
>
> I see in
> {code}
> curl -sk
> "https://pre-ci.cassandra.apache.org/job/cassandra-6.0/80/consoleText"; >
> /tmp/output.txt
> {code}
> first column is a line number
> {code}
> 1302501 [2026-07-07T09:12:38.293Z] #62 318.7 09:12:32,664 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-9j5ki1m8. tar.gz (49.227MB)
> 1302502 [2026-07-07T09:12:38.933Z] #62 350.1 09:12:36,708 ccm INFO Extracting
> /tmp/ccm-f07q4xj4.tar.gz as version 4.0.1 ...
> 1302503 [2026-07-07T09:12:38.933Z] #62 350.4 Current cluster is now: test
> 1302504 [2026-07-07T09:12:38.933Z] #62 350.4 09:12:36,987 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302505 [2026-07-07T09:12:38.933Z] #62 350.7 1
> 1302506 [2026-07-07T09:12:38.933Z] #62 350.8 09:12:37,399 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-a7as8an8. tar.gz (49.227MB)
> 1302507 [2026-07-07T09:12:39.080Z] #62 356.1 09:12:27,597 ccm INFO Extracting
> /tmp/ccm-lvyf_nqj.tar.gz as version 4.0.1 ...
> 1302508 [2026-07-07T09:12:39.080Z] #62 356.4 Current cluster is now: test
> 1302509 [2026-07-07T09:12:39.080Z] #62 356.4 09:12:27,874 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302510 [2026-07-07T09:12:39.080Z] #62 356.6 1
> 1302511 [2026-07-07T09:12:39.080Z] #62 356.7 09:12:28,235 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-g67x17v_. tar.gz (49.227MB)
> 1302512 [2026-07-07T09:12:40.064Z] #62 347.0 09:12:34,790 ccm INFO Extracting
> /tmp/ccm-rm4k5fk9.tar.gz as version 4.0.1 ...
> 1302513 [2026-07-07T09:12:40.064Z] #62 347.3 Current cluster is now: test
> 1302514 [2026-07-07T09:12:40.064Z] #62 347.3 09:12:35,69 ccm DEBUG using
> balanced tokens for non-vnode cluster
> 1302515 [2026-07-07T09:12:40.064Z] #62 347.5 1
> 1302516 [2026-07-07T09:12:40.064Z] #62 347.7 09:12:35,467 ccm INFO
> Downloading
> http://archive.apache.org/dist/cassandra/4.1.1/apache-cassandra-4.1.1-bin.tar.gz
> to /tmp/ccm-_pt6_55h. tar.gz (49.227MB)
> 1302517 [2026-07-07T09:12:45.076Z] #61 606.0 Building C* version
> git:cassandra-4.1 failed. Attempted to delete
> /home/cassandra/.ccm/repository/gitCOLONcassandra-4.1 but failed. This
> will need to be manually deleted
> 1302518 [2026-07-07T09:12:45.077Z] #61 606.0 Traceback (most recent call
> last):
> 1302519 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/bin/ccm", line 141, in <module>
> 1302520 [2026-07-07T09:12:45.077Z] #61 606.0 cmd.run()
> 1302521 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cmds/cluster_cmds.py",
> line 144, in run
> 1302522 [2026-07-07T09:12:45.077Z] #61 606.0 cluster =
> cluster_class(self.path, self.name, install_dir=self.options.install_dir,
> version=self.options.version, verbose=self.optio ns.verbose,
> options=self.options)
> 1302523 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 100, in __init__
> 1302524 [2026-07-07T09:12:45.077Z] #61 606.0 repo_dir, v =
> self.load_from_repository(version, verbose)
> 1302525 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/cluster.py", line
> 119, in load_from_repository
> 1302526 [2026-07-07T09:12:45.077Z] #61 606.0 return
> repository.setup(version, verbose)
> 1302527 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 58, in setup
> 1302528 [2026-07-07T09:12:45.077Z] #61 606.0 (cdir, version, fallback) =
> __setup(version, verbose)
> 1302529 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 91, in __setup
> 1302530 [2026-07-07T09:12:45.077Z] #61 606.0 clone_development(GIT_REPO,
> version, verbose=verbose)
> 1302531 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 277, in clone_development
> 1302532 [2026-07-07T09:12:45.077Z] #61 606.0 raise e
> 1302533 [2026-07-07T09:12:45.077Z] #61 606.0 File
> "/home/cassandra/env3.8/lib/python3.8/site-packages/ccmlib/repository.py",
> line 176, in clone_development
> 1302534 [2026-07-07T09:12:45.077Z] #61 606.0 assert out == 0, "Could not
> do a git clone"
> 1302535 [2026-07-07T09:12:45.077Z] #61 606.0 AssertionError: Could not do a
> git clone
> {code}
> My diagnosis is that "archive" is super slow to download from
> ● Your diagnosis is exactly right, and the Dockerfile confirms the mechanism.
> Here's the smoking gun (.build/docker/ubuntu-test.docker, lines 187–190):
> latest_4_1=$(curl -s https://downloads.apache.org/cassandra/ | grep -oP
> '4\.1\.[0-9]+' | sort -V | tail -1 | cut -d"." -f3)
> for i in $(seq 1 $latest_4_1); do ccm create --quiet -n 1 -v binary:4.1.$i
> test && ccm remove test ; done
> (and the identical pattern for 4.0.$i and 5.0.$i).
> Why it downloads 4.1.1 (and 4.1.2, 4.1.3, …), not just 4.1.11
> It's not downloading 4.1.1 instead of 4.1.11 — it downloads every point
> release from .1 up to the latest. The latest_4_1 step finds the newest (say
> 11), then seq 1 $latest_4_1
> loops 1,2,3,…,11 and runs ccm create -v binary:4.1.$i for all of them:
> 4.1.1, 4.1.2, … 4.1.11. Same for 4.0.x (which is up to ~4.0.18 now) and
> 5.0.x. The intent is to pre-bake
> every historical release into the image so upgrade dtests can spin up any
> old version offline.
> Why that's slow (your archive point)
> ccm downloads binary releases from archive.apache.org, and that host is
> deliberately rate-limited/throttled by ASF infra (it's meant for occasional
> historical access, not bulk/CI
> pulls). Only the latest release of each line lives on the fast
> downloads.apache.org CDN; every older point release (4.1.1–4.1.10, all the
> old 4.0.x, etc.) has been rotated off the
> mirror and exists only on the slow archive. So the loop pulls ~49 MB each
> for roughly:
> - 4.0.x: ~18 releases
> - 4.1.x: ~11 releases
> - 5.0.x: ~9 releases
> ≈ 35–40 tarballs, ~1.8 GB, and almost all of them from the throttled
> archive — one after another. That's easily 10+ minutes (you saw step 57/62
> sitting there), which is exactly
> what pushes the image build over its timeout.
> And this is why it "suddenly" started failing
> How to actually fix it
> - Stop pre-caching every point release. For upgrade tests you almost never
> need all of them — cache just the latest of each line (drop the seq 1 $latest
> loop; use
> binary:4.1.$latest_4_1 only), or an explicit small allow-list of the
> versions the dtests actually use.
> - Pull from a faster source: point ccm at a local artifact mirror /
> internal cache, or use downloads.apache.org where the version exists and hit
> archive only for the
> genuinely-needed old ones.
> - Make it resilient: wrap the loop in retries and/or bump the step timeout
> so a slow archive day doesn't abort the whole run.
> - Short term: just re-run — but it'll keep degrading as more releases ship
> until the loop is trimmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
