On Sat, 10 Aug 2024, Simon Josefsson wrote: > Yaroslav Halchenko <[email protected]> writes: > > Q2. is there work being done to establish/provide some kind of CDN > > or caching proxy setup for snapshot.debian.org?
> I've been using http://snapshot-cloudflare.debian.org/ and the majority > of files are possible to download, but there are some files that are > notorious hard to get that way and always tend to raise 503 errors or > just times out. I haven't been able to find a pattern, but I've managed > to download around 150TB of snapshot's 180TB of data via cloudflare so > it is reasonable usable. The speed from my point of view seems limited > to around 50MB/s so there is room for improvement still. Thank you Simon! I guess I could immediately "take advantage" from it but adding more of APT lines, i.e. pointing to the official repo and your mirror. Is your mirror updated daily as for the metadata/new snapshots? Could official snapshot folks assist in transferring those remaining TBs which cause 503s? PS on my end I mitigated by adjusting limits even further down: -RUN echo 'Acquire::http::Dl-Limit "1000";' >| /etc/apt/apt.conf.d/20snapshots \ - && echo 'Acquire::https::Dl-Limit "1000";' >> /etc/apt/apt.conf.d/20snapshots \ - && echo 'Acquire::Retries "5";' >> /etc/apt/apt.conf.d/20snapshots +RUN echo 'Acquire::http::Dl-Limit "200";' >| /etc/apt/apt.conf.d/20snapshots \ + && echo 'Acquire::https::Dl-Limit "200";' >> /etc/apt/apt.conf.d/20snapshots \ + && echo 'Acquire::Retries "10";' >> /etc/apt/apt.conf.d/20snapshots and managed to build that docker image on CI. Cheers, -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 WWW: http://www.linkedin.com/in/yarik
