On Mon, Sep 14, 2020 at 09:35:01PM +0200, Johannes Schauer wrote: > Hi, > > On Mon, 14 Sep 2020 18:50:44 +0200 Julian Andres Klode <j...@debian.org> > wrote: > > On Mon, Sep 14, 2020 at 05:18:20PM +0100, James Addison wrote: > > > Package: snapshot.debian.org > > > Followup-For: Bug #959518 > > > X-Debbugs-Cc: j...@jp-hosting.net > > > > > > The issue appears reproducible at the moment with apt 1.8.2.1 compiled > > > from source and the 'x.tar' configuration provided earlier. > > > > > > # apt source directory, post-build > > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \ > > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y > > > openjdk-11-jdk > > > > > > ... > > > > > > Get:261 > > > http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 > > > 11.0.7+10-3~deb10u1 [215 MB] > > > Err:261 > > > http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 > > > 11.0.7+10-3~deb10u1 > > > Undetermined Error [IP: 193.62.202.27 80] > > > > > > This has occurred for a couple of different server IP addresses, > > > including 185.17.185.185. > > > > We only care about unstable for this bug. There is a whole bunch of > > changes in http code and they won't be backported to stable releases. > > > > Also, the previous comment by Alex Thiessen indicated that this is not a > > bug in apt, but the server seems to close the connection, which means > > there is nothing actionable here. > > > > If you can produce an issue with the version of apt in unstable, > > and it does not reproduce with wget or curl, please open a new bug report > > for > > it. > > I'm very familiar with snapshot.d.o from the client perspective. Julian is > correct, that it's the server closing the connection. But that doesn't mean > that it's not at least a wishlist bug or feature request in apt. Let me > explain > a bit more. > > For several projects (debrebuild, debbisect, buildprofile QA, > bootstrap.debian.net...) I regularly interact with snapshot.d.o. Doing this > plainly with apt is deemed to fail miserably with errors like: > > # E: Failed to fetch [...] Error reading from server. Remote end closed > connection > # E: Failed to fetch [...] Hash Sum mismatch > # E: Failed to fetch [...] Bad header line Bad header data > # Err:118 [...] Connection timed out > > Yes, this is because of how snapshot.d.o throttles connections. For example > without additional measures, the following will fail:
Frankly, it should throttle connections not kill them. Generally you do that by refusing new GET requests or sending less data per time. > > $ curl > http://snapshot.debian.org/archive/debian/20200909T084102Z/pool/main/q/qtwebengine-opensource-src/qtwebengine-opensource-src_5.14.2+dfsg1.orig.tar.xz > >/dev/null > curl: (18) transfer closed with 217347024 bytes remaining to read > > There are a couple of things that can be done to work around this problem when > using curl by adding options like: > > --limit-rate=800k # this has the biggest effect > --retry 10 --retry-connrefused > --resolve snapshot.debian.org:80:193.62.202.27 > > But even those are not sufficient as snapshot.d.o will also cut the connection > early enough such that curl will fail with "network unreachable" which is not > a > transient error, so curl will not retry establishing the connection. > > The only thing that reliably worked for me with snapshot.d.o was the pycurl > based Python code at the end of this E-Mail. With that code, I can even > download for a full day reliably from snapshot.d.o without ever having hit the > Exception in the last line. > > But as things stand, it is impossible to reliably use apt together with > snapshot.d.o. I'm not sure how to solve this problem. One way could surely be > to approach snapshot.d.o and ask them to somehow lift their very heavy > throttling policies. But another way to solve this problem would be to make > apt > more resilient about mirrors with heavy throttling policies. I can think of > these wishlist bugs against apt: > > - allow to specify a maximum bytes per second value for downloads (this has > the largest effect if set low enough) That's Acquire::http::Dl-Limit > - allow to set an option that makes apt automatically retry when a transient > error occurs That's Acquire::Retries - well it retries in general I suppose. It misses a bits like DNS rotation, SRV rotation, but the goal is that this becomes the default sometime next year, with 3 retries per URL or something to work with flaky mirrors. > > - allow to set custom resolve addresses for domains like done in my code > below That we don't have. Gotta use /etc/hosts Anyway, if you know you use snapshots.d.o, and it's flaky, maybe make use of Dl-Limit and Retries option? -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en