Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: snapshot.debian.org Followup-For: Bug #959518 X-Debbugs-Cc: j...@jp-hosting.net (sorry: I realized I meant 'compile ... using bullseye' in that previous message, not buster (stable))
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Quoting Julian Andres Klode (2020-09-14 21:49:15) > > - allow to specify a maximum bytes per second value for downloads (this > > has the largest effect if set low enough) > > That's Acquire::http::Dl-Limit > > > - allow to set an option that makes apt automatically retry when a > > transient > >error occurs > > That's Acquire::Retries - well it retries in general I suppose. It misses a > bits > like DNS rotation, SRV rotation, but the goal is that this becomes the default > sometime next year, with 3 retries per URL or something to work with flaky > mirrors. > > > > > - allow to set custom resolve addresses for domains like done in my code > > below > > That we don't have. Gotta use /etc/hosts > > Anyway, if you know you use snapshots.d.o, and it's flaky, maybe make use of > Dl-Limit and Retries option? I stand corrected. I think apt already offers all the tools needed to use snapshot.debian.org. Setting Acquire::http::Dl-Limit indeed does the job. And the option is even documented in apt-transport-http(1) Thanks! cheers, josch signature.asc Description: signature
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: j...@jp-hosting.net
Understood. The environment I'd like to get this working for is based on
Debian stable, so we might be at an impasse unless I can compile the latest apt
sources using buster. I'm making progress on that and attempting to resolve
some time64-related compilation issues (unrelated to this bug).
I'd like to keep this issue open against the snapshot.debian.org meta package
since that seems a good way to resolve the problem assuming it is purely
server-side.
Given the discussion so far would it make sense if I use BTS to attempt to
(re)assign this bug from packages {apt,snapshot.debian.org} ->
{snapshot.debian.org}?
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: snapshot.debian.org Followup-For: Bug #959518 X-Debbugs-Cc: j...@jp-hosting.net The issue appears reproducible at the moment with apt 1.8.2.1 compiled from source and the 'x.tar' configuration provided earlier. # apt source directory, post-build $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \ $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y openjdk-11-jdk ... Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB] Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 Undetermined Error [IP: 193.62.202.27 80] This has occurred for a couple of different server IP addresses, including 185.17.185.185.
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
On Mon, Sep 14, 2020 at 09:35:01PM +0200, Johannes Schauer wrote: > Hi, > > On Mon, 14 Sep 2020 18:50:44 +0200 Julian Andres Klode > wrote: > > On Mon, Sep 14, 2020 at 05:18:20PM +0100, James Addison wrote: > > > Package: snapshot.debian.org > > > Followup-For: Bug #959518 > > > X-Debbugs-Cc: j...@jp-hosting.net > > > > > > The issue appears reproducible at the moment with apt 1.8.2.1 compiled > > > from source and the 'x.tar' configuration provided earlier. > > > > > > # apt source directory, post-build > > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \ > > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y > > > openjdk-11-jdk > > > > > > ... > > > > > > Get:261 > > > http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 > > > 11.0.7+10-3~deb10u1 [215 MB] > > > Err:261 > > > http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 > > > 11.0.7+10-3~deb10u1 > > > Undetermined Error [IP: 193.62.202.27 80] > > > > > > This has occurred for a couple of different server IP addresses, > > > including 185.17.185.185. > > > > We only care about unstable for this bug. There is a whole bunch of > > changes in http code and they won't be backported to stable releases. > > > > Also, the previous comment by Alex Thiessen indicated that this is not a > > bug in apt, but the server seems to close the connection, which means > > there is nothing actionable here. > > > > If you can produce an issue with the version of apt in unstable, > > and it does not reproduce with wget or curl, please open a new bug report > > for > > it. > > I'm very familiar with snapshot.d.o from the client perspective. Julian is > correct, that it's the server closing the connection. But that doesn't mean > that it's not at least a wishlist bug or feature request in apt. Let me > explain > a bit more. > > For several projects (debrebuild, debbisect, buildprofile QA, > bootstrap.debian.net...) I regularly interact with snapshot.d.o. Doing this > plainly with apt is deemed to fail miserably with errors like: > > # E: Failed to fetch [...] Error reading from server. Remote end closed > connection > # E: Failed to fetch [...] Hash Sum mismatch > # E: Failed to fetch [...] Bad header line Bad header data > # Err:118 [...] Connection timed out > > Yes, this is because of how snapshot.d.o throttles connections. For example > without additional measures, the following will fail: Frankly, it should throttle connections not kill them. Generally you do that by refusing new GET requests or sending less data per time. > > $ curl > http://snapshot.debian.org/archive/debian/20200909T084102Z/pool/main/q/qtwebengine-opensource-src/qtwebengine-opensource-src_5.14.2+dfsg1.orig.tar.xz > >/dev/null > curl: (18) transfer closed with 217347024 bytes remaining to read > > There are a couple of things that can be done to work around this problem when > using curl by adding options like: > > --limit-rate=800k # this has the biggest effect > --retry 10 --retry-connrefused > --resolve snapshot.debian.org:80:193.62.202.27 > > But even those are not sufficient as snapshot.d.o will also cut the connection > early enough such that curl will fail with "network unreachable" which is not > a > transient error, so curl will not retry establishing the connection. > > The only thing that reliably worked for me with snapshot.d.o was the pycurl > based Python code at the end of this E-Mail. With that code, I can even > download for a full day reliably from snapshot.d.o without ever having hit the > Exception in the last line. > > But as things stand, it is impossible to reliably use apt together with > snapshot.d.o. I'm not sure how to solve this problem. One way could surely be > to approach snapshot.d.o and ask them to somehow lift their very heavy > throttling policies. But another way to solve this problem would be to make > apt > more resilient about mirrors with heavy throttling policies. I can think of > these wishlist bugs against apt: > > - allow to specify a maximum bytes per second value for downloads (this has >the largest effect if set low enough) That's Acquire::http::Dl-Limit > - allow to set an option that makes apt automatically retry when a transient >error occurs That's Acquire::Retries - well it retries in general I suppose. It misses a bits like DNS rotation, SRV rotation, but the goal is that this becomes the default sometime next year, with 3 retries per URL or something to work with flaky mirrors. > > - allow to set custom resolve addresses for domains like done in my code > below That we don't have. Gotta use /etc/hosts Anyway, if you know you use snapshots.d.o, and it's flaky, maybe make use of Dl-Limit and Retries option? -- debian developer - deb.li/jak | jak-linux.org - free
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Hi, On Mon, 14 Sep 2020 18:50:44 +0200 Julian Andres Klode wrote: > On Mon, Sep 14, 2020 at 05:18:20PM +0100, James Addison wrote: > > Package: snapshot.debian.org > > Followup-For: Bug #959518 > > X-Debbugs-Cc: j...@jp-hosting.net > > > > The issue appears reproducible at the moment with apt 1.8.2.1 compiled from > > source and the 'x.tar' configuration provided earlier. > > > > # apt source directory, post-build > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \ > > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y > > openjdk-11-jdk > > > > ... > > > > Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 > > [215 MB] > > Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z > > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 > > Undetermined Error [IP: 193.62.202.27 80] > > > > This has occurred for a couple of different server IP addresses, including > > 185.17.185.185. > > We only care about unstable for this bug. There is a whole bunch of > changes in http code and they won't be backported to stable releases. > > Also, the previous comment by Alex Thiessen indicated that this is not a > bug in apt, but the server seems to close the connection, which means > there is nothing actionable here. > > If you can produce an issue with the version of apt in unstable, > and it does not reproduce with wget or curl, please open a new bug report for > it. I'm very familiar with snapshot.d.o from the client perspective. Julian is correct, that it's the server closing the connection. But that doesn't mean that it's not at least a wishlist bug or feature request in apt. Let me explain a bit more. For several projects (debrebuild, debbisect, buildprofile QA, bootstrap.debian.net...) I regularly interact with snapshot.d.o. Doing this plainly with apt is deemed to fail miserably with errors like: # E: Failed to fetch [...] Error reading from server. Remote end closed connection # E: Failed to fetch [...] Hash Sum mismatch # E: Failed to fetch [...] Bad header line Bad header data # Err:118 [...] Connection timed out Yes, this is because of how snapshot.d.o throttles connections. For example without additional measures, the following will fail: $ curl http://snapshot.debian.org/archive/debian/20200909T084102Z/pool/main/q/qtwebengine-opensource-src/qtwebengine-opensource-src_5.14.2+dfsg1.orig.tar.xz >/dev/null curl: (18) transfer closed with 217347024 bytes remaining to read There are a couple of things that can be done to work around this problem when using curl by adding options like: --limit-rate=800k # this has the biggest effect --retry 10 --retry-connrefused --resolve snapshot.debian.org:80:193.62.202.27 But even those are not sufficient as snapshot.d.o will also cut the connection early enough such that curl will fail with "network unreachable" which is not a transient error, so curl will not retry establishing the connection. The only thing that reliably worked for me with snapshot.d.o was the pycurl based Python code at the end of this E-Mail. With that code, I can even download for a full day reliably from snapshot.d.o without ever having hit the Exception in the last line. But as things stand, it is impossible to reliably use apt together with snapshot.d.o. I'm not sure how to solve this problem. One way could surely be to approach snapshot.d.o and ask them to somehow lift their very heavy throttling policies. But another way to solve this problem would be to make apt more resilient about mirrors with heavy throttling policies. I can think of these wishlist bugs against apt: - allow to specify a maximum bytes per second value for downloads (this has the largest effect if set low enough) - allow to set an option that makes apt automatically retry when a transient error occurs - allow to set custom resolve addresses for domains like done in my code below I'm not saying that we shouldn't look into maybe making snapshot.d.o throttle less, because as things stand, it's impossible to use it together with apt. But there certainly also some things that apt can do and which will not only benefit people working with snapshot.d.o but also people who are otherwise using a mirror or proxy with heavy throttling. Thanks! cheers, josch def download(url): f = BytesIO() maxretries = 10 for retrynum in range(maxretries): try: c = pycurl.Curl() c.setopt( c.URL, url, ) # even 100 kB/s is too much sometimes c.setopt(c.MAX_RECV_SPEED_LARGE, 800 * 1024) # bytes per second c.setopt(c.CONNECTTIMEOUT, 30) # the default is 300 # sometimes, curl stalls forever and even ctrl+c doesn't work start = time.time() def progress(*data):
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
On Mon, Sep 14, 2020 at 05:18:20PM +0100, James Addison wrote: > Package: snapshot.debian.org > Followup-For: Bug #959518 > X-Debbugs-Cc: j...@jp-hosting.net > > The issue appears reproducible at the moment with apt 1.8.2.1 compiled from > source and the 'x.tar' configuration provided earlier. > > # apt source directory, post-build > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \ > $ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y > openjdk-11-jdk > > ... > > Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 > [215 MB] > Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 > Undetermined Error [IP: 193.62.202.27 80] > > This has occurred for a couple of different server IP addresses, including > 185.17.185.185. We only care about unstable for this bug. There is a whole bunch of changes in http code and they won't be backported to stable releases. Also, the previous comment by Alex Thiessen indicated that this is not a bug in apt, but the server seems to close the connection, which means there is nothing actionable here. If you can produce an issue with the version of apt in unstable, and it does not reproduce with wget or curl, please open a new bug report for it. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Dear maintainer, I'm facing this issue (apt 2.1.10 (amd64)) and here are a couple observations that might help you to grasp it better. snapshot.d.o servers close the connection randomly, it took wget four continuation retries to download the file completely. When called again, the download is completed by wget flawlessly (might be a transparent proxy kicking in, though). logs: $ wget http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb --2020-08-25 09:22:15-- http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ... Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 207236812 (198M) Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ openjdk-11-jdk-headless_11.0.3+ 9%[===> ] 17.97M 3.65MB/sin 5.2s 2020-08-25 09:22:21 (3.46 MB/s) - Connection closed at byte 18845696. Retrying. --2020-08-25 09:22:22-- (try: 2) http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 207236812 (198M), 188391116 (180M) remaining Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ openjdk-11-jdk-headless_11.0.3+ 14%[===> ] 29.46M 3.45MB/sin 3.3s 2020-08-25 09:22:25 (3.45 MB/s) - Connection closed at byte 30896128. Retrying. --2020-08-25 09:22:27-- (try: 3) http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 207236812 (198M), 176340684 (168M) remaining Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ openjdk-11-jdk-headless_11.0.3+ 20%[==> ] 40.87M 3.45MB/sin 3.3s 2020-08-25 09:22:31 (3.45 MB/s) - Connection closed at byte 42856448. Retrying. --2020-08-25 09:22:34-- (try: 4) http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 207236812 (198M), 164380364 (157M) remaining Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ openjdk-11-jdk-headless_11.0.3+ 100%[+++==>] 197.64M 4.11MB/sin 40s 2020-08-25 09:23:15 (3.88 MB/s) - ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ saved [207236812/207236812] $ rm openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb $ wget http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb --2020-08-25 09:25:38-- http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ... Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 207236812 (198M) Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ openjdk-11-jdk-headless_11.0.3+ 100%[=>] 197.64M 2.97MB/sin 54s 2020-08-25 09:26:32 (3.69 MB/s) - ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ saved [207236812/207236812] Best regards, Alex Thiessen
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: apt,snapshot.debian.org Followup-For: Bug #959518 X-Debbugs-Cc: j...@jp-hosting.net Thank you both! I should've done a more digging into the commit history for the related HTTP fixes. I'm not sure I would've easily figured out the apt method issue but it's a relief to see there's a command-line flag for that. I'll try this again from master soon and glad to help with any other debug (although as evidenced I'm a bit slow to figure out the various best practices for testing all this).
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
On Wed, Aug 05, 2020 at 12:12:18AM +0100, James Addison wrote: > Package: apt,snapshot.debian.org > Followup-For: Bug #959518 > X-Debbugs-Cc: j...@jp-hosting.net > > FWIW, the series of steps in use for repro are: > > # prereq: install apt compile-time dependencies > # prereq: add apt deb sources, keys as per docs at http://snapshot.debian.org/ > $ git clone "https://salsa.debian.org/apt-team/apt.git; > $ cd apt > $ git checkout 1.8.2.z > $ git cherry-pick -m 1 7d222636954ec95382149e31b314e9828ba05a2e You also need the commit before that. This fixed the undetermined error on top of the previous fixes, which probably moved the error into a different code path :) Especially commit cb743d117bcc666dab4c5948b1227ed2edbd0578 Author: Julian Andres Klode Date: Mon Jun 29 11:45:45 2020 +0200 http: Only return false for EOF if we actually did not read anything This should avoid the need to Flush the buffer in Die(), because if we read anything, we are returning true, and not entering Die() at that point. Also Write() does not have a concept of EOF, so get rid of code handling that there. Was that copied from Read()? But probably the entire merge. These all act badly individually. Really, just try the master branch instead of going cherry-picking individual commits. > $ cmake -DCMAKE_INSTALL_PREFIX=/ . > $ make > $ cmdline/apt -o Acquire::Check-Valid-Until=false update > $ cmdline/apt install -y openjdk-11-jdk This is wrong, as David already pointed out - it's missing -o Dir::Bin::Methods I'd also build using dpkg-buildpackage so you get all the right flags. I can never remember them all, but there's also stuff like libexec and so on that it passes. > > And the error manifests as: > > Get:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 > [215 MB] > Err:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z > buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 > Undetermined Error [IP: 193.62.202.27 80] > > If time allows it might be possible to prepare a self-contained Dockerfile > with the repro steps if that'd be useful for further debug/investigation. No, that's not useful at all. You just put a sources.list and a trusted.gpg.d in a directory (e.g. $PWD/x) with the other files, and then do apt -o Dir=$PWD/x update apt -o Dir=$PWD/x download openjdk-11-jdk-headless I have attached x.tar for convenience. And that reproduces the issue. Well at least it did early yesterday. Which is why I could debug it and come up with a fix for it. Though, now i can't reproduce the undetermined error without the fix either, I just get weird other errors... apt -o Dir=$PWD/x install openjdk-11-jdk -y -d certainly did. And then you can run this from a source tree like this (at least I do): dpkg-buildpackage -b ./obj-x86_64-linux-gnu/cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/obj-x86_64-linux-gnu/methods update ./obj-x86_64-linux-gnu/cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/obj-x86_64-linux-gnu/methods update You also want to pin snapshot.debian.org to the failing IP I think, as the two servers are vastly different. It's not super reproducible: jak@jak-t480s:~/Projects/Debian/apt:pu/http-fixes$ lxc exec bu apt download openjdk-11-jdk-headless Get:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB] Err:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 Undetermined Error [IP: 2001:1af8:4020:b030:deb::185 80] W: Download is performed unsandboxed as root as file '/root/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20200502T085134Z/pool/updates/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb Undetermined Error [IP: 2001:1af8:4020:b030:deb::185 80] jak@jak-t480s:~/Projects/Debian/apt:pu/http-fixes$ lxc exec bu apt download openjdk-11-jdk-headless Get:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB] Fetched 215 MB in 20s (10.9 MB/s) W: Download is performed unsandboxed as root as file '/root/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) Despite having pinned the IP in /etc/hosts.
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
On Wed, Aug 05, 2020 at 12:12:18AM +0100, James Addison wrote: > $ cmdline/apt -o Acquire::Check-Valid-Until=false update > $ cmdline/apt install -y openjdk-11-jdk While you are using the built 'apt' in this way, apt is still talking to the methods (including http) installed on your system, not the ones you just built (and patched), so it would have been rather mysterious if it had worked. ☺ Try: $ cmdline/apt -o Acquire::Check-Valid-Until=false update -o dir::bin::methods=$(pwd)/methods $ cmdline/apt install -y openjdk-11-jdk -o dir::bin::methods=$(pwd)/methods (There are other things apt will still use the installed version of, including config files and such, but that isn't important now) Might be worthwhile to try the whole battery of the http changes Julian did instead of just the single one as they interact with each other even if they are somewhat independent. Best regards David Kalnischkies signature.asc Description: PGP signature
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: apt,snapshot.debian.org Followup-For: Bug #959518 X-Debbugs-Cc: j...@jp-hosting.net FWIW, the series of steps in use for repro are: # prereq: install apt compile-time dependencies # prereq: add apt deb sources, keys as per docs at http://snapshot.debian.org/ $ git clone "https://salsa.debian.org/apt-team/apt.git; $ cd apt $ git checkout 1.8.2.z $ git cherry-pick -m 1 7d222636954ec95382149e31b314e9828ba05a2e $ cmake -DCMAKE_INSTALL_PREFIX=/ . $ make $ cmdline/apt -o Acquire::Check-Valid-Until=false update $ cmdline/apt install -y openjdk-11-jdk And the error manifests as: Get:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 [215 MB] Err:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 Undetermined Error [IP: 193.62.202.27 80] If time allows it might be possible to prepare a self-contained Dockerfile with the repro steps if that'd be useful for further debug/investigation.
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: apt,snapshot.debian.org Followup-For: Bug #959518 X-Debbugs-Cc: j...@jp-hosting.net Thanks for the update! - the issue *does* remain reproducible at the moment, and unfortunately cherry-picking the fix (merge commit 7d22263) into the 1.8.2.z branch and building a version of apt from there doesn't seem to resolve it.
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
On Sun, May 03, 2020 at 09:49:42AM +, James Addison wrote:
> Package: apt
> Version: 1.8.2
> Severity: normal
>
> Dear Maintainer,
>
> During installation of packages under Debian Buster[1], I've encountered a
> repeatable (non-TLS) HTTP download error that occurs during download of
> openjdk-11-jdk-headless.
>
> It's possible this may expose a rare edge case in apt's HTTP client, and/or
> it may be a transient network condition; either way it seems worth reporting.
There are known bugs in the pipelining code that cause issues in an insignifcant
amount of cases, reproducing them is next to impossible. There's some kind of
race somewhere between the server traffic and the client.
I had a reproducer once for such an issue, but sadly, it did not last.
If you can still reproduce this, could you give the current master branch a try?
I went looking around for what *seems* wrong and fixed that, but uh, it might
not
help at all.
and/or try the attached patch to get the location where it failed.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;>
http://www.w3.org/1999/xhtml; xml:lang="en" lang="en">
debian Pastezone
debian Pastezone
Posting 1158950 from None posted at 2020-08-04 09:51:39 expires:
2020-08-11 09:51:39
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72diff --git a/methods/basehttp.cc b/methods/basehttp.cc
index e659da255..76bd69afc 100644
--- a/methods/basehttp.cc
+++ b/methods/basehttp.cc
@@ -35,6 +35,10 @@
/*}}}*/
using namespace std;
+#define S1(x) #x
+#define S2(x) S1(x)
+#define LOCATION Failure in __FILE__ : S2(__LINE__)
+
string BaseHttpMethod::FailFile;
int BaseHttpMethod::FailFd = -1;
time_t BaseHttpMethod::FailTime = 0;
@@ -609,11 +613,11 @@ int BaseHttpMethod::Loop()
switch (Server-Open())
{
case ResultState::FATAL_ERROR:
- Fail(false);
+ Fail(LOCATION, false);
Server = nullptr;
continue;
case ResultState::TRANSIENT_ERROR:
- Fail(true);
+ Fail(LOCATION, true);
Server = nullptr;
continue;
case ResultState::SUCCESSFUL:
@@ -790,11 +794,11 @@ int BaseHttpMethod::Loop()
switch (Result)
{
case ResultState::TRANSIENT_ERROR:
- Fail(true);
+ Fail(LOCATION, true);
break;
case ResultState::FATAL_ERROR:
case ResultState::SUCCESSFUL:
- Fail(false);
+ Fail(LOCATION, false);
break;
}
}
@@ -812,14 +816,14 @@ int BaseHttpMethod::Loop()
// Hard server error, not found or something
case ERROR_UNRECOVERABLE:
{
- Fail();
+ Fail(LOCATION);
break;
}
// Hard internal error, kill the connection and fail
case ERROR_NOT_FROM_SERVER:
{
- Fail();
+ Fail(LOCATION);
RotateDNS();
Server-Close();
break;
@@ -839,9 +843,9 @@ int BaseHttpMethod::Loop()
599, // Network Connect Timeout Error
};
if (std::find(std::begin(TransientCodes), std::end(TransientCodes), Req.Result) != std::end(TransientCodes))
- Fail(true);
+ Fail(LOCATION, true);
else
- Fail();
+ Fail(LOCATION);
break;
}
Show as text |
Download |
Without linenumbers |
Paste new |
Paste as new
Recent Posts
Make new paste
CntDracula
13 minute(s) ago
student
15 minute(s) ago
Wave
17 minute(s) ago
None
22 minute(s) ago
student
28 minute(s) ago
student
35 minute(s) ago
Buxy
40 minute(s) ago
hwgasdf
41 minute(s) ago
hwgasdf
45 minute(s) ago
student
49 minute(s) ago
Links
Add short URL
Debian
About
XML-RPC Interface
Clients
Admin
Alexander Wirt
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: snapshot.debian.org Followup-For: Bug #959518 Dear Maintainer, I've reassigned this issue to both 'apt' and 'snapshot.debian.org' since it seems possible that either an HTTP client edge case and/or a server-side connection-handling behaviour may be the cause(s). It also seems possible that the rate-limiting mentioned previously is being applied by the snapshot.debian.org server; that would be sensible given that my client has retried download of packages repeatedly. I'm not certain whether my bug tracker 'reassign' operation notified the maintainers of the 'snapshot.debian.org' pseudo-package, so this update also serves to (hopefully) ensure they're included on the thread. Thanks again, James -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 5.5.0-2-amd64 (SMP w/2 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8) Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Package: apt
Version: 1.8.2
Severity: normal
Dear Maintainer,
During installation of packages under Debian Buster[1], I've encountered a
repeatable (non-TLS) HTTP download error that occurs during download of
openjdk-11-jdk-headless.
It's possible this may expose a rare edge case in apt's HTTP client, and/or
it may be a transient network condition; either way it seems worth reporting.
The steps to reproduce the problem are:
1. Install GPG
apt-get update && apt-get install -y gpg
2. Approve apt GPG keys for snapshot.debian.org
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys
5E61B217265DA9807A23C5FF4DFAB270CAA96DFA
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys
6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys
80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE
gpg -a --export 5E61B217265DA9807A23C5FF4DFAB270CAA96DFA | apt-key add -
gpg -a --export 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517 | apt-key add -
gpg -a --export 80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE | apt-key add -
NB: gpg will fail to import, with a 'new key but contains no user ID - skipped'
message if the '--keyserver' option is omitted.
3. Configure retrieval of snapshot packages
export SNAPSHOT="20200502T085134Z"
rm /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/ buster
main\n" >> /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian-security/${SNAPSHOT}/
buster/updates main\n" >> /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/
buster-updates main\n" >> /etc/apt/sources.list
4. Attempt installation of openjdk-11-jdk
apt-get update && apt-get install -y openjdk-11-jdk
Expected Results:
Successful installation of packages.
Actual Results:
The following output appears during processing of the command:
Get:185 http://snapshot.debian.org/archive/debian/20200502T085134Z
buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB]
Get:186 http://snapshot.debian.org/archive/debian-security/20200502T085134Z
buster/updates/main amd64 openjdk-11-jre amd64 11.0.7+10-3~deb10u1 [34.3 kB]
Get:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z
buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
[215 MB]
Err:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z
buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
Get:188 http://snapshot.debian.org/archive/debian-security/20200502T085134Z
buster/updates/main amd64 openjdk-11-jdk amd64 11.0.7+10-3~deb10u1 [2607 kB]
Get:189 http://snapshot.debian.org/archive/debian/20200502T085134Z
buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
Get:190 http://snapshot.debian.org/archive/debian/20200502T085134Z
buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB]
Anecdotally, after retrying this process a number of times yesterday, my ISP
appeared to introduce some rate limiting (unfortunately I didn't record any
statistics).
During the slower/limited network condition, I was not able to
reproduce the problem.
Restrictions appear to have been lifted today and the problem reliably
appears each time I attempt the installation.
Related Issues:
I have tried applying a workaround for a similar-sounding issue[2] by
adding '-o Acquire::http::Pipeline-Depth="0"' to the apt-get install
command, but this does not resolve the issue.
Thank you,
James
[1] -
https://hub.docker.com/layers/debian/library/debian/buster-slim/images/sha256-ed8df275d4736ad00e69c4f58c2d9aa3f917911be47753e6d8d25368bcafa38d
[2] - https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1801338
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-modules";
APT::VersionedKernelPackages:: "linux-modules-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "linux-image-unsigned";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
