Bug#795284: troubles with range requests
Control: severity 795284 important On Tue, 25 Aug 2015 15:04:29 +0100 Mark Hindley wrote: > package apt-cacher > reassign 795284 apt > thanks > > I have looked at this some more and I think apt-cacher is following the spec > correctly. It seems as if apt-get doesn't realise that the 416 response with > the > Content-Range denominator equal to the size it already has indicates the file > is > complete. > > So I am going to reassign to apt. > > Apt team, if I have got this analysis wrong, sorry, do point out my error and > assign > back. Dear apt team, I came recently to erratical behavior of apt with apt-cacher-ng (not apt-cacher but similar). Checking the wireshark trace, I see strange things, see below. For me, what apt's http client is doing there DOES NOT MAKE MUCH SENSE. The ancient version, like 10 years ago, used a very simple and effective trick. It specified If-Range but also a Range which started one byte _before_ the previous body length. So the server could either return just a byte or start returning a completely new resource body with a 200 response, no extra rounds needed. What does the current version do? It requests the file but the with If-Range (which is okay) and Range which starts AFTER the body. This obviously causes a code 416 even if the response body hasn't changed. In response, apt makes another request, with "If-Modified-Since" from a day ago - so I guess it has another version in cache, which it takes as replacement now. It also does not specify the Range. So I guess the issue 795284 is mostly solved but not fully, we got another pointless behavior now. So, the current version behaves better than in 2015, but still worse than ~10 years ago. Please restore the original algorithm, which refetched the last byte. This causes less bandwith waste than doing a second request which will almost always become necessary. (local trace, input/output interleaved but you see the point). GET http://ftp.de.debian.org/debian/dists/testing/InRelease HTTP/1.1 Host: ftp.de.debian.org Cache-Control: max-age=0 Accept: text/* Range: bytes=128490- If-Range: Fri, 08 Oct 2021 14:16:00 GMT User-Agent: Debian APT-HTTP/1.3 (2.3.9) HTTP/1.1 416 Requested Range Not Satisfiable Content-Length: 512 Content-Type: text/html Date: Fri, 08 Oct 2021 18:26:45 GMT Server: Debian Apt-Cacher NG/3.7.3 ... href="http://www.unix-ag.uni-kl.de/~bloch/acng/";>Apt-Cacher NG homepage GET http://ftp.de.debian.org/debian/dists/testing/InRelease HTTP/1.1 Host: ftp.de.debian.org Cache-Control: max-age=0 Accept: text/* If-Modified-Since: Thu, 07 Oct 2021 14:15:24 GMT User-Agent: Debian APT-HTTP/1.3 (2.3.9) HTTP/1.1 200 OK Content-Type: octet/stream Last-Modified: Fri, 08 Oct 2021 14:16:00 GMT Content-Length: 128490 X-Original-Source: http://deb.debian.org/debian/dists/testing/InRelease Date: Fri, 08 Oct 2021 18:26:45 GMT Server: Debian Apt-Cacher NG/3.7.3 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Origin: Debian Label: Debian ... Best regards, Eduard.
Bug#795284: troubles with range requests
package apt-cacher reassign 795284 apt thanks I have looked at this some more and I think apt-cacher is following the spec correctly. It seems as if apt-get doesn't realise that the 416 response with the Content-Range denominator equal to the size it already has indicates the file is complete. So I am going to reassign to apt. Apt team, if I have got this analysis wrong, sorry, do point out my error and assign back. Thanks. Mark
Bug#795284: troubles with range requests
On Aug 13, Mark Hindley wrote: > I see there is a problem, but I think apt-cacher is following what the > spec says. Or am I misreading it? Yes, from what I can see apt-cacher is behaving correctly here. -- ciao, Marco pgpEe0_F_0EQ4.pgp Description: PGP signature
Bug#795284: troubles with range requests
On Thu, Aug 13, 2015 at 01:12:13PM +0200, Marco d'Itri wrote: > On Aug 13, Mark Hindley wrote: > > > Thanks, as far as I can see on a first look, apt-cacher is doing the > > right thing here. > I tend to agree. Would you be more specific? I see there is a problem, but I think apt-cacher is following what the spec says. Or am I misreading it? Please explain... Thanks. Mark
Bug#795284: troubles with range requests
On Aug 13, Mark Hindley wrote: > Thanks, as far as I can see on a first look, apt-cacher is doing the > right thing here. I tend to agree. -- ciao, Marco pgpVP2Krfcpl1.pgp Description: PGP signature
Bug#795284: troubles with range requests
On Wed, Aug 12, 2015 at 06:00:17PM +0200, Marco d'Itri wrote: > Package: apt-cacher > Version: 1.7.11 > Severity: normal > > Client: jessie, just installed. > > I *randomly* (i.e. from 2 to 5 files at every apt-get update run) get > errors caused by apt issuing a range request for a file that it has > already received fully. > I am not sure about who is at fault between apt and apt-cacher: if > I issue the same range request to a normal mirror running Apache I get > back the same error. Thanks, as far as I can see on a first look, apt-cacher is doing the right thing here. > Date: Wed, 12 Aug 2015 15:49:13 GMT > Via: 1.1 pxe.mil.seeweb.it:80 (apt-cacher/1.7.11) > Accept-Ranges: bytes > Age: 360 > ETag: "11a597-e59-51d18daedee40" > Server: Apache > Content-Length: 3673 > Content-Type: application/x-bzip2 > Last-Modified: Wed, 12 Aug 2015 08:17:53 GMT > > > Debug output: > > GET /debian/dists/jessie-updates/main/binary-amd64/Packages.bz2 HTTP/1.1 > Host: debian.seeweb.it > Cache-Control: max-age=0 > Range: bytes=3673- > If-Range: Wed, 12 Aug 2015 08:17:53 GMT > User-Agent: Debian APT-HTTP/1.3 (1.0.9.8) Request says if not modified since 12 Aug send bytes from 3673 > Answer for: > http://debian.seeweb.it/debian/dists/jessie-updates/main/binary-amd64/Packages.bz2 > HTTP/1.1 416 Invalid range: 3673- > Connection: Keep-Alive > Date: Wed, 12 Aug 2015 15:23:01 GMT > Accept-Ranges: bytes > Server: apt-cacher/1.7.11 > Content-Length: 0 > Content-Range: bytes */3673 apt-cacher replies no bytes beyond 3673 Or am I missing something? Rereading RFC 7233 the relevant bit is this: The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the request's Range header field (Section 3.1) overlap the current extent of the selected resource or that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges. For byte ranges, failing to overlap the current extent means that the first-byte-pos of all of the byte-range-spec values were greater than the current length of the selected representation. When this status code is generated in response to a byte-range request, the sender SHOULD generate a Content-Range header field specifying the current length of the selected representation (Section 4.2). For example: HTTP/1.1 416 Range Not Satisfiable Date: Fri, 20 Jan 2012 15:41:54 GMT Content-Range: bytes */47022 Note: Because servers are free to ignore Range, many implementations will simply respond with the entire selected representation in a 200 (OK) response. That is partly because most clients are prepared to receive a 200 (OK) to complete the task (albeit less efficiently) and partly because clients might not stop making an invalid partial request until they have received a complete representation. Thus, clients cannot depend on receiving a 416 (Range Not Satisfiable) response even when it is most appropriate. I wonder if apt-get is not correctly handling the 416 response and doesn't realise that it means the file it has is complete as it stands? Mark
Bug#795284: troubles with range requests
Package: apt-cacher Version: 1.7.11 Severity: normal Client: jessie, just installed. I *randomly* (i.e. from 2 to 5 files at every apt-get update run) get errors caused by apt issuing a range request for a file that it has already received fully. I am not sure about who is at fault between apt and apt-cacher: if I issue the same range request to a normal mirror running Apache I get back the same error. My workaround: I disabled support for range requests in apt-cacher. What is ok the disk: -rw-r--r-- 1 www-data www-data 3673 Aug 12 16:43 packages/debian_dists_jessie-updates_main_binary-amd64_Packages.bz2 How apt-cacher answers a normal HEAD request: Date: Wed, 12 Aug 2015 15:49:13 GMT Via: 1.1 pxe.mil.seeweb.it:80 (apt-cacher/1.7.11) Accept-Ranges: bytes Age: 360 ETag: "11a597-e59-51d18daedee40" Server: Apache Content-Length: 3673 Content-Type: application/x-bzip2 Last-Modified: Wed, 12 Aug 2015 08:17:53 GMT Debug output: GET /debian/dists/jessie-updates/main/binary-amd64/Packages.bz2 HTTP/1.1 Host: debian.seeweb.it Cache-Control: max-age=0 Range: bytes=3673- If-Range: Wed, 12 Aug 2015 08:17:53 GMT User-Agent: Debian APT-HTTP/1.3 (1.0.9.8) Answer for: http://debian.seeweb.it/debian/dists/jessie-updates/main/binary-amd64/Packages.bz2 HTTP/1.1 416 Invalid range: 3673- Connection: Keep-Alive Date: Wed, 12 Aug 2015 15:23:01 GMT Accept-Ranges: bytes Server: apt-cacher/1.7.11 Content-Length: 0 Content-Range: bytes */3673 -- ciao, Marco pgpYpoHBkbnef.pgp Description: PGP signature
