Re: Unable to get pack file - HTTP 416
Hi List, Am 14.10.20 um 00:34 schrieb Tim Düsterhus: > Am 14.10.20 um 00:31 schrieb Willy Tarreau: >> I wouldn't have thought about such a client-side issue, indeed! >> >> You may be interested in discussing it on the git mailing list: >> >> g...@vger.kernel.org >> >> (not subscription required, just like here). > > Yes, thank you. I'll definitely report this in a spare minute. > For the record: I was able to find a somewhat recent email about this issue in the git list archives and I replied to that thread. My reply can be found here if you'd like to follow this further: https://lore.kernel.org/git/419fd4ad-f9c2-927c-7ae0-c6083f70d...@bastelstu.be/ Best regards Tim Düsterhus
Re: Unable to get pack file - HTTP 416
Willy, Am 14.10.20 um 00:31 schrieb Willy Tarreau: > I wouldn't have thought about such a client-side issue, indeed! > > You may be interested in discussing it on the git mailing list: > > g...@vger.kernel.org > > (not subscription required, just like here). Yes, thank you. I'll definitely report this in a spare minute. > Time to get some sleep for me now, have a nice night :-) But I'll definitely grab some sleep as well first! Best regards Tim Düsterhus
Re: Unable to get pack file - HTTP 416
On Wed, Oct 14, 2020 at 12:24:20AM +0200, Tim Düsterhus wrote: > I assume the following happened: > > 1. git started downloading the packfile into a temporary file [0] > 2. git finished the download (I got all the bytes). > 3. I killed git before it was able to rename the temporary file. > 4. When trying again git tries to resume the download [1]. > 5. Because I already got the whole file git will send an invalid range. > 6. nginx does not like it and sends a 416. > 7. git hard fails instead of retrying without the 'Range' header and > without taking the 'content-range' response into account. > > So the stuff happening in step (7) would IMO be a git bug. I wouldn't have thought about such a client-side issue, indeed! You may be interested in discussing it on the git mailing list: g...@vger.kernel.org (not subscription required, just like here). > I already copied my local repository to test out whether truncating > works. I could fix the issue by truncating off the last byte, causing > git to send a valid range: > > truncate -s 21680736 pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack.temp > > After that worked I did the same on my real repository and called `fsck` > + `gc` to make sure I did not break anything. And this validates your theory above. Good job! > I believe the server side `git gc` would have fixed it as well, because > the pack file in question is completely gone now. It was replaced by a > single large pack :-) Yep indeed! Time to get some sleep for me now, have a nice night :-) Willy
Re: Unable to get pack file - HTTP 416
Willy, Am 14.10.20 um 00:14 schrieb Willy Tarreau: >>> [pid 11946] sendto(9, "GET >>> /git/haproxy.git/objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack >>> HTTP/1.1\r\nHost: git.haproxy.org\r\nRange: bytes=21680737-\r\nUser-Agent: >>> git/2.28.0\r\nAccept: */*\r\n\r\n", 181, MSG_NOSIGNAL, NULL, 0) = 181 >>> [...] >>> [pid 11946] recvfrom(9, "HTTP/1.1 416 Requested Range Not >>> Satisfiable\r\nserver: nginx\r\ndate: Tue, 13 Oct 2020 17:08:56 >>> GMT\r\ncontent-type: text/html\r\ncontent-length: 206\r\ncontent-range: >>> bytes */21680737\r\n\r\n\r\n416 Requested Range Not >>> Satisfiable\r\n\r\n416 >>> Requested Range Not >>> Satisfiable\r\nnginx\r\n\r\n\r\n", >>> 16384, 0, NULL, NULL) = 385 > > Now I understand why, it's exactly the file's size: > > $ ls -l objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > -r--r--r-- 1 christopher haproxy-stable 21680737 Oct 9 17:49 > objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > > However I don't know why it's trying to fetch more of it, which is a good > reason for this to fail! I assume the following happened: 1. git started downloading the packfile into a temporary file [0] 2. git finished the download (I got all the bytes). 3. I killed git before it was able to rename the temporary file. 4. When trying again git tries to resume the download [1]. 5. Because I already got the whole file git will send an invalid range. 6. nginx does not like it and sends a 416. 7. git hard fails instead of retrying without the 'Range' header and without taking the 'content-range' response into account. So the stuff happening in step (7) would IMO be a git bug. [0] https://github.com/git/git/blob/4f0a8be78499454eac3985b6e7e144b8376ab0a5/http.c#L2331 [1] https://github.com/git/git/blob/4f0a8be78499454eac3985b6e7e144b8376ab0a5/http.c#L2350-L2358 > I've just run git fsck and am seeing this: > > $ git fsck > Checking object directories: 100% (256/256), done. > Checking objects: 100% (84716/84716), done. > dangling commit 309200e7332312e4cd7bfe2d9b21d37a6da9a698 > > Christopher faced some permission issues the first time he tried to > publish his version, so I suspect that the issue is that one file > failed to be updated and that a pack is probably bad there. > > I've run a git gc and now git fsck is happy. Please try again! > I already copied my local repository to test out whether truncating works. I could fix the issue by truncating off the last byte, causing git to send a valid range: truncate -s 21680736 pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack.temp After that worked I did the same on my real repository and called `fsck` + `gc` to make sure I did not break anything. I believe the server side `git gc` would have fixed it as well, because the pack file in question is completely gone now. It was replaced by a single large pack :-) Best regards Tim Düsterhus
Re: Unable to get pack file - HTTP 416
On Tue, Oct 13, 2020 at 07:13:12PM +0200, Tim Düsterhus wrote: > Willy, > > Am 13.10.20 um 19:05 schrieb Willy Tarreau: > > Strange. I can get it from here as a whole (not tested with ranges > > though). The pack file is recent, it dates a few days ago only. I'll > > try to investigate further. > > > > I've attached 'strace' to 'git fetch'. It appears that I actually have > the full packfile on disk, because git attempts to the bytes starting at > the last byte of the pack file (21680737). Apparently the nginx does not > like that: > > > [pid 11946] sendto(9, "GET > > /git/haproxy.git/objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > > HTTP/1.1\r\nHost: git.haproxy.org\r\nRange: bytes=21680737-\r\nUser-Agent: > > git/2.28.0\r\nAccept: */*\r\n\r\n", 181, MSG_NOSIGNAL, NULL, 0) = 181 > > [...] > > [pid 11946] recvfrom(9, "HTTP/1.1 416 Requested Range Not > > Satisfiable\r\nserver: nginx\r\ndate: Tue, 13 Oct 2020 17:08:56 > > GMT\r\ncontent-type: text/html\r\ncontent-length: 206\r\ncontent-range: > > bytes */21680737\r\n\r\n\r\n416 Requested Range Not > > Satisfiable\r\n\r\n416 > > Requested Range Not > > Satisfiable\r\nnginx\r\n\r\n\r\n", > > 16384, 0, NULL, NULL) = 385 Now I understand why, it's exactly the file's size: $ ls -l objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack -r--r--r-- 1 christopher haproxy-stable 21680737 Oct 9 17:49 objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack However I don't know why it's trying to fetch more of it, which is a good reason for this to fail! I've just run git fsck and am seeing this: $ git fsck Checking object directories: 100% (256/256), done. Checking objects: 100% (84716/84716), done. dangling commit 309200e7332312e4cd7bfe2d9b21d37a6da9a698 Christopher faced some permission issues the first time he tried to publish his version, so I suspect that the issue is that one file failed to be updated and that a pack is probably bad there. I've run a git gc and now git fsck is happy. Please try again! Thanks! Willy
Re: Unable to get pack file - HTTP 416
Willy, Am 13.10.20 um 19:05 schrieb Willy Tarreau: > Strange. I can get it from here as a whole (not tested with ranges > though). The pack file is recent, it dates a few days ago only. I'll > try to investigate further. > I've attached 'strace' to 'git fetch'. It appears that I actually have the full packfile on disk, because git attempts to the bytes starting at the last byte of the pack file (21680737). Apparently the nginx does not like that: > [pid 11946] sendto(9, "GET > /git/haproxy.git/objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > HTTP/1.1\r\nHost: git.haproxy.org\r\nRange: bytes=21680737-\r\nUser-Agent: > git/2.28.0\r\nAccept: */*\r\n\r\n", 181, MSG_NOSIGNAL, NULL, 0) = 181 > [...] > [pid 11946] recvfrom(9, "HTTP/1.1 416 Requested Range Not > Satisfiable\r\nserver: nginx\r\ndate: Tue, 13 Oct 2020 17:08:56 > GMT\r\ncontent-type: text/html\r\ncontent-length: 206\r\ncontent-range: bytes > */21680737\r\n\r\n\r\n416 Requested Range Not > Satisfiable\r\n\r\n416 > Requested Range Not > Satisfiable\r\nnginx\r\n\r\n\r\n", > 16384, 0, NULL, NULL) = 385 So this is probably a bug in git and I should be able to work around it by truncating the temporary packfile in my repository. I assume a server side workaround would be stripping the `range` request header. Do you happen to know where I would report a git bug? Is it on the git mailing list? Best regards Tim Düsterhus
Re: Unable to get pack file - HTTP 416
Hi Tim, On Tue, Oct 13, 2020 at 06:55:44PM +0200, Tim Düsterhus wrote: > Hi List, > Willy, > > I'm unable to update my local repository: > > > $ g f > > error: Unable to get pack file > > http://git.haproxy.org/git/haproxy.git/objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > > The requested URL returned error: 416 Requested Range Not Satisfiable > > error: Unable to find db9f0524509801b83ec428a8ae2ca535ea3ae264 under > > http://git.haproxy.org/git/haproxy.git > > Cannot obtain needed object db9f0524509801b83ec428a8ae2ca535ea3ae264 > > error: fetch failed. > > This error persists even after retrying the `git fetch` and also after > running a `git fsck` / `git gc`. I'm not sure whether my local copy is > borked (I killed a previous `git fetch` because it hung) or whether > that's a server issue. > > Any ideas? Strange. I can get it from here as a whole (not tested with ranges though). The pack file is recent, it dates a few days ago only. I'll try to investigate further. Thanks! Willy
Unable to get pack file - HTTP 416
Hi List, Willy, I'm unable to update my local repository: > $ g f > error: Unable to get pack file > http://git.haproxy.org/git/haproxy.git/objects/pack/pack-4c6bfad2590afe3d5591d203ad9b3cf2da4aef97.pack > The requested URL returned error: 416 Requested Range Not Satisfiable > error: Unable to find db9f0524509801b83ec428a8ae2ca535ea3ae264 under > http://git.haproxy.org/git/haproxy.git > Cannot obtain needed object db9f0524509801b83ec428a8ae2ca535ea3ae264 > error: fetch failed. This error persists even after retrying the `git fetch` and also after running a `git fsck` / `git gc`. I'm not sure whether my local copy is borked (I killed a previous `git fetch` because it hung) or whether that's a server issue. Any ideas? Best regards Tim Düsterhus
