Bug#316337: Proof
Roan Kattouw [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: I don't get why you get a bzip2 error at all. The strace shows that apt fails to download Packages.bz2 files (which don't exist) and then it fetches Packages.gz instead. So why is it even trying to bunzip2 the file? It should fork the gunzip method instead. Are we talking about the same strace report? The reports in traces/ are the error-spewing ones and traces/traces2/ shows an apt-get update running fine but creating a bunch of empty partial files. The former series show that each error-spewing bzip2 process tries to read from stdin and gets EOF immediately. Upon further investigation stdin turns out to refer to an empty partial file. Also, I tried bunzip2'ing a .gz file, to which bunzip2 simply responded with This is not a valid .bz2 file, which is quite different from the weird and confusing errors you get when bunzip2'ing an empty file. I think so, yes. Look at what apt does in those traces. It grabs both .bz2 and .gz and still gives the bunzip2 error. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
Goswin von Brederlow wrote: I don't get why you get a bzip2 error at all. The strace shows that apt fails to download Packages.bz2 files (which don't exist) and then it fetches Packages.gz instead. So why is it even trying to bunzip2 the file? It should fork the gunzip method instead. Are we talking about the same strace report? The reports in traces/ are the error-spewing ones and traces/traces2/ shows an apt-get update running fine but creating a bunch of empty partial files. The former series show that each error-spewing bzip2 process tries to read from stdin and gets EOF immediately. Upon further investigation stdin turns out to refer to an empty partial file. Also, I tried bunzip2'ing a .gz file, to which bunzip2 simply responded with This is not a valid .bz2 file, which is quite different from the weird and confusing errors you get when bunzip2'ing an empty file. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
I ran strace -ff on an bzip2-error-spewing apt-get update, and came across one of the following traces (for a bzip2 subprocess): dup2(3, 0) = 0 /* ... */ execve(/usr/bin/bzip2, [bzip2, -d], [/* 28 vars */]) = 0 /* ... */ read(0, , 65536) = 0 read(0, , 65536) = 0 write(2, \nbzip2: Compressed file ends une..., 97) = 97 write(2, bzip2: Function not implemented\n, 32) = 32 write(2, \tInput file = (stdin), output fi..., 46) = 46 write(2, \nIt is possible that the compres..., 240) = 240 exit_group(2) = ? This proves that bzip2 is really fed an empty file. In the traces for one of the methods/ftp subprocesses I found: open(/var/lib/apt/lists/partial/ftp.debian.nl_debian_dists_stable_main_binary-i 386_Packages, O_RDONLY) = 3 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 pipe([4, 5]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil d_tidptr=0xb7cd4708) = 7029 This proves that the empty file bzip2 is fed is really a /var/lib/apt/lists/partial/* file. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
[EMAIL PROTECTED] writes: I ran strace -ff on an bzip2-error-spewing apt-get update, and came across one of the following traces (for a bzip2 subprocess): dup2(3, 0) = 0 /* ... */ execve(/usr/bin/bzip2, [bzip2, -d], [/* 28 vars */]) = 0 /* ... */ read(0, , 65536) = 0 read(0, , 65536) = 0 write(2, \nbzip2: Compressed file ends une..., 97) = 97 write(2, bzip2: Function not implemented\n, 32) = 32 write(2, \tInput file = (stdin), output fi..., 46) = 46 write(2, \nIt is possible that the compres..., 240) = 240 exit_group(2) = ? This proves that bzip2 is really fed an empty file. In the traces for one of the methods/ftp subprocesses I found: open(/var/lib/apt/lists/partial/ftp.debian.nl_debian_dists_stable_main_binary-i 386_Packages, O_RDONLY) = 3 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 pipe([4, 5]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil d_tidptr=0xb7cd4708) = 7029 This proves that the empty file bzip2 is fed is really a /var/lib/apt/lists/partial/* file. I think those are always partial files until they have been verified. The placement might not mean it actualy is partial. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
Roan Kattouw [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: I think those are always partial files until they have been verified. The placement might not mean it actualy is partial. Either way, I see no reason for those files to exist. They're empty and all they do is make apt-get unable to download certain package lists while throwing confusing and misleading error messages in the user's face. Somehow apt-get update should remove all empty files in /var/lib/apt/lists/partial on cleanup, on startup, when it notices they're not needed anymore, or, best of all, not create them in the first place. Now I am not at all familiar with the apt source, and probably won't be able to produce a patch anytime soon, but I do recommend that this bug be rated more severely because it prevents users from getting certain package lists updated, thereby making it impossible for them to download security updates. I think 0 size files are just a case of partial files that isn't handled special. With other partial files you want to keep them and resume downloading. With 0 bytes there is 0 gain in resuming but removing them would haven been extra code. I think the bigger question is where those files come from in the first place. Where does it start fetching them and quits after 0 bytes? Maybe that apt method should not create the files unless it actualy has data to write to them in the first place. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
Goswin von Brederlow wrote: I think those are always partial files until they have been verified. The placement might not mean it actualy is partial. Either way, I see no reason for those files to exist. They're empty and all they do is make apt-get unable to download certain package lists while throwing confusing and misleading error messages in the user's face. Somehow apt-get update should remove all empty files in /var/lib/apt/lists/partial on cleanup, on startup, when it notices they're not needed anymore, or, best of all, not create them in the first place. Now I am not at all familiar with the apt source, and probably won't be able to produce a patch anytime soon, but I do recommend that this bug be rated more severely because it prevents users from getting certain package lists updated, thereby making it impossible for them to download security updates. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
Goswin von Brederlow wrote: I think 0 size files are just a case of partial files that isn't handled special. With other partial files you want to keep them and resume downloading. With 0 bytes there is 0 gain in resuming but removing them would haven been extra code. I think the bigger question is where those files come from in the first place. Where does it start fetching them and quits after 0 bytes? Maybe that apt method should not create the files unless it actualy has data to write to them in the first place. I uploaded my strace reports to a public location: http://members.lycos.nl/kattouw/traces/ contains reports of an apt-get update spewing errors, while http://members.lycos.nl/kattouw/traces/traces2 contains reports of apt-get update running on an empty partial dir. According to the second series, the methods/ftp helper program is responsible for creating those files. After negotiating a lot with the remote FTP server (strace cuts strings at 32 chars, so I didn't see those negotiations. The uploaded reports do show them, but I didn't really feel like reading them at 1 AM) and stat(2)ing a non-existent partial file, it creates one by open(2)ing it with the O_CREAT flag set (i.e. opening it in destructive write mode). These reports may clarify things a little about why those empty partial files are created. The most logical scheme would be to create a partial file only when we're sure we're actually going to receive data to put in it, backup all the data we receive into it, and remove it as soon as we know we've got everything. An alternative would be to remove all empty partial files at cleanup, but that's ugly and still causes trouble when apt-get dies unexpectedly. NB: If you're really curious, the command line used to generate the strace reports was strace -o trace -ff -s 300 apt-get update -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#316337: Proof
Roan Kattouw [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: I think 0 size files are just a case of partial files that isn't handled special. With other partial files you want to keep them and resume downloading. With 0 bytes there is 0 gain in resuming but removing them would haven been extra code. I think the bigger question is where those files come from in the first place. Where does it start fetching them and quits after 0 bytes? Maybe that apt method should not create the files unless it actualy has data to write to them in the first place. I uploaded my strace reports to a public location: http://members.lycos.nl/kattouw/traces/ contains reports of an apt-get update spewing errors, while http://members.lycos.nl/kattouw/traces/traces2 contains reports of apt-get update running on an empty partial dir. According to the second series, the methods/ftp helper program is responsible for creating those files. After negotiating a lot with the remote FTP server (strace cuts strings at 32 chars, so I didn't see those negotiations. The uploaded reports do show them, but I didn't really feel like reading them at 1 AM) and stat(2)ing a non-existent partial file, it creates one by open(2)ing it with the O_CREAT flag set (i.e. opening it in destructive write mode). These reports may clarify things a little about why those empty partial files are created. The most logical scheme would be to create a partial file only when we're sure we're actually going to receive data to put in it, backup all the data we receive into it, and remove it as soon as we know we've got everything. An alternative would be to remove all empty partial files at cleanup, but that's ugly and still causes trouble when apt-get dies unexpectedly. NB: If you're really curious, the command line used to generate the strace reports was strace -o trace -ff -s 300 apt-get update I don't get why you get a bzip2 error at all. The strace shows that apt fails to download Packages.bz2 files (which don't exist) and then it fetches Packages.gz instead. So why is it even trying to bunzip2 the file? It should fork the gunzip method instead. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
