Re: Compression error? -- Inflate (token) returned -5
On 8/14/2010 3:09 AM, Matt McCutchen wrote: On Fri, 2010-08-13 at 11:26 -0400, Brian K. White wrote: I have sample data that exposes this repeatably: http://lists.samba.org/archive/rsync/2008-October/021889.html Thanks, but we figured out the problem several months ago and it should be fixed in rsync 3.0.7: https://lists.samba.org/archive/rsync/2009-December/024441.html Ahh great... Confirmed. I had a mix of 3.0.6 and 3.0.7 machines 32 and 64bit, but I guess in all that I never did a 3.0.7-3.0.7 test just to be sure. My test data does not fail when both sides of the connection are 3.0.7 Sorry for the noise. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
On Fri, 2010-08-13 at 11:26 -0400, Brian K. White wrote: I have sample data that exposes this repeatably: http://lists.samba.org/archive/rsync/2008-October/021889.html Thanks, but we figured out the problem several months ago and it should be fixed in rsync 3.0.7: https://lists.samba.org/archive/rsync/2009-December/024441.html -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Compression error? -- Inflate (token) returned -5
I have sample data that exposes this repeatably: http://lists.samba.org/archive/rsync/2008-October/021889.html If you place old/file.dat on a remote box and then try to overwrite it with new/file.dat using rsync -az and any other options except --no-z or -W, it will fail almost immediately, every time. If I retry the exact same command but simply remove -z, or add -W, it will work. If I delete file.dat from the remote machine, sending new/file.dat will also work normally with or without -z The error happens before 10 megs of data have been sent, and the files are each just over 1G in size, yet trying the same tests using only the beginning 100 or even 200 megs of the same files (extracted with dd) does not trigger the error, so I couldn't reduce the test data below 2G. At least it's highly compressible data and the tar.bz2 is only 70M Is this useful to anyone for finally having an example to debug and fix up rsync to detect and handle this situation instead of crashing? I probably have currently at least about 4 more examples on various boxes so it happens more often than extremely rarely at least to me. One of those examples I already had to fix and I didn't grab copies of the files before fixing but they were even a little bigger at about 1.2G each. As I get around to fixing the others I'll try to see if I can find a smaller dataset that reproduces the compression/checksum bug. I can give the sample data to pretty much anyone that wants it for a legitimate purpose like this, but it's a customer's data, not especially sensitive or useful, but I still can't publish it completely publicly. I'll have to direct anyone who emails me back directly to a download url. This list is publicly archived so I don't feel good posting it here. I've tarred up the two files and included a txt file with clear simple steps to exibit both the problem with -z and the fact that the same data without -z, or with -W, has no problem. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
On Mon, Oct 06, 2008 at 11:28:31PM +0200, Bas van Schaik wrote: Since this is a production environment I'd like to stick to 3.0.3 (which is already newer than I'd like to use...) in stead of upgrading to 3.0.4, unless have very good reasons to believe it is really fixed in that last version. There has not been any fixing done to the compression code in any recent version. The cause is probably the receiver not applying quite the same compressed data as the sender. I wonder if there was a false checksum match in the file? In such a case, the sender and the receiver would compress different matching data, and could potentially get out of sync. It should be possible to synthesize a file like that and see if it triggers the inflate error. If so, I doubt there's a fix other than to not allow rsync to include matched (unsent) data in the compression stream. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
On Thu, 2008-10-09 at 07:54 -0700, Wayne Davison wrote: I wonder if there was a false checksum match in the file? In such a case, the sender and the receiver would compress different matching data, and could potentially get out of sync. That would explain it. If so, I doubt there's a fix other than to not allow rsync to include matched (unsent) data in the compression stream. That would lose a major advantage of the -z option. Would it work to just abort the file transfer with the ERROR: file failed verification message when compression is found to be out of sync? Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
Paul Slootman wrote: On Fri 05 Sep 2008, Bas van Schaik wrote: Well, why not try upgrading to the latest? If that's not an option for you, what do you want us to do? Retro-actively fix your version by jumping in our time machine? ;-) You're right, I should have been more clear about this. I think there exist about three different threads about this problem, which all ended up in some rsync developer asking for the problematic files and the rsync user admitting that he had already deleted those. Since the cause of the problem is (AFAIK) still unknown, I assumed it wasn't fixed in the last few months. If you have reason to believe it actually _is_ fixed, I'll of course upgrade my rsync installation. But do you have reasons to believe this? I haven't seen any messages to the list about problems with compressed transfers on recent versions, so I have no reason to believe that the problems persist. I never had problems myself, so I can't personally say if it's changed. I've upgraded both client and server to rsync 3.0.3-2 from Ubuntu Intrepid and the problem still exists. Client reports: rsync: writefd_unbuffered failed to write 199 bytes [sender]: Connection reset by peer (104) inflate (token) returned -5 Since this is a production environment I'd like to stick to 3.0.3 (which is already newer than I'd like to use...) in stead of upgrading to 3.0.4, unless have very good reasons to believe it is really fixed in that last version. Regards, Bas -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
On Fri 05 Sep 2008, Bas van Schaik wrote: Well, why not try upgrading to the latest? If that's not an option for you, what do you want us to do? Retro-actively fix your version by jumping in our time machine? ;-) You're right, I should have been more clear about this. I think there exist about three different threads about this problem, which all ended up in some rsync developer asking for the problematic files and the rsync user admitting that he had already deleted those. Since the cause of the problem is (AFAIK) still unknown, I assumed it wasn't fixed in the last few months. If you have reason to believe it actually _is_ fixed, I'll of course upgrade my rsync installation. But do you have reasons to believe this? I haven't seen any messages to the list about problems with compressed transfers on recent versions, so I have no reason to believe that the problems persist. I never had problems myself, so I can't personally say if it's changed. Paul Slootman -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
Hi all, A little less than a year ago I replied to this thread, but I'm still having similar problems. Again, the Inflate (token) returned -5 error occurred while syncing a file: 2008/09/04 22:11:20 [7017] name lookup failed for 172.26.3.224: Name or service not known 2008/09/04 22:11:20 [7017] connect from UNKNOWN (172.26.3.224) 2008/09/04 22:11:20 [7017] rsync to backups/tetra-networking/latest/ from [EMAIL PROTECTED] (172.26.3.224) 2008/09/04 22:11:20 [7017] receiving file list 2008/09/04 22:11:47 [7017] inflate (token) returned -5 2008/09/04 22:11:47 [7017] rsync error: error in rsync protocol data stream (code 12) at token.c(476) [receiver=2.6.9] 2008/09/04 22:11:47 [7017] rsync: connection unexpectedly closed (3301591 bytes received so far) [generator] 2008/09/04 22:11:47 [7017] rsync error: error in rsync protocol data stream (code 12) at io.c(454) [generator=2.6.9] Above you see a snippet of the server log, the client only reports: rsync: writefd_unbuffered failed to write 1 bytes [sender]: Connection reset by peer (104) rsync: connection unexpectedly closed (291575 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(454) [sender=2.6.9] And again, when using triple verbose mode the last match at lines almost matches the filesize of the file to transfer: (...) match at 68541840 last_match=68541840 j=8278 len=8280 n=0 match at 68550120 last_match=68550120 j=8279 len=8280 n=0 match at 68558400 last_match=68558400 j=8280 len=250 n=0 rsync: writefd_unbuffered failed to write 1 bytes [sender]: Connection reset by peer (104) (...) $ ls -la /backups/tetra-networking/latest/10.23.23.1/w/home/connie/Foto's BBQ 23-06-06.zip -rwxr--r-- 9 rsync nogroup 68558650 2008-08-19 23:42 /backups/tetra-networking/latest/10.23.23.1/w/home/connie/Foto's BBQ 23-06-06.zip (note that 68558400 + 250 = 68558650) The file to be transferred is quite small: about 66MB. However, if I try to sync the file on its own (not the whole directory structure) the error does not occur! The tail of the output of the successful rsync: $ rsync -ratzvvv --delete --compress-level=9 --timeout=3600 --password-file=/backups/rsync_wallis.secret client-original.zip [EMAIL PROTECTED]::tmp/ (...) match at 68550120 last_match=68550120 j=8279 len=8280 n=0 match at 68558400 last_match=68558400 j=8280 len=250 n=0 done hash search sending file_sum false_alarms=0 hash_hits=8281 matches=8281 sender finished client-original.zip send_files phase=1 send_files phase=2 send files finished total: matches=8281 hash_hits=8281 false_alarms=0 data=0 sent 104 bytes received 58005 bytes 16602.57 bytes/sec total size is 68558650 speedup is 1179.83 _exit_cleanup(code=0, file=main.c, line=977): about to call exit(0) Clearly it has something to do with the context which the file is in and not the contents of this single file. The only known workaround I know is disabling the compression... FYI: both client and server are using rsync 2.6.9 (from Ubuntu Hardy Heron). Does anyone have more information regarding this issue? Kind regards, -- Bas -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
On Fri 05 Sep 2008, Bas van Schaik wrote: is disabling the compression... FYI: both client and server are using rsync 2.6.9 (from Ubuntu Hardy Heron). Does anyone have more information regarding this issue? Well, why not try upgrading to the latest? If that's not an option for you, what do you want us to do? Retro-actively fix your version by jumping in our time machine? ;-) Paul Slootman -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error? -- Inflate (token) returned -5
Paul Slootman wrote: On Fri 05 Sep 2008, Bas van Schaik wrote: is disabling the compression... FYI: both client and server are using rsync 2.6.9 (from Ubuntu Hardy Heron). Does anyone have more information regarding this issue? Well, why not try upgrading to the latest? If that's not an option for you, what do you want us to do? Retro-actively fix your version by jumping in our time machine? ;-) You're right, I should have been more clear about this. I think there exist about three different threads about this problem, which all ended up in some rsync developer asking for the problematic files and the rsync user admitting that he had already deleted those. Since the cause of the problem is (AFAIK) still unknown, I assumed it wasn't fixed in the last few months. If you have reason to believe it actually _is_ fixed, I'll of course upgrade my rsync installation. But do you have reasons to believe this? -- Bas -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html