[Bug 1384503] Re: rsync fails on large files with compression
Simply move compression from rsync to ssh: -e "ssh -C" -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Hi, I have just published a docker compose stack to test this issue: https://github.com/NDrive/rsync-compress-bug Actually, I couldn't reproduce the bug with success. Maybe it only happens in some certain files. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I am also suffering from this bug with a 1GB file with the --compress flag in Ubuntu 14.04 with Rsync 3.1.0. The bug is pretty consistent and after I removed the compress flag I got no more broken pipes. I will try to make this bug reproducible with Docker and publish here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Hi Andrew, Installing that version (rsync_3.1.1-3 from the Vivid repo in this case - no dependency problems) on the machine initiating (which is also the target) the rsync didn't fix it. However, installing it on the source machine as well seems to have fixed the problem. Regards, Tom -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Sorry Charles, but a patch that is known to work just in one case isn't really sufficient for an SRU. It is useful and I appreciate your contribution, but alongside it we need to analyse and understand the regression risk for other valid use cases. This is the "Regression Potential" section of the SRU paperwork, together with the SRU Verification process that is documented. I asked for this in comment #8 in the same bug, but nobody responded. This isn't unnecessary or bureaucratic red tape. The process is there to ensure that we do not regress millions of users by trying to fix a bug that doesn't affect them. I'm not willing to upload your patch for SRU review because I don't know what other behaviour it might regress. In order to perform the required analysis, a complete failure case (which in this case needs the correct test data for a failure case) would be most useful. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I attempted to SRU a patch almost a year ago. And that was after two other people confirmed the patch worked for them. In reply to my email Alberto Salvia Novella said on Mon, Jan 12, 2015 at 5:14 PM C Peters: > https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1300367 > Can someone nominate this for a SRU? Done ;) Please see Comment #6 of bug report https://bugs.launchpad.net/rsync/+bug/1300367. The fixed package has been available in my Launchpad PPA since 2014-10-20. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I'd still like to see this fixed, but based on Ryan's comment from when he investigated this I don't think we can do this without a failure case. I presume most reporters won't want to share their data publicly. But we need a failure case in order to make progress. Until we have one, I'm marking this bug as Incomplete, as I think that sets a more accurate expectation. If someone can provide us with a failure case, please do so and we can set the bug back to Triaged. If you don't want to share the failure case publicly but are willing to share it with a Canonical engineer then we can probably arrange something around that. Please get in touch. Alternatively if someone doesn't want to share the failure case at all but is willing to prepare a patch and drive the SRU then that's fine, too. We can believe you if you report success during pre-upload testing and SRU verification and others who are affected can confirm. ** Changed in: rsync (Ubuntu Trusty) Status: Confirmed => Invalid ** Changed in: rsync (Ubuntu Trusty) Status: Invalid => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Excluding all compression options was not an acceptable solution for me. What has worked since I initially found this problem back in August was to install rsync from the utopic repository where the bug was fixed. That same version (3.1.1-3) is still available in vivid and wily unchanged as far as I can tell. Example: wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_amd64.deb sudo dpkg -i rsync_3.1.1-3_amd64.deb Or for x86: wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_i386.deb sudo dpkg -i rsync_3.1.1-3_i386.deb Problem solved. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1384503] Re: rsync fails on large files with compression
Hi I think this is fixed in future rsync versions. For Ubuntu 14.04, exclude ALL compression when rsyncing, e.v. chancve -az to -a. Regards, Jan On 11 January 2016 at 12:56, Tom Worley wrote: > I can confirm I still get this error on Ubuntu 14.04 (all up to date) > rsync: > Both machines: 3.1.0-2ubuntu0.1 > > > File is a .sql.gz file, 1.5 GB in size, as above, --skip-compress=gz > doesn't help: > > . > dbdump.sql.gz > 0 0%0.00kB/s0:00:00 > inflate returned -3 (0 bytes) > rsync error: error in rsync protocol data stream (code 12) at token.c(557) > [receiver=3.1.0] > rsync: connection unexpectedly closed (4092364 bytes received so far) > [generator] > rsync error: error in rsync protocol data stream (code 12) at io.c(226) > [generator=3.1.0] > > -- > You received this bug notification because you are subscribed to a > duplicate bug report (1431709). > https://bugs.launchpad.net/bugs/1384503 > > Title: > rsync fails on large files with compression > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I can confirm I still get this error on Ubuntu 14.04 (all up to date) rsync: Both machines: 3.1.0-2ubuntu0.1 File is a .sql.gz file, 1.5 GB in size, as above, --skip-compress=gz doesn't help: . dbdump.sql.gz 0 0%0.00kB/s0:00:00 inflate returned -3 (0 bytes) rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.0] rsync: connection unexpectedly closed (4092364 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.0] -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
It's an ongoing problem with rsync that's been around for about 10 years or so. Compressed files transferred with -z fail on occasion - though once you find a file that fails, that file will always fail I even use --skip-compress=bz2 but that doesn't work either Last time I got it was ... today. 1.6G -rw-rw-r-- 1 1004 1004 1.6G Dec 14 14:59 log.bz2 Wed Dec 16 07:01:01 AEDT 2015 receiving incremental file list log.bz2 inflate returned -3 (0 bytes) rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.0] rsync: connection unexpectedly closed (163300 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.0] Wed Dec 16 07:02:01 AEDT 2015 Same LTS version as above post rsync: Installed: 3.1.0-2ubuntu0.1 Candidate: 3.1.0-2ubuntu0.1 Version table: *** 3.1.0-2ubuntu0.1 0 500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages 100 /var/lib/dpkg/status 3.1.0-2 0 500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I also have run into this issue on a 3.1.0-2ubuntu0.1 rsync. maglub@XXX:~$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS" Can't do a "apport-collect 1384503", as I don't have a browser available on this system. What other information can I supply to help? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I just found this report myself. I've been experiencing the problem for a while but had to rule out other causes. This is a pretty serious bug in an LTS release. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I've attempted to reproduce this issue but I've not been able to at this point with stock 14.04 rsync versions between two 14.04 (up-to-date) systems. It may be highly dependent on the actual data; if you can reproduce this with something publicly available (like a large iso catted together multiple times or something else that I can create bit- for-bit); that'd help me narrowing down the fix. Could someone who can reproduce this issue please run apport-collect 1384503 so I can get more information from the systems where you can reproduce? W.r.t 3.1.1, we cannot backport the newer package into Trusty, see https://wiki.ubuntu.com/StableReleaseUpdates for the policy and rationale. However, I will take a look at the changes in 3.1.1 to enable the new- style compression as well as the logic to detect when there is a mismatch; If that patchset is small and sane, we may be able to SRU just that change to fix this issue in Trusty. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
** Changed in: rsync (Ubuntu) Assignee: (unassigned) => Ryan Harper (raharper) ** Also affects: rsync (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: rsync (Ubuntu) Status: Confirmed => Fix Released ** Changed in: rsync (Ubuntu Trusty) Status: New => Confirmed ** Changed in: rsync (Ubuntu Trusty) Assignee: (unassigned) => Ryan Harper (raharper) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Just discovered this bug report - after fighting with hanging backup scripts that have worked for years, but broke after upgrading to 14.04. I agree with Joe. Can't 3.1.1-2 be back-ported to 14.04 LTS? This bug has now been open for just shy of 8 months Thanks, Kevin -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
This is a relief, and thanks for giving your attention to this bug. Utopic has 3.1.1-2. Can it be pushed as an update to 14.04? Thanks! --jh-- -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
I installed rsync_3.1.1-2_amd64.deb from the vivid repo. It installed without issue on trusty and, fortunately, resolves this major bug. Gerald -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384503] Re: rsync fails on large files with compression
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: rsync (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1384503 Title: rsync fails on large files with compression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs