[Bug 1384503] Re: rsync fails on large files with compression

[Anakan]

Simply move compression from rsync to ssh:  -e "ssh -C"

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[André Freitas]

I am also suffering from this bug with a 1GB file with the --compress
flag in Ubuntu 14.04 with Rsync 3.1.0. The bug is pretty consistent and
after I removed the compress flag I got no more broken pipes.

I will try to make this bug reproducible with Docker and publish here.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[André Freitas]

Hi, I have just published a docker compose stack to test this issue:
https://github.com/NDrive/rsync-compress-bug

Actually, I couldn't reproduce the bug with success. Maybe it only
happens in some certain files.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Tom Worley]

Hi Andrew,
Installing that version (rsync_3.1.1-3 from the Vivid repo in this case - no 
dependency problems) on the machine initiating (which is also the target) the 
rsync didn't fix it.

However, installing it on the source machine as well seems to have fixed the 
problem.
Regards,
Tom

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Tom Worley]

I can confirm I still get this error on Ubuntu 14.04 (all up to date)
rsync: 
Both machines: 3.1.0-2ubuntu0.1


File is a .sql.gz file, 1.5 GB in size, as above, --skip-compress=gz doesn't 
help:

.
dbdump.sql.gz
  0   0%0.00kB/s0:00:00
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) 
[receiver=3.1.0]
rsync: connection unexpectedly closed (4092364 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.0]

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Robie Basak]

Sorry Charles, but a patch that is known to work just in one case isn't
really sufficient for an SRU. It is useful and I appreciate your
contribution, but alongside it we need to analyse and understand the
regression risk for other valid use cases. This is the "Regression
Potential" section of the SRU paperwork, together with the SRU
Verification process that is documented. I asked for this in comment #8
in the same bug, but nobody responded.

This isn't unnecessary or bureaucratic red tape. The process is there to
ensure that we do not regress millions of users by trying to fix a bug
that doesn't affect them.

I'm not willing to upload your patch for SRU review because I don't know
what other behaviour it might regress. In order to perform the required
analysis, a complete failure case (which in this case needs the correct
test data for a failure case) would be most useful.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

Re: [Bug 1384503] Re: rsync fails on large files with compression

[Jan Groenewald]

Hi

I think this is fixed in future rsync versions. For Ubuntu 14.04, exclude
ALL compression when rsyncing, e.v. chancve -az to -a.

Regards,
Jan

On 11 January 2016 at 12:56, Tom Worley  wrote:

> I can confirm I still get this error on Ubuntu 14.04 (all up to date)
> rsync:
> Both machines: 3.1.0-2ubuntu0.1
>
>
> File is a .sql.gz file, 1.5 GB in size, as above, --skip-compress=gz
> doesn't help:
>
> .
> dbdump.sql.gz
>   0   0%0.00kB/s0:00:00
> inflate returned -3 (0 bytes)
> rsync error: error in rsync protocol data stream (code 12) at token.c(557)
> [receiver=3.1.0]
> rsync: connection unexpectedly closed (4092364 bytes received so far)
> [generator]
> rsync error: error in rsync protocol data stream (code 12) at io.c(226)
> [generator=3.1.0]
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1431709).
> https://bugs.launchpad.net/bugs/1384503
>
> Title:
>   rsync fails on large files with compression
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Andrew]

Excluding all compression options was not an acceptable solution for me.

What has worked since I initially found this problem back in August was
to install rsync from the utopic repository where the bug was fixed.
That same version (3.1.1-3) is still available in vivid and wily
unchanged as far as I can tell.

Example:

wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_amd64.deb
sudo dpkg -i rsync_3.1.1-3_amd64.deb

Or for x86:

wget http://mirrors.kernel.org/ubuntu/pool/main/r/rsync/rsync_3.1.1-3_i386.deb
sudo dpkg -i rsync_3.1.1-3_i386.deb

Problem solved.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Robie Basak]

I'd still like to see this fixed, but based on Ryan's comment from when
he investigated this I don't think we can do this without a failure
case. I presume most reporters won't want to share their data publicly.
But we need a failure case in order to make progress.

Until we have one, I'm marking this bug as Incomplete, as I think that
sets a more accurate expectation. If someone can provide us with a
failure case, please do so and we can set the bug back to Triaged.

If you don't want to share the failure case publicly but are willing to
share it with a Canonical engineer then we can probably arrange
something around that. Please get in touch.

Alternatively if someone doesn't want to share the failure case at all
but is willing to prepare a patch and drive the SRU then that's fine,
too. We can believe you if you report success during pre-upload testing
and SRU verification and others who are affected can confirm.

** Changed in: rsync (Ubuntu Trusty)
   Status: Confirmed => Invalid

** Changed in: rsync (Ubuntu Trusty)
   Status: Invalid => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Charles Peters II]

I attempted to SRU a patch almost a year ago.  And that was after two
other people confirmed the patch worked for them.

In reply to my email Alberto Salvia Novella said on Mon, Jan 12, 2015 at 5:14 PM
C Peters:
> https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1300367
> Can someone nominate this for a SRU?

Done ;)


Please see Comment #6 of bug report 
https://bugs.launchpad.net/rsync/+bug/1300367.  The fixed package has been 
available in my Launchpad PPA since 2014-10-20.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[A]

It's an ongoing problem with rsync that's been around for about 10 years
or so.

Compressed files transferred with -z fail on occasion - though once you find a 
file that fails, that file will always fail
I even use --skip-compress=bz2 but that doesn't work either

Last time I got it was ... today.

1.6G -rw-rw-r-- 1 1004 1004 1.6G Dec 14 14:59 log.bz2

Wed Dec 16 07:01:01 AEDT 2015
receiving incremental file list
log.bz2
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) 
[receiver=3.1.0]
rsync: connection unexpectedly closed (163300 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) 
[generator=3.1.0]
Wed Dec 16 07:02:01 AEDT 2015

Same LTS version as above post

rsync:
  Installed: 3.1.0-2ubuntu0.1
  Candidate: 3.1.0-2ubuntu0.1
  Version table:
 *** 3.1.0-2ubuntu0.1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 
Packages
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 
Packages
100 /var/lib/dpkg/status
 3.1.0-2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Magnus Lubeck]

I also have run into this issue on a 3.1.0-2ubuntu0.1 rsync.

maglub@XXX:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"

Can't do a "apport-collect 1384503", as I don't have a browser available
on this system. What other information can I supply to help?

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Andrew]

I just found this report myself. I've been experiencing the problem for
a while but had to rule out other causes. This is a pretty serious bug
in an LTS release.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Ryan Harper]

I've attempted to reproduce this issue but I've not been able to at this
point with stock 14.04 rsync versions between two 14.04 (up-to-date)
systems.  It may be highly dependent on the actual data; if you can
reproduce this with something publicly available (like a large iso
catted together multiple times or something else that I can create bit-
for-bit); that'd help me narrowing down the fix.

Could someone who can reproduce this issue please run  apport-collect
1384503 so I can get more information from the systems where you can
reproduce?

W.r.t 3.1.1, we cannot backport the newer package into Trusty, see
https://wiki.ubuntu.com/StableReleaseUpdates for the policy and
rationale.

However, I will take a look at the changes in 3.1.1 to enable the new-
style compression as well as the logic to detect when there is a
mismatch;  If that patchset is small and sane, we may be able to SRU
just that change to fix this issue in Trusty.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Robie Basak]

** Changed in: rsync (Ubuntu)
 Assignee: (unassigned) = Ryan Harper (raharper)

** Also affects: rsync (Ubuntu Trusty)
   Importance: Undecided
   Status: New

** Changed in: rsync (Ubuntu)
   Status: Confirmed = Fix Released

** Changed in: rsync (Ubuntu Trusty)
   Status: New = Confirmed

** Changed in: rsync (Ubuntu Trusty)
 Assignee: (unassigned) = Ryan Harper (raharper)

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Kevin_Traas]

Just discovered this bug report - after fighting with hanging backup
scripts that have worked for years, but broke after upgrading to 14.04.
I agree with Joe.  Can't 3.1.1-2 be back-ported to 14.04 LTS?

This bug has now been open for just shy of 8 months

Thanks,
Kevin

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Joe Harrington]

This is a relief, and thanks for giving your attention to this bug.
Utopic has 3.1.1-2. Can it be pushed as an update to 14.04?

Thanks!

--jh--

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to rsync in Ubuntu.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Gerald Villemure]

I installed rsync_3.1.1-2_amd64.deb from the vivid repo.

It installed without issue on trusty and, fortunately, resolves this
major bug.

Gerald

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to rsync in Ubuntu.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1384503] Re: rsync fails on large files with compression

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: rsync (Ubuntu)
   Status: New = Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to rsync in Ubuntu.
https://bugs.launchpad.net/bugs/1384503

Title:
  rsync fails on large files with compression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1384503/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs