Bug#971627: 100% cpu on resume of transfer
On Sun, Oct 04, 2020 at 02:55:11PM +0200, Paul Slootman wrote: > On Sat 03 Oct 2020, Kurt Roeckx wrote: > > On Sat, Oct 03, 2020 at 02:08:54PM +0200, Paul Slootman wrote: > > > On Sat 03 Oct 2020, Kurt Roeckx wrote: > > > > > > > > I was transfering a large file using rsync (3 TB). The connection > > > > broke after about 1 TB. I was using the -P option. So I restarted > > > > the transfer. But that transfer resulted in 100% CPU usage on the > > > > sender side, and limiting the transfer to about 1.5 MB/s. > > > > > > > > It also seems that when I restart the transfer, it first reads the > > > > 1 TB on the client side, and then reads the 1 TB on the sender > > > > side, causing a large delay before restarting the transfer. > > > > > > This is normal, as rsync first has to verify that the 1TB on the > > > destination is correct and identical to the data on the source. > > > > > > If you are certain that the part already transmitted to the destination > > > is correct, you can use --append to only send the remaining part of the > > > file. > > > > Is the 100% CPU usage also expected? > > Yes, that's the checksum algorithm running. So you're saying it can only checksum at 1.5 MB/s? Kurt
Bug#971627: 100% cpu on resume of transfer
On Sat 03 Oct 2020, Kurt Roeckx wrote: > On Sat, Oct 03, 2020 at 02:08:54PM +0200, Paul Slootman wrote: > > On Sat 03 Oct 2020, Kurt Roeckx wrote: > > > > > > I was transfering a large file using rsync (3 TB). The connection > > > broke after about 1 TB. I was using the -P option. So I restarted > > > the transfer. But that transfer resulted in 100% CPU usage on the > > > sender side, and limiting the transfer to about 1.5 MB/s. > > > > > > It also seems that when I restart the transfer, it first reads the > > > 1 TB on the client side, and then reads the 1 TB on the sender > > > side, causing a large delay before restarting the transfer. > > > > This is normal, as rsync first has to verify that the 1TB on the > > destination is correct and identical to the data on the source. > > > > If you are certain that the part already transmitted to the destination > > is correct, you can use --append to only send the remaining part of the > > file. > > Is the 100% CPU usage also expected? Yes, that's the checksum algorithm running. Paul
Bug#971627: 100% cpu on resume of transfer
On Sat, Oct 03, 2020 at 02:08:54PM +0200, Paul Slootman wrote: > On Sat 03 Oct 2020, Kurt Roeckx wrote: > > > > I was transfering a large file using rsync (3 TB). The connection > > broke after about 1 TB. I was using the -P option. So I restarted > > the transfer. But that transfer resulted in 100% CPU usage on the > > sender side, and limiting the transfer to about 1.5 MB/s. > > > > It also seems that when I restart the transfer, it first reads the > > 1 TB on the client side, and then reads the 1 TB on the sender > > side, causing a large delay before restarting the transfer. > > This is normal, as rsync first has to verify that the 1TB on the > destination is correct and identical to the data on the source. > > If you are certain that the part already transmitted to the destination > is correct, you can use --append to only send the remaining part of the > file. Is the 100% CPU usage also expected? As one debian mirror admins, I also run an rsync server. I often see rsync using 100% cpu, but I never knew why. I now assume that the transfer was broken when they were mirroring the .iso image from me. Kurt
Bug#971627: 100% cpu on resume of transfer
Package: rsync Version: 3.1.3-6 Hi, I was transfering a large file using rsync (3 TB). The connection broke after about 1 TB. I was using the -P option. So I restarted the transfer. But that transfer resulted in 100% CPU usage on the sender side, and limiting the transfer to about 1.5 MB/s. It also seems that when I restart the transfer, it first reads the 1 TB on the client side, and then reads the 1 TB on the sender side, causing a large delay before restarting the transfer. Both sides have the same version installed. Kurt