Re: [BackupPC-users] Slow local backup
Hi there Friday, June 15, 2018, 3:06:30 PM, you wrote: BB> It finally finished after 24 hours. That gives about 13G/hour or about BB> 3.8M/s. BB> The CPUs were not busy. That's what I was confused about. I would have BB> expected to see a bottleneck at some point, but nothing seemed to be BB> busy. The CPUs were all at or below 20% and iowait was close to 0 most BB> of the time. I'm not sure how I would determine if the loopback was BB> saturated. what file system are you using? I am operating a mailserver on a dedicated machine. There are some millions of small files on that machine, so the situation is comparable to yours. When I set up that machine some years ago, I compared various file systems. Surprisingly, ext4 came out as winner to handle these small files. I am just using ext4 on top of the hardware, no lvm on that machine. And of course having a lot of RAM may also speed up your machine, because the server can read ahead data. I think it may be a matter of tuning that server to get better performance. https://www.kernel.org/doc/Documentation/filesystems/ext4.txt may give some hints on read ahead tuning. best regards --- Michael Schumacher -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/18/2018 10:55 AM, Les Mikesell wrote: > On Mon, Jun 18, 2018 at 9:38 AM, Bowie Bailey wrote: >> I have another big local backup that's been running all weekend. I am >> trying sudo/rsync rather than rsyncd or ssh/rsync this time. This >> directory has millions of small files (and is even bigger than the first >> one), so I expect it to take longer. >> >> This time, I'm seeing higher IO-wait numbers: > Moving the disk head around is almost always the slowest operation and > on the first full it has to traverse every file. And backups tend to > not fit the normal OS caching mechanisms that try to hide that > slowness. Right. I only pointed it out since I did not see this on the previous backup that ran for 24 hours. I'm still not sure where the bottleneck was there since I was never able to see anything that looked busy during the process. -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On Mon, Jun 18, 2018 at 9:38 AM, Bowie Bailey wrote: > > I have another big local backup that's been running all weekend. I am > trying sudo/rsync rather than rsyncd or ssh/rsync this time. This > directory has millions of small files (and is even bigger than the first > one), so I expect it to take longer. > > This time, I'm seeing higher IO-wait numbers: Moving the disk head around is almost always the slowest operation and on the first full it has to traverse every file. And backups tend to not fit the normal OS caching mechanisms that try to hide that slowness. -- Les Mikesell lesmikes...@gmail.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/16/2018 9:34 PM, ED Fochler wrote: >> On 2018, Jun 15, at 9:06 AM, Bowie Bailey wrote: >> >> The CPUs were not busy. That's what I was confused about. I would have >> expected to see a bottleneck at some point, but nothing seemed to be >> busy. The CPUs were all at or below 20% and iowait was close to 0 most >> of the time. I'm not sure how I would determine if the loopback was >> saturated. > 20% loaded with 6 or more cores would be a single thread running full speed > on one core. A single task may not stay on one core, it may get moved around > enough that it doesn't appear that any of the cores are heavily loaded. > Doubly true if you have hyperthreading which causes additional CPU shuffling. > > I think you ran into single thread compression performance, plus any of the > normal process, network, and IO latency that would compound with everything > being on one machine. That sounds about right to me. First backups are > slow. SSH probably wouldn't have made any difference at that speed. No, only 2 cores on this machine. I have another big local backup that's been running all weekend. I am trying sudo/rsync rather than rsyncd or ssh/rsync this time. This directory has millions of small files (and is even bigger than the first one), so I expect it to take longer. This time, I'm seeing higher IO-wait numbers: %Cpu0 : 0.0 us, 1.7 sy, 0.0 ni, 5.7 id, 92.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.3 us, 0.7 sy, 0.0 ni, 86.3 id, 12.4 wa, 0.0 hi, 0.3 si, 0.0 st But I don't see high cpu usage. The IO-wait seems to come and go. It alternates with mostly idle periods: %Cpu0 : 3.0 us, 0.7 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 5.0 us, 2.0 sy, 0.0 ni, 92.3 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st Neither cpu seems to go higher than 6%. Once this one finishes, I'll have a first round of full backups for all of the large hosts. Hopefully, the backup times will be a bit more reasonable after that. -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] backup random hang with rsync/rsyncd
Hi I'm using BackupPC 4.1.3 on a CentOS server to back up a bunch of other Linux servers. In total I have 40 hosts that are backed up daily. All hosts are backed up with rsyncd, almost identical configs. I have one server with a lot of files (around 7mil), on multiple partitions. It's a cPanel server on CentOS 6, hosting a bunch of domains, email, sites. This one server has a backup problem. I do a full backup of it, the first backup, which finishes every time. After that, I have incremental backups which randomly hang, and never finish. The rsync processes are running, but are doing nothing at all. I've tried with rsyncd AND rsync+ssh, I've tried upgrading rsync to the latest on both sides (on the client side rsync, on the server side rsync_bpc), it didn't help. There are 5 partitions on this server, and I'm backing them up one by one (by specifying multiple shares on BackupPC). The hangs occur randomly on any of the partitions. I tried setting --timeout= to a low value, but it doesn't time out, it stays there for days, and when I kill rsync on the client side, then the backuppc process moves on, and finishes with an error. This is a screenshot of the running job, that's been running for 2 days... All other servers were backup up twice in this time, but this one hangs. There is no firewall between the two (there is but the IP addresses are explicitly allowed). [cid:part1.C128905D.EEF6AE84@upc.ro] I don't know what to do about this... Please give me some hints. I repeat: the other 39 hosts are working just fine (they are CentOS 6, CentOS 7, Debian, Ubuntu hosts, various versions). They are on the same network, in the same location as this one server. -- Imre Gergely Telephony & ISP Services Engineer UPC Romania -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
