On 04/27/2016 06:06 PM, Andy Monkiewicz wrote: > On Tuesday, April 26, 2016 at 4:27:32 PM UTC-4, Andy Monkiewicz wrote: >> I am trying to get backups working with Bareos and have been having a lot of >> trouble with the fd just giving up doing anything with any speed after a >> while. The backup starts smooth at 10-20MB/s (I'm watching with bmon), then >> after an hour or 2, just sits at 35KB/s. >> >> Both machines are new, and only one backup is configured. Both are running >> Ubuntu 14.04 and whatever the newest copy of Bareos is as of last week. They >> are on the same GB switch, and iperf tests show the expected available >> bandwidth as ~950 Mbits/sec. >> >> The director/storage computer is running postgresql as the db backend, and >> if I sit and watch top as the job is running, only occasionally does >> bareos-sd hit 1%, the rest of the time sitting completely idle. Generally >> looking like this: >> >> Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.5 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 >> st >> >> The source computer has folder with the data to be backed up, which is >> ~130GB and ~2M files. It isn't doing anything but being a backup source. >> >> I did notice the source computer with some i/o wait time: >> Tasks: 219 total, 1 running, 217 sleeping, 0 stopped, 1 zombie >> %Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 74.9 id, 25.0 wa, 0.0 hi, 0.0 si, 0.0 >> st >> >> But that doesn't seem to explain why it was trucking along at 10-20MB/sec >> and then just dying after an hour and a half or so. I don't feel like 2 >> million files and 100 gigs is a lot these days? In any case it seems like a >> transfer rate of 30-60KB/s is just horrid performance. >> >> The filesystem is ext3 (and is on an encrypted volume). Is there a better >> file system I should be using that would enhance performance? I have >> attribute spooling on for this job. >> >> And now after sitting at 35K/s for an hour while writing this post, it has >> jumped back up to 10MB/s for 30 seconds and then dropped back to 50KB/s! >> >> Help! > > I have switched the filesystem from ext3 to ext4 mounted with noatime and now > I'm all the way up to 52KB/s. It was running strong: (Files=673,386 > Bytes=73,533,094,047 Bytes/sec=18,187,755) until this point in the backup, > about an hour in, then dropped speed all the way down to 45-50KB/s. >
That's an average file size of about 100 KB, but you may have a few very big files and and large number of very small files. It's normal that speed goes down with a large number of small files. Also the filesystem cache is relevant here. You could measure indepentently from Bareos how much time it takes on your system to read all the files by using tar like this time tar --totals -cf /dev/null /path/to/your/data and observe iostat as you did before. To make sure all is really read from disk, clear the cache before, by using echo 2 > /proc/sys/vm/drop_caches The caching for sure depends on how much RAM you have and how much of it is available for caching. Repeat the tar command without clearing the cache to see the effect. Normally, nowadays relatime is used by default, which only modifies atime on changes. So probably noatime won't make much difference. If you don't use Data Spooling (see http://doc.bareos.org/master/html/bareos-manual-main-reference.html#DataSpooling), make sure to use Attribute Spooling by setting Spool Attributes = yes See http://doc.bareos.org/master/html/bareos-manual-main-reference.html#directiveDirJobSpool%20Attributes Otherwise data is written into the Catalog DB while the backup is running, this may slow down the backup process. When using attribute spooling, all Catalog DB writes will happen batched at the end of the job. Regards -- Stephan Dühr [email protected] Bareos GmbH & Co. KG Phone: +49 221-630693-90 http://www.bareos.com Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646 Komplementär: Bareos Verwaltungs-GmbH Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz, M. v. Wieringen -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. For more options, visit https://groups.google.com/d/optout.
