Re: [BackupPC-users] BackupPC writes to disk very slow
Ari, I have been using BackupPC successfully with ZFS On Linux for a few years now. This was simply convenient as I had a lot of old hardware with 3TB drives that I was able to repurpose for the task. We have about a dozen hosts with either ~60TB or ~160TB usable in Raid Z1+spare. Despite having no issues I would probably redo them as raidz2 with no hot spare. I use config management to set the number of concurrent jobs per hosts based on the number of cores on the systems. The data being backed up ranges from many small files to few very large files. The only tweaking I've done is configure so that we do many incrementals but 1 full every 90 days (just to reduce traffic to primary storage). I also disable compression in backuppc letting the file system handle that. Other than the manual act of rebuilding the zfs modules between kernel upgrades it works very well! Matt -Original Message- From: Ari Sovijärvi Sent: Sunday, December 2, 2018 10:46 AM To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] BackupPC writes to disk very slow On 2.12.2018 15.32, Tapio Lehtonen wrote: > On new BackupPC host backups go very slow. I believe I have determined > the network connection is not at fault, from Windows machine > Speedtest.net shows a little less than gigabit speeds and from this > backuppc host speedtest.cli shows over 800 Mbits / sek. Out of curiosity, have you experimented with other filesystems? I have couple relatively large setups (pool at ~ 9 terabytes) with ext4 and those still crunch backups happily. XFS has been a bit hit and miss for me, I know many swear by it, but where I've tested it, I've hit all kinds of random problems to a degree that I haven't bothered with it any more. I've been recently experimenting with separating the pc and pool directories so pc directories are on SSD storage and the pool on HDDs. Jury is still out on real life speedups on this, but it seems possible with BPC4. -- Ari Sovijärvi ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC writes to disk very slow
> -Original Message- > From: Ari Sovijärvi > Sent: den 2 december 2018 16:46 > To: backuppc-users@lists.sourceforge.net > Subject: Re: [BackupPC-users] BackupPC writes to disk very slow > > XFS has been a bit hit and miss for me, I know many swear by it, but > where I've tested it, I've hit all kinds of random problems to a degree > that I haven't bothered with it any more. Thank you. I've seen similar problems too and I also don't bother with XFS anymore on my BPC-installs. -- //Sorin smime.p7s Description: S/MIME cryptographic signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC writes to disk very slow
On 2018-12-03 04:22, Tapio Lehtonen wrote: Out of curiosity, have you experimented with other filesystems? I have couple relatively large setups (pool at ~ 9 terabytes) with ext4 and those still crunch backups happily. Previously I have used EXT4, only problem was once running out of inodes. But I find it hard to believe XFS with default options would be so much worse than any other file system with default options. I performed a number of benchmarks, specifically with BackupPC and a handful of file systems. My informal results were that XFS was slightly faster *unless* there was subtle corruption in the filesystem, in which case, the only remedy seemed to be a complete rebuild of XFS. (The fsck tools were capable of detecting it, but not repairing it.) I did not pursue this further, but my point is that it's probably worth forcing a check of XFS as well as examining your drives for low-level retries if your write speed is slower than you expect. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC writes to disk very slow
Out of curiosity, have you experimented with other filesystems? I have couple relatively large setups (pool at ~ 9 terabytes) with ext4 and those still crunch backups happily. Previously I have used EXT4, only problem was once running out of inodes. But I find it hard to believe XFS with default options would be so much worse than any other file system with default options. XFS has been a bit hit and miss for me, I know many swear by it, but where I've tested it, I've hit all kinds of random problems to a degree that I haven't bothered with it any more. I have another host, same model and same RAID card. But disks are two 300 GB SAS disks in RAID1. There I got 11,2 MB/sec reading from 100Mbit/sek ethernet, and 127 MB/s with dd reading from /dev/zero. I got the speed down to 23.9 MB/s using 4k blocksize. I'll have to examine the problem host more. I've been recently experimenting with separating the pc and pool directories so pc directories are on SSD storage and the pool on HDDs. Jury is still out on real life speedups on this, but it seems possible with BPC4. Interesting approach. Have to experiment when I set up next BackupPC host. -- Tapio Lehtonen OSK Satatuuli <>___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC writes to disk very slow
On 2.12.2018 15.32, Tapio Lehtonen wrote: On new BackupPC host backups go very slow. I believe I have determined the network connection is not at fault, from Windows machine Speedtest.net shows a little less than gigabit speeds and from this backuppc host speedtest.cli shows over 800 Mbits / sek. Out of curiosity, have you experimented with other filesystems? I have couple relatively large setups (pool at ~ 9 terabytes) with ext4 and those still crunch backups happily. XFS has been a bit hit and miss for me, I know many swear by it, but where I've tested it, I've hit all kinds of random problems to a degree that I haven't bothered with it any more. I've been recently experimenting with separating the pc and pool directories so pc directories are on SSD storage and the pool on HDDs. Jury is still out on real life speedups on this, but it seems possible with BPC4. -- Ari Sovijärvi ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC writes to disk very slow
On new BackupPC host backups go very slow. I believe I have determined the network connection is not at fault, from Windows machine Speedtest.net shows a little less than gigabit speeds and from this backuppc host speedtest.cli shows over 800 Mbits / sek. But writing to disk is slow, about 3MBytes / second. Disk system is hw RAID 1 with two 4 TB SATA disks, and file system is XFS. In fact it is not just BackupPC, if I download large files the speed drops down to about 3 MBytes / sek. 01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) root@fuji:/srv# xfs_info /dev/sdb2 meta-data=/dev/sdb2 isize=512agcount=4, agsize=242689472 blks = sectsz=512 attr=2, projid32bit=1 = crc=1finobt=1 spinodes=0 rmapbt=0 = reflink=0 data = bsize=4096 blocks=970757888, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=474002, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 I ran Phoronix Test Suite, results: http://taleman.fi/Fujitsu_RX300/fuji-diskbench-2018-11-29/ Those measurements show 4 k block size random IO speeds down to 1 MByte / sek. With larger block sizes in sequential writes speeds are very good, better than I need with a gigabit connection. Is my setup somehow sub-optimal? Why would BackupPC write 4k blocks and use random IO? I noticed the slowness with the first big backup, disk was almost empty so BackupPC could just write stream to disk which to my mind should go with disk media speed, which should be > 100MBytes / sek even on this slowish SATA disks. I did not think disk system speed would be an issue, so I did not think about blocksize, separate log device etc when creating the file system. I chose XFS so I do not need to worry about inodes running out. Setup is defaults, BackupPC on Debian GNU/Linux 9.6. Why is writing to disk so slow? What can I do to make it faster? The current situation is annoying, I get a gigabit connection so backups go fast and end up with system where 0.03 GBit/sec connection would be just as good. Possible solutions? === Host has also two SSD disks, OS is installed there. Disks have about 150 GB unpartitioned space, so I could put XFS log or journal or something there. But looks like these must be set up during mkfs time, so now I need xfsdump, make better fs, xfsrestore. Bigger block size? I have not yet looked into what are sizes of files that bet backed up. Backup client hosts are web servers, e-mail servers and name servers. The performance tests indicate small block sizes are slow in random IO. -- Tapio Lehtonen OSK Satatuuli <>___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
