On 11/04/2017 09:22 AM, Douglas K. Rand wrote:
On 11/4/17 1:47 AM, Bob Farmer wrote:
Hi all, I'm testing out Bareos as a potential replacement to our
current backup software and have run into performance issues.
Wondering if any solutions are available.
Details of a fairly basic test I'm doing:
I have a Linux backup server (Director & SD) and a Linux backup client
(FD) connected with a private 10 gigabit network. Jumbo frames are
enabled on this net, and iperf3 confirms 9.9 gigabits bandwidth
between the two.
On the backup server, I have a disk-based volume (no tapes involved at
this point) on a disk system that can write >1 GB/s sequentially. On
the backup client, I have a large test file contained on a disk system
that can read >1 GB/s sequentially.
I can FTP the test file from the client to the server's backup disk at
>700 MB/s. But when I try to back it up with Bareos, the
transfer speed never goes above roughly 200 MB/s.
I've tried playing with the "Maximum Network Buffer Size"
extensively and it does change things a little bit, but still doesn't
go much above 200 MB/s. I've verified with strace that the setting
is actually working, looks like it affects the setsockopt() for
buffer size that Bareos does, as well as the chunk size in which it
reads the file during the backup and the amount it send()s at a time
over the TCP socket (although the latter seems capped at 1,000,000
regardless of what you set).
Any ideas? Is Bareos known to actually ever be able to achieve speeds
over 200 MB/s?
First try turning on Spool Attirbutes first. The bottleneck could be
writing the per-file details to the database. This was a big win for me.
If that doesn't get you where you need to be, try turning on Spool Data.
I think this will be less of a win for you on disk base backups,
especially with one client, but it would be good to try. With Spool Data
don't forget to specify a reasonable Spool Directory in your Device
section.
For me I care less about per-host performance but the overall
performance as I'm doing 10 concurrent backups of a set of about 100
hosts. With my on-disk backups I get good performance, albeit with a
1Gb/s NIC, with just Spool Attributes on.
Don't think Spool Attributes would be relevant here because, as I
mentioned, I'm just backing up a single very large test file. First I'm
trying to evaluate Bareos's potential throughput, but of course the plan
later will be "real" backups, so I'll definitely check that out once I
get to that stage.
I did discover the first bottleneck, after going through the source code
of the file daemon. The default fileset .conf file contains "Signature
= MD5". Even on a fairly nice CPU, that was capping the file daemon's
transmission speed to roughly 200 MB/sec because of the calculations
required. Now I'm able to get up to 350-500 MB/s which still isn't
anywhere near what it should be, but at least the first bottleneck is down.
At this point, with 350-500 MB/s coming into the storage daemon, the
storage daemon is using 100% of a CPU, which I'm guessing is the next
bottleneck. Not sure why it would be using so much, but that doesn't
seem to bode well for Bareos handling large amounts of incoming data. I
guess I'll try to track down what's going on by taking a look at the
storage daemon code now.
Thanks for the reply.
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
