On 11/05/2017 10:48 AM, Bruno Friedmann wrote:
On samedi, 4 novembre 2017 17.56:36 h CET Bob Farmer wrote:
[snip]
Don't think Spool Attributes would be relevant here because, as I
mentioned, I'm just backing up a single very large test file. First I'm
trying to evaluate Bareos's potential throughput, but of course the plan
later will be "real" backups, so I'll definitely check that out once I
get to that stage.
I did discover the first bottleneck, after going through the source code
of the file daemon. The default fileset .conf file contains "Signature
= MD5". Even on a fairly nice CPU, that was capping the file daemon's
transmission speed to roughly 200 MB/sec because of the calculations
required. Now I'm able to get up to 350-500 MB/s which still isn't
anywhere near what it should be, but at least the first bottleneck is down.
At this point, with 350-500 MB/s coming into the storage daemon, the
storage daemon is using 100% of a CPU, which I'm guessing is the next
bottleneck. Not sure why it would be using so much, but that doesn't
seem to bode well for Bareos handling large amounts of incoming data. I
guess I'll try to track down what's going on by taking a look at the
storage daemon code now.
Thanks for the reply.
Did you have any kind of compression involved in your configuration ?
Perhaps a lsof of the bareos-sd and some parts of your configuration can help.
No compression or encryption involved here. Should have mentioned that.
Also, all my testing is with the 17.2 release candidate.
As I mentioned, after removing "Signature = MD5" from the fileset, it
doesn't look like there's any bottleneck on the File Daemon side
anymore. That process isn't using too much CPU at this point. (Both
the server and the client system in this test are dual Xeon E5-2623 v4
2.6GHz CPUs by the way, fairly reasonable power.)
On the Storage Daemon side, I was able to get speeds up a little more by
disabling Block Checksum on the disk device. Up to around 600-700 MB/s
at times. At this point I did profiling on the Storage Daemon, and
found that it seems to spend all its time inside memcpy(), so I guess
there's too much copying around of buffers to really reach 10 gigabit
speeds with its single-threaded operation?
I tried one other thing, given the memcpy() dependence, and since this
is a dual CPU NUMA system, I used the numactl command to pin the storage
daemon's CPU and memory to a single NUMA node. And that seemed to boost
performance a little bit more, I even saw up to 800 MB/sec at some
points. Again, Storage Daemon still using 100% CPU. Not sure I could
do much else besides changing to a CPU with maximum per-core speed.
Bob
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
