On dimanche, 5 novembre 2017 22.29:22 h CET Bob Farmer wrote: > On 11/05/2017 10:48 AM, Bruno Friedmann wrote: > > On samedi, 4 novembre 2017 17.56:36 h CET Bob Farmer wrote: > > [snip] > > > >> Don't think Spool Attributes would be relevant here because, as I > >> mentioned, I'm just backing up a single very large test file. First I'm > >> trying to evaluate Bareos's potential throughput, but of course the plan > >> later will be "real" backups, so I'll definitely check that out once I > >> get to that stage. > >> > >> I did discover the first bottleneck, after going through the source code > >> of the file daemon. The default fileset .conf file contains "Signature > >> = MD5". Even on a fairly nice CPU, that was capping the file daemon's > >> transmission speed to roughly 200 MB/sec because of the calculations > >> required. Now I'm able to get up to 350-500 MB/s which still isn't > >> anywhere near what it should be, but at least the first bottleneck is > >> down. > >> > >> At this point, with 350-500 MB/s coming into the storage daemon, the > >> storage daemon is using 100% of a CPU, which I'm guessing is the next > >> bottleneck. Not sure why it would be using so much, but that doesn't > >> seem to bode well for Bareos handling large amounts of incoming data. I > >> guess I'll try to track down what's going on by taking a look at the > >> storage daemon code now. > >> > >> Thanks for the reply. > > > > Did you have any kind of compression involved in your configuration ? > > Perhaps a lsof of the bareos-sd and some parts of your configuration can > > help. > No compression or encryption involved here. Should have mentioned that. > Also, all my testing is with the 17.2 release candidate. > > As I mentioned, after removing "Signature = MD5" from the fileset, it > doesn't look like there's any bottleneck on the File Daemon side > anymore. That process isn't using too much CPU at this point. (Both > the server and the client system in this test are dual Xeon E5-2623 v4 > 2.6GHz CPUs by the way, fairly reasonable power.) > > On the Storage Daemon side, I was able to get speeds up a little more by > disabling Block Checksum on the disk device. Up to around 600-700 MB/s > at times. At this point I did profiling on the Storage Daemon, and > found that it seems to spend all its time inside memcpy(), so I guess > there's too much copying around of buffers to really reach 10 gigabit > speeds with its single-threaded operation? > > I tried one other thing, given the memcpy() dependence, and since this > is a dual CPU NUMA system, I used the numactl command to pin the storage > daemon's CPU and memory to a single NUMA node. And that seemed to boost > performance a little bit more, I even saw up to 800 MB/sec at some > points. Again, Storage Daemon still using 100% CPU. Not sure I could > do much else besides changing to a CPU with maximum per-core speed. > > Bob
Thanks Bob for the additionnal informations. I think you should open a bugreport (with the information) you have. I should have a deeper look inside what's happening here with the storage daemon under 10Gps I have, but the network is different, and it's doing 10to 15 jobs at the same time from several clients 3 of them being also 10gbps. What I know for sure is that we use 1500 mtu on that network. -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. For more options, visit https://groups.google.com/d/optout.
