>Did you verify that the NAS system can actually handle a sustained >data-stream of 300+ MB/s?
didn't i have done this with iperf3? i have spool attributes already enabled. pgsql is tuned with pgtune >So I guess we can rule out signatures and compression then, as that's >happening on the FD. so, as i have enough power on the FD side, would reenable lz4 be an option? >I really don't know. The only thing I can say for sure is that when >iperf reports only around 3 GBit/s, you definitely won't see a backup >transmitting more than 300 MB/s. so, the value looking for, is the rate on the actual network card? There i get 300 MB/s sometimes during job execution. On FD job report after 2 hours runtime i get: Files=4,775,011 Bytes=1,626,243,569,572 Bytes/sec=239,048,003 Errors=0 which is much better. the last change i've made was the change the signature from md5 to XXH128 Andreas Rogge schrieb am Donnerstag, 19. September 2024 um 14:15:36 UTC+2: > Am 19.09.24 um 13:06 schrieb Markus Dubois: > > the iperf looks not "at its best" because the bareos director and the > > storage daemon are installed on a NAS system with an Intel Atom 2,1 Ghz > > 4core CPU > Maybe that system simply cannot fully utilize 10 GBe. > Did you verify that the NAS system can actually handle a sustained > data-stream of 300+ MB/s? > > > On backup server CPU is on 30-50% overall > Overall as in 30-50% of all four cores? > Because that could mean one core is busy with bareos-sd and it simply > cannot work faster. > Also when bareos-dir and the postgresql database are on the same system, > you may run out of resources (CPU, memory or IOPS). > > > as mentioned above, i've launched a job without tls and lz4, the > > signature is configured as XXH128 > > I can see that the NAS is reporting 250 MB/s in average and 300 MB/s max > > during job runtime. > > But i have also, how to say this properly, speed crashes from 250 MB/s > > to 80 MB/s. The network rate is not constantly high. > So basically your maximum throughput is in the ballpark of what iperf > shows. That's not bad. > > Depending on the structure of files you're backing up, the drop to 80 > MB/s might be normal. If - for example - you have a set of large files, > these will be backed up around 250 MB/s. But as soon as the FD gets to a > directory with a lot of really small files, the rates will drop. > Even if the FD could keep up with it, the SD will send the file metadata > for every file to the director which will have to put it into the database. > Maybe it is a good idea to run a backup with (almost) only large files > (i.e. some video archive) and one with lots and lots of small files, you > might see a pattern evolving. > If it really is the database inserts, you can enable "Spool Attributes" > on the job. This will spool the metadata and only insert it to the > database when the job is finished. > > > the bareos FD is running on a 30 Core 192GB RAM server, i can see 75% > > CPU of one core usage. Overall server (to be backued up) is on 34 % CPU > So I guess we can rule out signatures and compression then, as that's > happening on the FD. > > > so, what do you mean with "network perform properly"? As mentioned, i > > have some hardware limitations on backup server side, but i think even > > those limitations shouldn't lead to those numbers.... > > Should i fiddle around with systemctl settings or max nettwork buffers? > I really don't know. The only thing I can say for sure is that when > iperf reports only around 3 GBit/s, you definitely won't see a backup > transmitting more than 300 MB/s. > > -- > Andreas Rogge [email protected] > Bareos GmbH & Co. KG Phone: +49 221-630693-86 <+49%20221%2063069386> > http://www.bareos.com > > Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646 > Komplementär: Bareos Verwaltungs-GmbH > Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz > -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/ad0766a7-dd8b-45c7-82b4-a166a6c643a3n%40googlegroups.com.
