Am Dienstag, den 18.07.2017, 10:21 -0600 schrieb Jason Bailey: > I've got a mail server full of millions of tiny little files > (messages) > that I back up to my Bareos server at another location. It works, but > it > takes several days to complete, and I've had to use LVM snapshots to > get > steady, consistent backups, as the data changes fairly rapidly from > hour > to hour. > > In total I've got around 1.2 terabytes of data spread across 18-20 > million files that needs to be backed up to eliminate any possible > conflicts (uplink bandwidth on the switch during the day, for > example).
This sounds like maildir mailstorage of cyrus imapd. If you're using cyrus imapd, I wouldn't worry about snapshots at all. Just reconstruct/repair (if they got messed up during backup - very unlikely) and update them after a restore with cyrus tools. > I'm trying to see if there is a way to configure Bareos so that > there's > more efficient use of the bandwidth, or just find ways to speed the > whole process up. I've fiddled with setting the network buffer at > the > bareos-sd level, enabling compression, enabling/disabling spooling, > etc. > I'm getting10-15 MB/s at best, but looking uniquely at bandwidth > alone, > I should see even better transfer speeds (network bandwidth is at > least > 1 Gbps along the entire path). > > Bandwidth between locations isn't an issue. From what I can see, the > excess overhead of transferring so many small files is what is > causing > the majority of the slowdowns. Read speed from the disk (snapshot > I'm > coping is riding off of an Areca RAID card) is 10-12 Mbps. also turn off any kind of checksumming and other modification/access time checking / comparing etc (avoid stat() calls). (In case of cyrus) just back up the databases and any new files. Once a mail is there it's a file that will never change again - only vanish if it gets deleted or moved. Regards, Tom -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. For more options, visit https://groups.google.com/d/optout.
