Thank you for the reply. My responses are below:
On 07/19/2017 08:58 AM, Tom Weber wrote:
Am Dienstag, den 18.07.2017, 10:21 -0600 schrieb Jason Bailey:
I've got a mail server full of millions of tiny little files
(messages)
that I back up to my Bareos server at another location. It works, but
it
takes several days to complete, and I've had to use LVM snapshots to
get
steady, consistent backups, as the data changes fairly rapidly from
hour
to hour.
In total I've got around 1.2 terabytes of data spread across 18-20
million files that needs to be backed up to eliminate any possible
conflicts (uplink bandwidth on the switch during the day, for
example).
This sounds like maildir mailstorage of cyrus imapd. If you're using
cyrus imapd, I wouldn't worry about snapshots at all. Just
reconstruct/repair (if they got messed up during backup - very
unlikely) and update them after a restore with cyrus tools.
dovecot, actually. Even a full backup in Bareos (no checksumming, no
worry about the state of the capture, etc) still takes forever as it is.
I'm trying to see if there is a way to configure Bareos so that
there's
more efficient use of the bandwidth, or just find ways to speed the
whole process up. I've fiddled with setting the network buffer at
the
bareos-sd level, enabling compression, enabling/disabling spooling,
etc.
I'm getting10-15 MB/s at best, but looking uniquely at bandwidth
alone,
I should see even better transfer speeds (network bandwidth is at
least
1 Gbps along the entire path).
Bandwidth between locations isn't an issue. From what I can see, the
excess overhead of transferring so many small files is what is
causing
the majority of the slowdowns. Read speed from the disk (snapshot
I'm
coping is riding off of an Areca RAID card) is 10-12 Mbps.
also turn off any kind of checksumming and other modification/access
time checking / comparing etc (avoid stat() calls).
(In case of cyrus) just back up the databases and any new files. Once a
mail is there it's a file that will never change again - only vanish if
it gets deleted or moved.
While I did disable checksumming, I have not tried disabling
atime/mtime. I'm not sure if dovecot would be sensitive to that or not,
particularly in the event of a restore. Ironically, I actually have more
background with cyrus.
Regards,
Tom
Thanks,
Jason
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
