Re: SIS and Filesystem level backups (was just Re: Filesystem level backups?)
Bummer, apparently things haven't changed (for the better) with respct to backup sand SIS... :( On 4/9/2018, 12:28:31 PM, Tanstaafl wrote: > On 4/9/2018, 11:34:40 AM, Ivan Warren wrote: >> Le 4/9/2018 à 4:56 PM, DurgaPrasad - DatasoftComnet a écrit : >>> Does doveadm backup backup the attachments as well when using SIS? > >> As far as I know, it does (it de-shares shared attachments) >> >> I've used that solution to stop using SIS at one point (it created >> more problems than it solved - especially permission issues) > > Hi Ivan, > > Mind if I ask for details? > > (and maybe Aki or Timo): > > I was considering implementing SIS on a new server I'm planning (won't > be for a few months now probably). I had decided against it a long time > ago because it was new, and there were drawbacks (backups not being > properly accounted for being one of the big ones). I was hoping it had > matured a lot since then and the drawbacks at the time were mostly > history. Maybe that is not the case? > > If not, are there open bugs dealing with the issues, and plan for > addressing them? Or is SIs just probably not ever going to be ready for > prime time? >
RE: SIS and Filesystem level backups (was just Re: Filesystem level backups?)
Hi all, I am deeply interested in this topic considering that SAN level snapshots are not possible. How do we effectively backup and restore SIS enable mdboxes? Regards Durga Prasad +919849111010 -Original Message- From: dovecot [mailto:dovecot-boun...@dovecot.org] On Behalf Of Tanstaafl Sent: 09 April 2018 21:59 To: dovecot@dovecot.org Subject: SIS and Filesystem level backups (was just Re: Filesystem level backups?) On 4/9/2018, 11:34:40 AM, Ivan Warren wrote: > Le 4/9/2018 à 4:56 PM, DurgaPrasad - DatasoftComnet a écrit : >> Does doveadm backup backup the attachments as well when using SIS? > As far as I know, it does (it de-shares shared attachments) > > I've used that solution to stop using SIS at one point (it created > more problems than it solved - especially permission issues) Hi Ivan, Mind if I ask for details? (and maybe Aki or Timo): I was considering implementing SIS on a new server I'm planning (won't be for a few months now probably). I had decided against it a long time ago because it was new, and there were drawbacks (backups not being properly accounted for being one of the big ones). I was hoping it had matured a lot since then and the drawbacks at the time were mostly history. Maybe that is not the case? If not, are there open bugs dealing with the issues, and plan for addressing them? Or is SIs just probably not ever going to be ready for prime time? --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
SIS and Filesystem level backups (was just Re: Filesystem level backups?)
On 4/9/2018, 11:34:40 AM, Ivan Warren wrote: > Le 4/9/2018 à 4:56 PM, DurgaPrasad - DatasoftComnet a écrit : >> Does doveadm backup backup the attachments as well when using SIS? > As far as I know, it does (it de-shares shared attachments) > > I've used that solution to stop using SIS at one point (it created > more problems than it solved - especially permission issues) Hi Ivan, Mind if I ask for details? (and maybe Aki or Timo): I was considering implementing SIS on a new server I'm planning (won't be for a few months now probably). I had decided against it a long time ago because it was new, and there were drawbacks (backups not being properly accounted for being one of the big ones). I was hoping it had matured a lot since then and the drawbacks at the time were mostly history. Maybe that is not the case? If not, are there open bugs dealing with the issues, and plan for addressing them? Or is SIs just probably not ever going to be ready for prime time?
Re: Filesystem level backups?
Le 4/9/2018 à 4:56 PM, DurgaPrasad - DatasoftComnet a écrit : Hello all, Does doveadm backup - backup the attachments as well when using SIS ? As far as I know, it does (it de-shares shared attachments) I've used that solution to stop using SIS at one point (it created more problems than it solved - especially permission issues). I backup up all my mailboxes - then made them active - all attachments were back in the individual mailboxes. --Ivan (Again sorry for the duplicate) smime.p7s Description: Signature cryptographique S/MIME
Re: Filesystem level backups?
> On Apr 9, 2018, at 5:34 AM, Marc Stürmer wrote: > > Am 2018-04-09 00:55, schrieb Steve Atkins: > >> Will a simple backup of all the files of an multi-dbox give me a >> consistent state when I recover, or do I need to do something (e.g. >> lock writes to the mailbox) while I run a backup? > > If you are doing this backup with a fully operational mail storage: no, it > won't. The reason is quite simple: mdbox saves the data of an email in > several files, so there's a good chance that those files might slightly > change during backup, while with Maildir all necessary data for the restore > is stored in the file names. > > If you want to do a consistent backup, you do need file system snapshots and > should make sure that this snapshot is consistent or to turn off the mail > store during backup completely Thanks. Time to read up on snapshots! Cheers, Steve
Re: Filesystem level backups?
Le 4/9/2018 à 2:34 PM, Marc Stürmer a écrit : Am 2018-04-09 00:55, schrieb Steve Atkins: Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? If you are doing this backup with a fully operational mail storage: no, it won't. The reason is quite simple: mdbox saves the data of an email in several files, so there's a good chance that those files might slightly change during backup, while with Maildir all necessary data for the restore is stored in the file names. If you want to do a consistent backup, you do need file system snapshots and should make sure that this snapshot is consistent or to turn off the mail store during backup completely. But just doing rsync on the fly as with Maildir is just begging on your side to run into problems sooner or later... Personally, I always do a doveadm backup -u to a different location (in mdbox format). Then do a git -A, git commit, git repack *THEN* rsync the resulting .git directories to an off-site location (thus only sending the difference between one bakup and the next - in packed format). The advantage is I can retrieve the status of any mailbox at any point in time (right now I have a year long of hour per hour backup of my servers mailboxes with a volume that is less than twice the size of the entire mailbox sizes). Granted, it's a low volume, low trafic server, but it works ! --Ivan (PS : Sorry Mark for the duplicate as I answered to you directly instead of the list) smime.p7s Description: Signature cryptographique S/MIME
RE: Re: Filesystem level backups?
Hello all,
Does doveadm backup - backup the attachments as well when using SIS ?
Regards
Durga Prasad
+919849111010
-Original Message-
From: dovecot [mailto:dovecot-boun...@dovecot.org] On Behalf Of Marc Stürmer
Sent: 09 April 2018 18:05
To: Steve Atkins
Cc: dovecot@dovecot.org
Subject: {Spam?} Re: Filesystem level backups?
Am 2018-04-09 00:55, schrieb Steve Atkins:
> Will a simple backup of all the files of an multi-dbox give me a
> consistent state when I recover, or do I need to do something (e.g.
> lock writes to the mailbox) while I run a backup?
If you are doing this backup with a fully operational mail storage: no, it
won't. The reason is quite simple: mdbox saves the data of an email in
several files, so there's a good chance that those files might slightly
change during backup, while with Maildir all necessary data for the restore
is stored in the file names.
If you want to do a consistent backup, you do need file system snapshots and
should make sure that this snapshot is consistent or to turn off the mail
store during backup completely.
But just doing rsync on the fly as with Maildir is just begging on your side
to run into problems sooner or later...
--
Greetings,
Marc
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Re: Filesystem level backups?
Ah, I didn't know there was a diff. b/w maildir and mdbox in terms of storage. I should have mentionned I am using Maildir. Yassine. On Monday, April 9, 2018, 1:35:15 PM GMT+1, Marc Stürmer wrote: Am 2018-04-09 00:55, schrieb Steve Atkins: > Will a simple backup of all the files of an multi-dbox give me a > consistent state when I recover, or do I need to do something (e.g. > lock writes to the mailbox) while I run a backup? If you are doing this backup with a fully operational mail storage: no, it won't. The reason is quite simple: mdbox saves the data of an email in several files, so there's a good chance that those files might slightly change during backup, while with Maildir all necessary data for the restore is stored in the file names. If you want to do a consistent backup, you do need file system snapshots and should make sure that this snapshot is consistent or to turn off the mail store during backup completely. But just doing rsync on the fly as with Maildir is just begging on your side to run into problems sooner or later... -- Greetings, Marc
Re: Filesystem level backups?
Am 2018-04-09 00:55, schrieb Steve Atkins: Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? If you are doing this backup with a fully operational mail storage: no, it won't. The reason is quite simple: mdbox saves the data of an email in several files, so there's a good chance that those files might slightly change during backup, while with Maildir all necessary data for the restore is stored in the file names. If you want to do a consistent backup, you do need file system snapshots and should make sure that this snapshot is consistent or to turn off the mail store during backup completely. But just doing rsync on the fly as with Maildir is just begging on your side to run into problems sooner or later... -- Greetings, Marc
Re: Filesystem level backups?
Been rsyncing my whole /var/vmail subdir (virtual users) for a while. So far so good. Yassine. On Sunday, April 8, 2018, 11:56:29 PM GMT+1, Steve Atkins wrote: Up until now I've been backing up my IMAP server by taking an atomic snapshot of the entire VM, so I've not really had to worry about it. I'm moving to hosting where I have neither VM snapshots nor a filesystem that can take snapshots, so I need to think about consistent backups a bit more. Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? (I could use doveadm backup if I were backing up to local storage I controlled, but I'm pushing it to offsite buckets so I'd prefer to avoid backing up the entire thing locally, then copying that offsite). Cheers, Steve
