Re: [Dovecot] backing up maildir dovecot files
Am 13.02.2013 08:20, schrieb Steffen Kaiser: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 12 Feb 2013, David Mehler wrote: Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes who would rename them and why? signature.asc Description: OpenPGP digital signature
Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 13 Feb 2013, Reindl Harald wrote: Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes who would rename them and why? /folder/ would be a Maildir on a production mail server. If rsync runs (without help from other funtionality, such as LVM or ZFS snapshots) and at the same time someone accesses the Maildir and: a) sees a message, which lets Dovecot rename the message file from ***/new/ to ***/cur b) tags a message with a keyword, which lets Dovecot rename the file to have some lower-case letter c) untags a keyword - remove that letter from the filename d) changes Deleted, Read, Answered status -- add/remove an upper-case letter from filename rsync might have cached a message with an old filename no longer physically present on disk. You will see a XYZ vanished message and that particular message is not backuped, but removed from backup. Some scripts use rsync in a loop in assumption, that no Maildir is accessed that often, that you get a clean run of rsync eventually. Maybe, if you tag a message with a new keyword, rsync already copied dovecot-keywords without the new keyword, but copy the message, when its filename has the keyword-letter. In fact, if I want to make 100% sure I get an exact copy, I do this: rsync /from /to rsync /from /to mv /from /from_locked sleep 1 rsync /from_locked /to mv /from_locked /from /from is the base directory of the Maildir. In my environment that causes tempfails on delivery and internal server errors on IMAP/POP3 access. But the time between the two mv's is very small, because the main differences are handled with the first two rsync's. As you seen, I copy one user after another, which breaks hardlinks between users, but keeps memory footprint of rsync low and as well as the downtime. I do make my usual daily backups without the loop and the mv's, because it is very seldom I get the vanished messages. I suppose, because the script runs two hours after midnight. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURuQIF3r2wJMiz2NAQKgjgf8CtyeO6pONFU038x8O2wggcntdvGyeg8Z Uq8KR0QFVg1n0HNDaa33OYN4IxSHX8zNvElf+wc0ejQ3NUOPVDl2mdm4iEihyOYv Veb/p2iK671Nrs8nB7USwx7OE9vY8IYoB/ZSXrXGWowqOqRQIcJWHAfZ9Ewj3Rg/ iRGMUNCn7UfEDfWl+F5yWpdp/+3xJGxoWeWaegW/yfTzlJ5nKffS/SAfJlUm7zuV u31JL4fjk25uGG7alzrCxOq0z4A3PvcpGag2nkfIRbrLLmo4Wzr+09Bd2zqmSn74 /PaHlxFS6a2uy7ugqdd5kxfiZHnOS2/d6JgV428I8qS0CWugEPuFbg== =mw5e -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Feb 2013, Reindl Harald wrote: Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.rei...@thelounge.net wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my rsync.sh but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand --delete-after this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know hey my source contains exactly what i need, not more and lot less this is not what i would like and never did in 10 years IT If you use --delete-after you could add --fuzzy, which theoretically could save transfers because of filename renames. Anyway: Any form of --delete is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not --delete, the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in maildir/new and later moved to maildir/cur. So one ends with the same message in new and in cur. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
Hello, Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? Thanks. Dave. On 2/12/13, Steffen Kaiser skdove...@smail.inf.fh-brs.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Feb 2013, Reindl Harald wrote: Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.rei...@thelounge.net wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my rsync.sh but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand --delete-after this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know hey my source contains exactly what i need, not more and lot less this is not what i would like and never did in 10 years IT If you use --delete-after you could add --fuzzy, which theoretically could save transfers because of filename renames. Anyway: Any form of --delete is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not --delete, the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in maildir/new and later moved to maildir/cur. So one ends with the same message in new and in cur. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 12 Feb 2013, David Mehler wrote: Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes. On 2/12/13, Steffen Kaiser skdove...@smail.inf.fh-brs.de wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Feb 2013, Reindl Harald wrote: Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.rei...@thelounge.net wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my rsync.sh but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand --delete-after this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know hey my source contains exactly what i need, not more and lot less this is not what i would like and never did in 10 years IT If you use --delete-after you could add --fuzzy, which theoretically could save transfers because of filename renames. Anyway: Any form of --delete is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not --delete, the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in maildir/new and later moved to maildir/cur. So one ends with the same message in new and in cur. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -END PGP SIGNATURE- - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURs+tV3r2wJMiz2NAQIvngf/c37Hbg30lIq0wFPCif9IAG+BO8yDRAa3 lYY2t7DlszWopeEBvV22HjUC8SlzywECYEhBFRWhJVfqNyu55oI867of+RZ6lO0A lNcThmNixX5IsFiUnN9S9NYrTENC2qhccPdrTCAcm6A6CtaR1ydyeZDxlmvmTWe4 Za60LVqlsIVoZQ146yGE5nSIctix8JOE5kfGO5NurYXTfHt9CrPj5JvlPmRUdasp Mtn+QpxaxzFa8NGECSThKDVilMPFvqfT+JrhjWnkM1v31hNjJj4F3DYhk1L0YlAL 4wUjskXq1ytoM8+k9c8rqCy5DvM7g8N2ip0c0Buvjt+RctaHE4F9jg== =moaG -END PGP SIGNATURE-
[Dovecot] backing up maildir dovecot files
Hello, I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. Recommendations? Thanks. Dave.
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes signature.asc Description: OpenPGP digital signature
Re: [Dovecot] backing up maildir dovecot files
On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.rei...@thelounge.net wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. Of course there are arguments for instant deletion too, like the fact that a person runs the risk of accidentally deleting the wrong one, or the fact that a huge maildir folder could take hours to delete. Personally, I never delete til I'm sure it got to the right place and is backed up there, and sometimes I keep it on the old box for a month just in case if the old box won't be used anymore. SteveT
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald h.rei...@thelounge.net wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my rsync.sh but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand --delete-after this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know hey my source contains exactly what i need, not more and lot less this is not what i would like and never did in 10 years IT signature.asc Description: OpenPGP digital signature
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 21:34, schrieb David Mehler: Hello, I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. Recommendations? Thanks. Dave. try rsync, also dsync or imapsync may help Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: [Dovecot] Backing Up To Windows File Server
On 12/13/2011 04:21 PM, Asai wrote: Greetings, Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? Don't have any answers for you, but I know that : is an invalid character in Windows filenames, so that could have something to do with it.
Re: [Dovecot] Backing Up To Windows File Server
On 12/22/2011 01:30 AM, Willie Gillespie wrote: On 12/13/2011 04:21 PM, Asai wrote: Greetings, Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? Don't have any answers for you, but I know that : is an invalid character in Windows filenames, so that could have something to do with it. Whoops, should have finished going through the old thread. Seems like MyBSD already answered.
Re: [Dovecot] Backing Up To Windows File Server
On 12/13/2011 5:34 PM, Jerry wrote: On Tue, 13 Dec 2011 16:21:09 -0700 Asai articulated: Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? I have backed up files to a Windows server before but have never experienced the problems you are describing. Could you please list the Windows Server specifics, ie. version, etc. This is a Win 2008 R2 server with NFS shares enabled. I initially copied the backups from my former backup server to this Windows box, and that's when I noticed the file names had been changed.
Re: [Dovecot] Backing Up To Windows File Server
On Wed, 14 Dec 2011 10:17:58 -0700 Asai articulated: On 12/13/2011 5:34 PM, Jerry wrote: On Tue, 13 Dec 2011 16:21:09 -0700 Asai articulated: Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? I have backed up files to a Windows server before but have never experienced the problems you are describing. Could you please list the Windows Server specifics, ie. version, etc. This is a Win 2008 R2 server with NFS shares enabled. I initially copied the backups from my former backup server to this Windows box, and that's when I noticed the file names had been changed. IMHO, I think you might be better served by posting your inquiry to a Microsoft forum dedicated to the 2008 server framework. -- Jerry ♔ Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __
Re: [Dovecot] Backing Up To Windows File Server
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Asai said the following on 14/12/11 17:52:
Thanks, Luigi, I may fall back to that.
Should you need it, here's the script I use to do it.
In my configurations /var/spool/mail contains one dir for each domain and each
of that dir contains a maildir for each domain. So the email of
foo...@acme.com is in /var/spool/mail/acme.com/foobar/
The 4th line checks lostfound because /bar/spool/mail is a different file
system.
I create the .tgz file on local /tmp for performance reasons.
TARGET=/backup
for DOMAINPATH in /var/spool/mail/*
do
if [ ${DOMAINPATH} != /var/spool/mail/lost+found ]
then
DOMAIN=`echo $DOMAINPATH | cut -d '/' -f 5`
for USERPATH in ${DOMAINPATH}/*
do
USER=`echo $USERPATH | cut -d '/' -f 6`
tar cvzf /tmp/$DOMAIN-$USER.tgz $USERPATH /dev/null
cp -f /tmp/$DOMAIN-$USER.tgz $TARGET/mail
rm -f /tmp/$DOMAIN-$USER.tgz
done
fi
done
Ciao,
luigi
- --
/
+--[Luigi Rosa]--
\
I don't think we have the right or the wisdom to interfere,
however a planet is evolving.
--James Kirk, The Omega Glory
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7o3jYACgkQ3kWu7Tfl6ZSfpQCgvzhOJH6mnJMu8ZzvKu5y8um+
46wAoKd0eXxBNPad9EZao7VKjZBkACer
=dhk0
-END PGP SIGNATURE-
[Dovecot] Backing Up To Windows File Server
Greetings, Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? -- --asai
Re: [Dovecot] Backing Up To Windows File Server
On Tue, 13 Dec 2011 16:21:09 -0700 Asai articulated: Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? I have backed up files to a Windows server before but have never experienced the problems you are describing. Could you please list the Windows Server specifics, ie. version, etc. -- Jerry ♔ Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __
Re: [Dovecot] Backing Up To Windows File Server
On Tuesday, December 13, 2011 at 23:21:09 UTC, a...@globalchangemusic.org confabulated: Greetings, Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? AFAIK, the colon is not valid in a windows file name. -- If at first you don't succeed, so much for skydiving.
Re: [Dovecot] Backing Up To Windows File Server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Asai said the following on 14/12/11 00:21: Working with dsync and setting up backups to a Windows file server. Problem seems to be that Windows is renaming the dovecot mail files (maildir) to Windows friendly filenames, and losing the Dovecot name. For example, 1323817925.M36368P32049.triata.globalchangemultimedia.net,S=2255,W=2318:2,S becomes 1AETPH~X. Does anyone have any ideas about how to tell Windows to not do this? Or is it not possible? Got same problem rsync-ing to some low cost NAS and, of course, windows share. My solution is to tar.gz before copying to Windows. Ciao, luigi - -- / +--[Luigi Rosa]-- \ Ask ten different scientists about the environment, population control, genetics, and you'll get ten different answers, but there's one thing every scientist on the planet agrees on. Whether it happens in a hundred years or a thousand years or a million years, eventually our Sun will grow cold and go out. When that happens, it won't just take us. It'll take Marilyn Monroe, and Lao-Tzu, and Einstein, and Morobuto, and Buddy Holly, and Aristophanes... [and] all of this... all of this... was for nothing. Unless we go to the stars. --Jeffrey Sinclair, Infection -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7oNjYACgkQ3kWu7Tfl6ZTaBACaAmu1MtZkcBEGxMM3iXfezLpr KEsAnjczFO4QVnIcHtvC2MbWGbU7AuO2 =cGog -END PGP SIGNATURE-
Re: [Dovecot] Backing up dovecot
Spyros Tsiolis wrote on 11/12/2010: a. How safe will I be backing up dovecot b. What folders/files to backup It's enough to backup your Dovecot configuration files (usually the folder /etc/dovecot or /usr/local/etc/dovecot) and your SSL certs (if used). You should also backup your mailboxes (/var/MailRoot/domains) on a regular basis (hourly/daily with rsync for example). If you must reinstall your server you have to install Dovecot and restore your Dovecot config files from your backup. That should be all. P.S.: Depending of your authentication backend (database, PAM, LDAP..) you should also backup these informations so that users can login after a restore with the same credentials as before. -- Daniel
[Dovecot] Backing up dovecot
Hello people, Well, what the subjects says :-) I have a dovecot/Horde installation and would like to know : a. How safe will I be backing up dovecot b. What folders/files to backup Let me see now; I am running dovecot v1.2.15 Here's a dump of dovecot -n : [r...@mailgate ~]# dovecot -n # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-194.17.1.el5PAE i686 CentOS release 5.5 (Final) ext3 base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/dovecot-info.log ssl: required ssl_parameters_regenerate: 48 verbose_ssl: yes login_dir: /var/run/dovecot//login login_executable: /usr/local/dovecot/libexec/dovecot/imap-login login_greeting: * Dovecot ready * login_max_processes_count: 32 mail_location: maildir:/var/MailRoot/domains/%d/%n/Maildir mail_plugins: zlib auth default: verbose: yes debug: yes debug_passwords: yes passdb: driver: passwd-file args: /etc/dovecot/passwd passdb: driver: pam userdb: driver: static args: uid=vmail gid=vmail home=/home/vmail/%u userdb: driver: passwd Thank you all, Best regards, spyros I merely function as a channel that filters music through the chaos of noise - Vangelis
Re: [Dovecot] backing up emails
Hi Noel, I use exim4 and it delivers right into ~/.Maildir, so I assume /home contains all the emails, right? why we need to back up /etc and /var/mail? Thanks, Angelo auth default: passdb: driver: pam userdb: driver: passwd You're using system accounts so yes, but I'd hope that if server is on any importance you would be doing more than just backing up that. /home /etc /var/mail or where ever your MTA stores its mail before users get it should be backed up daily at a bare minimum. If this is a server of importance you should be doing nightly tar backups of at least /etc, once a week a full rsync of the entire box, and a nightly rsync of your mail store, in your case /home and /var/mail (or /var/spool/postfix if you're using that), and use that on a rolling 7 day basis... providing you have the space for it on your backup server. Cheers
Re: [Dovecot] backing up emails
On Tue, 2010-06-29 at 22:28 +0800, Angelo Chen wrote: Hi Noel, I use exim4 and it delivers right into ~/.Maildir, so I assume /home contains all the emails, right? why we need to back up /etc and /var/mail? Thanks, No real experience with Exim, last time I looked at it was 10 years ago, but if that's where it stores all received messages, read or new-unread, then /home it is. /etc contains most of your system config stuff, always good idea to back it up, if this is a server it should have no GUI crap installed so it should be fairly small, and /var/vmail was an example only, just like i'd say /var/www if this was a web server, or /var/named if DNS server. /var/mail /var/vmail etc is not applicable to you as you use system and not virtual users. Cheers You're using system accounts so yes, but I'd hope that if server is on any importance you would be doing more than just backing up that. /home /etc /var/mail or where ever your MTA stores its mail before users get it should be backed up daily at a bare minimum. If this is a server of importance you should be doing nightly tar backups of at least /etc, once a week a full rsync of the entire box, and a nightly rsync of your mail store, in your case /home and /var/mail (or /var/spool/postfix if you're using that), and use that on a rolling 7 day basis... providing you have the space for it on your backup server. Cheers
Re: [Dovecot] backing up emails
On Mon, 2010-06-28 at 14:27 +0800, Angelo Chen wrote: auth default: passdb: driver: pam userdb: driver: passwd You're using system accounts so yes, but I'd hope that if server is on any importance you would be doing more than just backing up that. /home /etc /var/mail or where ever your MTA stores its mail before users get it should be backed up daily at a bare minimum. If this is a server of importance you should be doing nightly tar backups of at least /etc, once a week a full rsync of the entire box, and a nightly rsync of your mail store, in your case /home and /var/mail (or /var/spool/postfix if you're using that), and use that on a rolling 7 day basis... providing you have the space for it on your backup server. Cheers
[Dovecot] backing up emails
hi, i have a ubuntu server running Dovecot imap, how to backup everybody's email? rsync /home is enough? Thanks, Angelo
Re: [Dovecot] backing up emails
It depens where dovecot are storing the email! Do you know where is it ? post the output of # dovecot -n []'sf.rique On Sat, Jun 26, 2010 at 8:35 PM, Angelo Chen angelochen...@gmail.comwrote: hi, i have a ubuntu server running Dovecot imap, how to backup everybody's email? rsync /home is enough? Thanks, Angelo
Re: [Dovecot] Backing Up
Hello all. I read al this tread and still wondering - what FS best to use to manage up to 1,5Tb of maildir spool and make a near to real time back up of it? Firstly i plan to make it all on FreeBSD with UFS2 and use rsync, but I never rsync such much of space and files. If using solaris best is ZFS - but i don`t have any experience on this OS. What Linux can provide for this task? -- Best reagrds, Proskurin Kirill
Re: [Dovecot] Backing Up
Proskurin Kirill schrieb: Hello all. I read al this tread and still wondering - what FS best to use to manage up to 1,5Tb of maildir spool and make a near to real time back up of it? Firstly i plan to make it all on FreeBSD with UFS2 and use rsync, but I never rsync such much of space and files. If using solaris best is ZFS - but i don`t have any experience on this OS. What Linux can provide for this task? Hi, i use maildir format, its no problem to backup big maildirs with rsync, at backup i see no real relation with with filessystem until you use a modern one ( i.e. ext3... ), perhaps you might use nfs or gfs for redundancy solutions after all this type of backing up ( tar , rsync, i.e by cron ) is only a snapshot ( maybe ok on your side ), but i.e you may use always_bcc i.e with postfix or imapsync etc choosing zfs maybe is the right choice anyway i ve read much good about it, but i dont see what it might helpfull at backup -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Backing Up
Dave McGuire wrote: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave I have to wonder. I have a mailserver that I do a bootable complete image copy of with all files and O/S in two hours to an Ultrium-2 tape, 95 GB. When I switch to maildir, I will go from some 25,000 mbox files to 2.5 to 3 million files...I can't believe that isn't going to hurt and will force me into incrementals. My thoughts on rsync. You may want to consider that incremental backups wont help you much if you use Maildir. Incremental or full rsync still has to generate a list of all the files. Whether itll work for you is impossible to say. I guess youll just have to make a test. But you're right that the large amount of files will be an issue. Rsync seems to be loading information about each file into memory before comparing the lists of files and doing the actual transfer. That may be a lot of memory if you have a lot of files. I sometimes overcome this by rsyncing each user or domain one at a time. That way you will also limit issues of files no longer existing once the transfer begins (makes rsync generate errors). You can estimate the time needed to list all the files. Try and use iostat to get a rough idea of how many OIPS your system handles under max stress load and how many it handles under normal operation. The difference is the amount available to you during the backup. Divide the total number of files with the number of available IOPS. Say you have 100 IOPS available then it will take roughly 8 hours (3,000,000/100/3600=8.3 hours) to generate the list of 3,000,000 files. The afterwards transfer will probably be a lot faster. I'm not sure whether reading information about one file take up one IO operation. But that way of calculating the time to generate the lists wasn't much off last time I tried. One option that I would prefer if I were to backup the entire store with one command would be generating a snapshot of the file system. And then rsync or cp that snapshot. That way youll always get a consistent backup and you wont have to worry about how long the backup takes to finish. Regards, Mikkel
Re: [Dovecot] Backing Up
[EMAIL PROTECTED] wrote: Rsync seems to be loading information about each file into memory before comparing the lists of files and doing the actual transfer. That may be a lot of memory if you have a lot of files. I sometimes overcome this by rsync’ing each user or domain one at a time. That way you will also limit issues of files no longer existing once the transfer begins (makes rsync generate errors). If you are using rsync2 then definitely this is good advice. Massive memory consumption to backup large mailboxes (and a long time before anything starts happening, ie snapshot useful) However, with rsync3 you should look at the options required to do use the incremental protocol. This trades a bit of efficiency on hardlinked files for lower memory and perhaps faster sync speeds. I haven't personally tried this, but reports on the web seem promising. You need rsync3 at both ends of the link and to examine your sync options a little However, one thing which is sadly missing on rsync is a fuzzy option which can spot files moving from /new to /cur... This may well cause additional load for imap backups which is potentially avoidable with a simple copy. I suspect it would be easy to patch a custom bit of code to handle this though..? One option that I would prefer if I were to backup the entire store with one command would be generating a snapshot of the file system. And then rsync or cp that snapshot. That way you’ll always get a consistent backup and you won’t have to worry about how long the backup takes to finish. Snapshot seems like an excellent idea to avoid files missing files moving between /cur and /new. However, it should be pointed out that this is extra io for the server (with LVM at least) whilst the backup is running I should think rsync3 incremental, plus some custom patching to look for files moving between /cur and /new would be very efficient for backing up maildir filestores (at least to the extent your filesystem allows efficient iterating over lots of files) Ed W
Re: [Dovecot] Backing Up
On Thu, 2008-10-30 at 14:42 -0400, Allen Belletti wrote: I'd like to add my vote here as well; dbox would be *the* feature that would make me happy. I'm the guy who asked a few weeks ago about ways to speed access on our GFS clustered mail environment. Meanwhile, I've done some preliminary testing with mbox. As expected, it's vastly faster than the Maildirs that we're using now. Of course it pains me to go backwards but that may be the interim solution. I got stopped temporarily when it seemed that I couldn't nest folders using mbox, but hopefully that's untrue. You could use Maildir++ layout with mboxes too: mail_location = mbox:~/mail:LAYOUT=maildir++ mbox handling code is less stable than maildir code though. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Backing Up
[EMAIL PROTECTED] wrote: One option that I would prefer if I were to backup the entire store with one command would be generating a snapshot of the file system. And then rsync or cp that snapshot. That way youll always get a consistent backup and you wont have to worry about how long the backup takes to finish. Snapshot seems like an excellent idea to avoid files missing files moving between /cur and /new. However, it should be pointed out that this is extra io for the server (with LVM at least) whilst the backup is running I only have experience wuth UFS (FreeBSD) and ZFS (Solaris). Snapshots on UFS is a horrible thing for large file systems. Snapshots on ZFS is marvellous (which I use). It does not result in any extra IO whatsoever due to some clever designing. If you have the option of using ZFS it's definitely the best way to do it. Regards, Mikkel
Re: [Dovecot] Backing Up
I use the tar/bzip method, and have been wondering about the rsync. All my users have system accounts on the dovecot server, and use Maildir format. If i rsync the mail to another box where the users do not have system accounts, will the ownerships/ permissions etc. be goofed up ? Correctly, or incorrectly, I've been using tar to preserve all that information. Cal Gordon Sotiris Tsimbonis wrote: Scott Silva wrote, On 10/30/2008 12:34 AM: on 10-29-2008 3:18 PM Dave McGuire spake the following: On Oct 29, 2008, at 5:32 PM, Arkadiusz Miskiewicz wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. Due to connection turnaround latency, I assume? (I've never looked at the rsync protocol) If that's the case, then I stand very much corrected, thank you. I was going from the same logic regarding mbox vs. maildir in the context of backups. One new message delivered and a 400MB mail spool gets backed up again.. -Dave Rsync adds some latency as it indexes and compares files on both ends. Obviously it would take more time to compare 40,000 1K files then 1000 40K files even though the data size is similar. It would still be better than tar/bzip/scp which has to compress everything and transfer the lot every time. Maildirsync it an Online synchronizer for Maildir-format mailboxes See http://hacks.dlux.hu/maildirsync/ Sot.
Re: [Dovecot] Backing Up
Calvin Gordon wrote: I use the tar/bzip method, and have been wondering about the rsync. All my users have system accounts on the dovecot server, and use Maildir format. If i rsync the mail to another box where the users do not have system accounts, will the ownerships/ permissions etc. be goofed up ? Correctly, or incorrectly, I've been using tar to preserve all that information. rsync preserves all that too, but you should preserve uid-username and gid-groupname mappings too, otherwise all that information is not as useful. Saving the password files is usually sufficient, assuming you are doing backups for disaster recovery, and not just for the occasional restore after an oops, I deleted all my mail! phonecall. rsnapshot is nice too. It uses rsync and hard links to make as many snapshots of the filesystem as you like. This creates many 'restore points' with total disk usage being just over what a single full backup would take. Ken Cal Gordon Sotiris Tsimbonis wrote: Scott Silva wrote, On 10/30/2008 12:34 AM: on 10-29-2008 3:18 PM Dave McGuire spake the following: On Oct 29, 2008, at 5:32 PM, Arkadiusz Miskiewicz wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. Due to connection turnaround latency, I assume? (I've never looked at the rsync protocol) If that's the case, then I stand very much corrected, thank you. I was going from the same logic regarding mbox vs. maildir in the context of backups. One new message delivered and a 400MB mail spool gets backed up again.. -Dave Rsync adds some latency as it indexes and compares files on both ends. Obviously it would take more time to compare 40,000 1K files then 1000 40K files even though the data size is similar. It would still be better than tar/bzip/scp which has to compress everything and transfer the lot every time. Maildirsync it an Online synchronizer for Maildir-format mailboxes See http://hacks.dlux.hu/maildirsync/ Sot. -- Ken Anderson Pacific.Net
Re: [Dovecot] Backing Up
Dave McGuire wrote: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave I have to wonder. I have a mailserver that I do a bootable complete image copy of with all files and O/S in two hours to an Ultrium-2 tape, 95 GB. When I switch to maildir, I will go from some 25,000 mbox files to 2.5 to 3 million files...I can't believe that isn't going to hurt and will force me into incrementals.
Re: [Dovecot] Backing Up
On Thu, 2008-10-30 at 11:00 -0400, Stewart Dean wrote: Dave McGuire wrote: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave I have to wonder. I have a mailserver that I do a bootable complete image copy of with all files and O/S in two hours to an Ultrium-2 tape, 95 GB. When I switch to maildir, I will go from some 25,000 mbox files to 2.5 to 3 million files...I can't believe that isn't going to hurt and will force me into incrementals. One possibility is to just wait for dbox with multiple-messages-per-file feature. I can't really say when it'll be ready (or when I'll even start implementing it), but I know I want to use it myself and some companies have also recently been asking about it. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Backing Up
Timo Sirainen: One possibility is to just wait for dbox with multiple-messages-per-file feature. I can't really say when it'll be ready (or when I'll even start implementing it), but I know I want to use it myself and some companies have also recently been asking about it. Have you considered making dbox a major priority for v. 1.2? I have been holding back on v.1.2 because I dont really see the big improvements in it that I saw in v.1.0 and v.1.1. With 1.0 and 1.1 I hurried off using them in production environments even while they where still in beta (of course only after proper testing) because they posed so many advantages (primarily speed and stability) over other solutions. Since Im focused almost entirely on stability and speed, and very little on fancy functionality, what v.1.0 offers in terms of functionality is just fine. What drove me towards 1.1 were speed improvements (and stability on NFS). I remember you made a post about not many people testing v.1.2. I think the reason may be that most users feel the same as me. Theyd like to se a major feature that benefits their primary needs, which isnt in term of functionality but more in term of speed improvements. Dbox could be that feature as I think there isnt much room for further developing the Maildir format (and as far as I can see you have gone as far as possible with regards to optimizing speed while working within the boundaries of the Maildir standard). Maildir is nice compared to mbox but it really isnt optimal. In days where IOPS is the most difficult resource to get into your server (and dovecot already using close to nothing in terms of CPU time and memory) having one file per e-mail is less than sub-optimal especially when a large amount of users just downloads the whole mailbox using POP3 (not to mention backing up Maildirs). Now don't take this as a critic, I love your software. I just would really like to se dbox evolve and think it would be a major driving force for v.1.2 :) Develop dbox, Do it. Do it naoughw! (preferably pronounced with a schwarzeneggerish accent like in the last three seconds of this splendid video http://www.youtube.com/watch?v=adc3MSS5Ydc). Best regards, Mikkel
Re: [Dovecot] Backing Up
I'd like to add my vote here as well; dbox would be *the* feature that would make me happy. I'm the guy who asked a few weeks ago about ways to speed access on our GFS clustered mail environment. Meanwhile, I've done some preliminary testing with mbox. As expected, it's vastly faster than the Maildirs that we're using now. Of course it pains me to go backwards but that may be the interim solution. I got stopped temporarily when it seemed that I couldn't nest folders using mbox, but hopefully that's untrue. Allen [EMAIL PROTECTED] wrote: Timo Sirainen: One possibility is to just wait for dbox with multiple-messages-per-file feature. I can't really say when it'll be ready (or when I'll even start implementing it), but I know I want to use it myself and some companies have also recently been asking about it. Have you considered making dbox a major priority for v. 1.2? I have been holding back on v.1.2 because I don’t really see the big improvements in it that I saw in v.1.0 and v.1.1. With 1.0 and 1.1 I hurried off using them in production environments even while they where still in beta (of course only after proper testing) because they posed so many advantages (primarily speed and stability) over other solutions. Since I’m focused almost entirely on stability and speed, and very little on fancy functionality, what v.1.0 offers in terms of functionality is just fine. What drove me towards 1.1 were speed improvements (and stability on NFS). I remember you made a post about not many people testing v.1.2. I think the reason may be that most users feel the same as me. They’d like to se a major feature that benefits their primary needs, which isn’t in term of functionality but more in term of speed improvements. Dbox could be that feature as I think there isn’t much room for further developing the Maildir format (and as far as I can see you have gone as far as possible with regards to optimizing speed while working within the boundaries of the Maildir standard). Maildir is nice compared to mbox but it really isn’t optimal. In days where IOPS is the most difficult resource to get into your server (and dovecot already using close to nothing in terms of CPU time and memory) having one file per e-mail is less than sub-optimal especially when a large amount of users just downloads the whole mailbox using POP3 (not to mention backing up Maildirs). Now don't take this as a critic, I love your software. I just would really like to se dbox evolve and think it would be a major driving force for v.1.2 :) Develop dbox, Do it. Do it naoughw! (preferably pronounced with a schwarzeneggerish accent like in the last three seconds of this splendid video http://www.youtube.com/watch?v=adc3MSS5Ydc). Best regards, Mikkel -- Allen Belletti [EMAIL PROTECTED] 404-894-6221 Phone Industrial and Systems Engineering404-385-2988 Fax Georgia Institute of Technology
Re: [Dovecot] Backing Up
On Oct 30, 2008, at 2:35 PM, [EMAIL PROTECTED] wrote: Maildir is nice compared to mbox but it really isn’t optimal. In days where IOPS is the most difficult resource to get into your server (and dovecot already using close to nothing in terms of CPU time and memory) having one file per e-mail is less than sub-optimal especially when a large amount of users just downloads the whole mailbox using POP3 (not to mention backing up Maildirs). It seems to me that a database like Postgres or MySQL would be the best bet. -Dave -- Dave McGuire Port Charlotte, FL
Re: [Dovecot] Backing Up
on 10-30-2008 11:42 AM Allen Belletti spake the following: I'd like to add my vote here as well; dbox would be *the* feature that would make me happy. I'm the guy who asked a few weeks ago about ways to speed access on our GFS clustered mail environment. Meanwhile, I've done some preliminary testing with mbox. As expected, it's vastly faster than the Maildirs that we're using now. Of course it pains me to go backwards but that may be the interim solution. I got stopped temporarily when it seemed that I couldn't nest folders using mbox, but hopefully that's untrue. You can nest folders with mbox, but you can't have folders that contain both messages and other folders. A folder in mbox can either hold messages or other folders, but not both. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Backing Up
On Oct 30, 2008, at 2:35 PM, [EMAIL PROTECTED] wrote: Maildir is nice compared to mbox but it really isnt optimal. In days where IOPS is the most difficult resource to get into your server (and dovecot already using close to nothing in terms of CPU time and memory) having one file per e-mail is less than sub-optimal especially when a large amount of users just downloads the whole mailbox using POP3 (not to mention backing up Maildirs). It seems to me that a database like Postgres or MySQL would be the best bet. That's a matter of opinion. Moving mail storage to a database would probably be the last thing I would ever do (I'm not saying it's not the right thing for some people. I'm just not one of them). I'm using mysql for storing the users database but thats another story. Adding a database is one additional level of complexity. One more program to govern. In my opinion it's nice to know that as long as the disk is readable nothing can go completely wrong. The database in my case would be roughly 400 GB holding some 60 million records. Just imagine if one single byte got written to the wrong place. Power outage, OS crash, software bug or whatever could easily result in this (I regularly experience mysql tables that crash on their own from heavy use). Having to run a repair on a table of that size whilst all users are eager to get to their data must be a nightmare of proportions. Just imagine backing the thing up, exporting 60.000.000 SQL queries. Not to say importing them again if something should go really wrong. Actually I'n not even sure it would be faster. When the index files grow to several gigabytes they kind of loose their purpose. Maildir is very resilient to various errors. It is virtually impossible to corrupt a maildir (at least I've never experienced anything). Also you can backup up the thing without worrying about anything accessing it at the same time. Mbox less so but still a lot better than having one huge database. Dbox would be the ultimate compromise between crash resilience and a low number of files (not to mention the enormous potential for speed gains). Regards, Mikkel
Re: [Dovecot] Backing Up
Scott Silva wrote: Rsync will use more memory on large filesystems, but it is usually lighter in CPU, network, and IO time. But tar gives you multiple backups. To achieve that with rsync you need the rbackup script or rsnapshot. Also check snapback2 (similar to tools you mentioned above) And brackup looks quite interesting for backing up maildir... (same chap who wrote memcached) Ed W
Re: [Dovecot] Backing Up
[EMAIL PROTECTED] wrote: On Oct 30, 2008, at 2:35 PM, [EMAIL PROTECTED] wrote: Maildir is nice compared to mbox but it really isn’t optimal. In days where IOPS is the most difficult resource to get into your server (and dovecot already using close to nothing in terms of CPU time and memory) having one file per e-mail is less than sub-optimal especially when a large amount of users just downloads the whole mailbox using POP3 (not to mention backing up Maildirs). It seems to me that a database like Postgres or MySQL would be the best bet. That's a matter of opinion. Moving mail storage to a database would probably be the last thing I would ever do (I'm not saying it's not the right thing for some people. I'm just not one of them). I'm using mysql for storing the users database but that’s another story. Adding a database is one additional level of complexity. One more program to govern. In my opinion it's nice to know that as long as the disk is readable nothing can go completely wrong. I have to jump in here and go a bit tangential by saying there are databases and want-to-be's. The database in my case would be roughly 400 GB holding some 60 million records. Fair sized but not really big. Just imagine if one single byte got written to the wrong place. Power outage, OS crash, software bug or whatever could easily result in this (I regularly experience mysql tables that crash on their own from heavy use). Having to run a repair on a table of that size whilst all users are eager to get to their data must be a nightmare of proportions. There is the difference between an enterprise database and MySQL. Yes, yes, yes lots of /enterprises/ run applications that use MySQL but most of those apps have throw away data or they are not using the free version of MySQL. Just imagine backing the thing up, exporting 60.000.000 SQL queries. Not to say importing them again if something should go really wrong. Actually I'n not even sure it would be faster. When the index files grow to several gigabytes they kind of loose their purpose. There are many businesses backing up way-more data than that and it it isn't 60,000,000 queries -- it is one command. But if you use serious hardware backing up isn't really needed. RAID, redundant/hot-swap servers, etc. make backing up /extra redundancy/. :-) And I bring this up because Archiveopteryx http://www.archiveopteryx.org/ uses a database - PostgreSQL. Rod -- Maildir is very resilient to various errors. It is virtually impossible to corrupt a maildir (at least I've never experienced anything). Also you can backup up the thing without worrying about anything accessing it at the same time. Mbox less so but still a lot better than having one huge database. Dbox would be the ultimate compromise between crash resilience and a low number of files (not to mention the enormous potential for speed gains). Regards, Mikkel
Re: [Dovecot] Backing Up
[EMAIL PROTECTED] wrote: Just imagine backing the thing up, exporting 60.000.000 SQL queries. Not to say importing them again if something should go really wrong. Actually I'n not even sure it would be faster. When the index files grow to several gigabytes they kind of loose their purpose. There are many businesses backing up way-more data than that and it it isn't 60,000,000 queries -- it is one command. But if you use serious hardware backing up isn't really needed. RAID, redundant/hot-swap servers, etc. make backing up /extra redundancy/. :-) Why make things complicated and expensive when you can make them cheap and simple? Anything is possible if you wanna pay for it (in terms of hardware, administration and licenses). I have focused primarily on making it as simple as possible. And while running a 400 GB with 60.000.000 records database isn't impossible it would be if it were to run on the same hardware that now comprises the system. Roughly 1000 IOPS is plenty to handle all mail operations. I seriously doubt that it would be enough to even supply one lookup a second on that huge db (and even less over NFS as is now being used). And I assume that a hundreds of lookups a second would be required to handle the load. So it would require a lot more resources and still give nothing but trouble (risk of crashed database and backup issues that now aren't there). By the way data is stored in a SAN it needs to be backed up. 500 GB SATA disks takes a day to synchronize if one breaks down and we can't really take that chance (Yes I will eventually move the data to smaller 15.000 RPM disks but there is no need to pay for them before its necessary). Also there is the risk of data being deleted by a mistake, hacker attacks or software malfunctioning. But we really are moving off-topic here. Regards, Mikkel
[Dovecot] Backing Up
What is the best way to do a (server-side) backup of all mail in a user's mail? I don't think I'm doing anything weird as far as configs go; here's dovecot -n if it helps: # 1.1.4: /etc/dovecot/dovecot.conf protocols: imaps listen: *, [::] ssl_cert_file: /etc/ssl/dovecot/cert.pem ssl_key_file: /etc/ssl/dovecot/key.pem login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login mail_location: maildir:~/.maildir auth default: mechanisms: plain login passdb: driver: pam args: * userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix
Re: [Dovecot] Backing Up
on 10-29-2008 12:25 PM Neil spake the following: What is the best way to do a (server-side) backup of all mail in a user's mail? I don't think I'm doing anything weird as far as configs go; here's dovecot -n if it helps: # 1.1.4: /etc/dovecot/dovecot.conf protocols: imaps listen: *, [::] ssl_cert_file: /etc/ssl/dovecot/cert.pem ssl_key_file: /etc/ssl/dovecot/key.pem login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login mail_location: maildir:~/.maildir auth default: mechanisms: plain login passdb: driver: pam args: * userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Backing Up
On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave -- Dave McGuire Port Charlotte, FL
Re: [Dovecot] Backing Up
on 10-29-2008 12:47 PM Dave McGuire spake the following: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave Mbox syncs fairly quickly also. Rsync is very good at working with large text files like mbox, even if users purge stuff from the middle. But since he did show Maildir was in use, I left out also backing up /var/spool/mail/* for the inboxes on a default mbox installation. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Backing Up
On Wednesday 29 of October 2008, Dave McGuire wrote: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. -Dave -- Arkadiusz MiśkiewiczPLD/Linux Team arekm / maven.plhttp://ftp.pld-linux.org/
Re: [Dovecot] Backing Up
On 29 Oct 2008, at 16:02, Scott Silva wrote: on 10-29-2008 12:47 PM Dave McGuire spake the following: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave Mbox syncs fairly quickly also. Rsync is very good at working with large text files like mbox, even if users purge stuff from the middle. But since he did show Maildir was in use, I left out also backing up /var/spool/mail/* for the inboxes on a default mbox installation. Yeah, the maildir line was mostly why I put the dovecot -n there. Do you think rsync will be easier on my servers than tarball/bzip2/scp ? Thanks for the help, -Neil.
Re: [Dovecot] Backing Up
On Oct 29, 2008, at 5:32 PM, Arkadiusz Miskiewicz wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. Due to connection turnaround latency, I assume? (I've never looked at the rsync protocol) If that's the case, then I stand very much corrected, thank you. I was going from the same logic regarding mbox vs. maildir in the context of backups. One new message delivered and a 400MB mail spool gets backed up again.. -Dave -- Dave McGuire Port Charlotte, FL
Re: [Dovecot] Backing Up
on 10-29-2008 2:46 PM Neil spake the following: On 29 Oct 2008, at 16:02, Scott Silva wrote: on 10-29-2008 12:47 PM Dave McGuire spake the following: On Oct 29, 2008, at 3:42 PM, Scott Silva wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! -Dave Mbox syncs fairly quickly also. Rsync is very good at working with large text files like mbox, even if users purge stuff from the middle. But since he did show Maildir was in use, I left out also backing up /var/spool/mail/* for the inboxes on a default mbox installation. Yeah, the maildir line was mostly why I put the dovecot -n there. Do you think rsync will be easier on my servers than tarball/bzip2/scp ? Thanks for the help, -Neil. Rsync will use more memory on large filesystems, but it is usually lighter in CPU, network, and IO time. But tar gives you multiple backups. To achieve that with rsync you need the rbackup script or rsnapshot. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Backing Up
on 10-29-2008 3:18 PM Dave McGuire spake the following: On Oct 29, 2008, at 5:32 PM, Arkadiusz Miskiewicz wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. Due to connection turnaround latency, I assume? (I've never looked at the rsync protocol) If that's the case, then I stand very much corrected, thank you. I was going from the same logic regarding mbox vs. maildir in the context of backups. One new message delivered and a 400MB mail spool gets backed up again.. -Dave Rsync adds some latency as it indexes and compares files on both ends. Obviously it would take more time to compare 40,000 1K files then 1000 40K files even though the data size is similar. It would still be better than tar/bzip/scp which has to compress everything and transfer the lot every time. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Backing Up
Scott Silva wrote, On 10/30/2008 12:34 AM: on 10-29-2008 3:18 PM Dave McGuire spake the following: On Oct 29, 2008, at 5:32 PM, Arkadiusz Miskiewicz wrote: What is the best way to do a (server-side) backup of all mail in a user's mail? I usually just rsync the /home directories to another server. The inital sync can take a while, but it gets faster after there is a base to work from. ...and it's much less painful if you're using maildir instead of mbox! Not for rsyncing. Tons of small files means much slower rsync. Due to connection turnaround latency, I assume? (I've never looked at the rsync protocol) If that's the case, then I stand very much corrected, thank you. I was going from the same logic regarding mbox vs. maildir in the context of backups. One new message delivered and a 400MB mail spool gets backed up again.. -Dave Rsync adds some latency as it indexes and compares files on both ends. Obviously it would take more time to compare 40,000 1K files then 1000 40K files even though the data size is similar. It would still be better than tar/bzip/scp which has to compress everything and transfer the lot every time. Maildirsync it an Online synchronizer for Maildir-format mailboxes See http://hacks.dlux.hu/maildirsync/ Sot.
[Dovecot] Backing up mail?
My mail server is running on a Fedora FC5 box. I need to move the mail server to another computer and so I need to back up my user's mail and move it. I have found mail in /var/spoo/mail/[users] and in their home directories there is a 'mail' directory. All users have an 'Inbox' file that is empty and I think the one IMAP user has a 'Trash' file that is huge! Are those the only files I need to copy or are there others lurking out there that I don't know about? Thanks, -- Knute Johnson Molon Labe...
