Re: [Dovecot] mdbox - healthy rotation size vs default
On 2013-08-24 4:47 AM, Michael Grimm trash...@odo.in-berlin.de wrote: Don't ask me why I did chose 100m, I cannot remember;-) Ok, if one of such mdbox files will become corrupt, I will loose a lot of mail, but on the other hand I am running two dovecot servers in parallel (replicator/dsync) and I do take hourly snapshots (ZFS) of my mail storage file system as well. Well, if they are stored on ZFS, I guess the chances of corruption are extremely minimal (much less than for other filesystems)... I'm curious, is this on FreeBSD? Linux? I'm interested in details, as I'd love to be able to use ZFS on my gentoo linux box without having to enable modules... -- Best regards, */Charles/*
Re: [Dovecot] mdbox - healthy rotation size vs default
On 25.08.2013, at 15:37, Jan-Frode Myklebust janfr...@tanso.net wrote: On Sat, Aug 24, 2013 at 10:47:56AM +0200, Michael Grimm wrote: I am running mdbox_rotate_size = 100m for approx. a year now on a small server (a handful of users, only). All mailboxes are around 1G each with a lot of attachments. I never had an issue so far. How much space are your mdboxes using, compared to to quota usage? Sorry, but I do not understand your question. I.e. how much space is wasted on deleted messages? (not sure this will be easy to measure, because of compression..) True, it is hard to answer ;-) As a very rough estimate I do estimate a 5% waste of space regarding deleted messages. But, my handful users are very disciplined in purging their deleted messages on a regular basis (I told them to do), and thus my regular doveadm purge -A runs will reduce that amount of wasted disk space to a minimum. Not very helpful, I know, Michael
Re: [Dovecot] mdbox - healthy rotation size vs default
On 26.08.2013, at 20:35, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-24 4:47 AM, Michael Grimm trash...@odo.in-berlin.de wrote: Don't ask me why I did chose 100m, I cannot remember;-) Ok, if one of such mdbox files will become corrupt, I will loose a lot of mail, but on the other hand I am running two dovecot servers in parallel (replicator/dsync) and I do take hourly snapshots (ZFS) of my mail storage file system as well. Well, if they are stored on ZFS, I guess the chances of corruption are extremely minimal (much less than for other filesystems)... Haven't had any file system corruption for a very long time now, even before switching to ZFS. I'm curious, is this on FreeBSD? Yes I migrated my servers to FreeBSD some years ago, and I am using ZFS for approx. two years now. Linux? I'm interested in details, as I'd love to be able to use ZFS on my gentoo linux box without having to enable modules... Sorry, but I never used ZFS with Linux. But, ZFS and snapshots as such are pretty awesome and helped me a lot in the past when it comes to recovering from human mistakes ;-) Regrads, Michael
Re: [Dovecot] mdbox - healthy rotation size vs default
On 2013-08-26 3:05 PM, Michael Grimm trash...@odo.in-berlin.de wrote: On 26.08.2013, at 20:35, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-24 4:47 AM, Michael Grimm trash...@odo.in-berlin.de wrote: Don't ask me why I did chose 100m, I cannot remember;-) Ok, if one of such mdbox files will become corrupt, I will loose a lot of mail, but on the other hand I am running two dovecot servers in parallel (replicator/dsync) and I do take hourly snapshots (ZFS) of my mail storage file system as well. Well, if they are stored on ZFS, I guess the chances of corruption are extremely minimal (much less than for other filesystems)... Haven't had any file system corruption for a very long time now, even before switching to ZFS. I know, me neither (knock on wood), which is why I put the 'extremely' in there... ;) I'm curious, is this on FreeBSD? Yes I migrated my servers to FreeBSD some years ago, and I am using ZFS for approx. two years now. Linux? I'm interested in details, as I'd love to be able to use ZFS on my gentoo linux box without having to enable modules... Sorry, but I never used ZFS with Linux. But, ZFS and snapshots as such are pretty awesome and helped me a lot in the past when it comes to recovering from human mistakes ;-) Heh - that (and the resistance to hidden/silent filesystem corruption) is the main reason I'm interested in using it. :)
Re: [Dovecot] mdbox - healthy rotation size vs default
On 2013-08-26 2:58 PM, Michael Grimm trash...@odo.in-berlin.de wrote: As a very rough estimate I do estimate a 5% waste of space regarding deleted messages. But, my handful users are very disciplined in purging their deleted messages on a regular basis (I told them to do), and thus my regular doveadm purge -A runs will reduce that amount of wasted disk space to a minimum. Not very helpful, I know, Are you sure about that? There was a thread a while back (I recently posted a response to it) about this, and it sounded like the mdbox files would *never* be 'compacted' (reduced in size from deleted messages)... my reply was on 8/23, thread titled Dovecot never release preallocated space in mdbox'... -- Best regards, */Charles/*
Re: [Dovecot] mdbox - healthy rotation size vs default
On 2013-08-26 3:23 PM, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-26 2:58 PM, Michael Grimm trash...@odo.in-berlin.de wrote: As a very rough estimate I do estimate a 5% waste of space regarding deleted messages. But, my handful users are very disciplined in purging their deleted messages on a regular basis (I told them to do), and thus my regular doveadm purge -A runs will reduce that amount of wasted disk space to a minimum. Not very helpful, I know, Are you sure about that? There was a thread a while back (I recently posted a response to it) about this, and it sounded like the mdbox files would *never* be 'compacted' (reduced in size from deleted messages)... my reply was on 8/23, thread titled Dovecot never release preallocated space in mdbox'... Ooops, sorry, it was about *automatically* compacting them... I think... **
Re: [Dovecot] mdbox - healthy rotation size vs default
On 26.08.2013, at 21:23, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-26 2:58 PM, Michael Grimm trash...@odo.in-berlin.de wrote: As a very rough estimate I do estimate a 5% waste of space regarding deleted messages. But, my handful users are very disciplined in purging their deleted messages on a regular basis (I told them to do), and thus my regular doveadm purge -A runs will reduce that amount of wasted disk space to a minimum. Not very helpful, I know, Are you sure about that? There was a thread a while back (I recently posted a response to it) about this, and it sounded like the mdbox files would *never* be 'compacted' (reduced in size from deleted messages)... my reply was on 8/23, thread titled Dovecot never release preallocated space in mdbox'... I must have missed that thread, sorry. My observations are as follows: 1) if I delete mails in my mail client, mdbox files will not become reduced accordingly 2) if I do run something in my client like remove all deleted mails from my account (purged in client), mdbox files will not become reduced accordingly 3) if I do run doveadm purge -A on the server, mdbox files will become modified, see an example of a purge run a couple of minutes ago: before (all my mail, ~800 mails purged in client): -rw--- 1 vmail dovecot 104856511 Aug 14 20:20 /var/mail/mike/storage/m.77 -rw--- 1 vmail dovecot 104769054 Aug 25 03:14 /var/mail/mike/storage/m.89 -rw--- 1 vmail dovecot 104848809 Aug 24 18:33 /var/mail/mike/storage/m.90 -rw--- 1 vmail dovecot 24762837 Aug 26 21:26 /var/mail/mike/storage/m.91 after (all my mail, after doveadm purge -A): -rw--- 1 vmail dovecot 104856511 Aug 14 20:20 /var/mail/mike/storage/m.77 -rw--- 1 vmail dovecot 104803218 Aug 26 21:26 /var/mail/mike/storage/m.92 -rw--- 1 vmail dovecot 104802874 Aug 26 21:26 /var/mail/mike/storage/m.93 -rw--- 1 vmail dovecot 21580496 Aug 26 21:26 /var/mail/mike/storage/m.94 Thus, from my point of view one needs to run doveadm purge -A on a regular basis *and* educate users to purge deleted mails in their clients on a regular basis as well. (I hope I didn't misunderstand you right from the beginning.) Regards, Michael
Re: [Dovecot] mdbox - healthy rotation size vs default
On Mon, Aug 26, 2013 at 03:31:20PM -0400, Charles Marcus wrote: On 2013-08-26 2:58 PM, Michael Grimm trash...@odo.in-berlin.de wrote: As a very rough estimate I do estimate a 5% waste of space regarding deleted messages. But, my handful users are very disciplined in purging their deleted messages on a regular basis (I told them to do), and thus my regular doveadm purge -A runs will reduce that amount of wasted disk space to a minimum. Are you sure about that? There was a thread a while back (I recently posted a response to it) about this, and it sounded like the mdbox files would *never* be 'compacted' (reduced in size from deleted messages)... my reply was on 8/23, thread titled Dovecot never release preallocated space in mdbox'... Ooops, sorry, it was about *automatically* compacting them... I think... And Timo seemed to reply that hole punching was something doveadm purge could conceivably do, but doesn't do at the moment. Timo, could you please clearify a bit here? Does non-preallocated (mdbox_preallocate_space=no) m-files get hole punched (or space re-used for new messages) after running doveadm purge? Or can we end up with a huge $mdbox_rotate_size size m-file, with only a single small message remaining after all other messages has been purged? -jf
Re: [Dovecot] mdbox - healthy rotation size vs default
Another intesting thing for this thread: if you set a very high value for mdbox rotate settings, your incremental backups will be awful. If you have hundreds of messages in a mdbox and you doveadm purge one of them, the full .m file must be copied in the incremental / diferential backup. I use 10 MB+zlib for main storage and 250 MB+bzip2 for alternate storage. Regards Javier
Re: [Dovecot] mdbox - healthy rotation size vs default
On 26.08.2013, at 21:59, Javier de Miguel RodrÃguez javierdemiguel-...@us.es wrote: If you have hundreds of messages in a mdbox and you doveadm purge one of them, the full .m file must be copied in the incremental / diferential backup. Good point! I won't suffer from that, but those with thousands of users will suffer for sure, see my example mailed before. Three mdbox files became deleted and copied to w ones. Regards, Michael
Re: [Dovecot] mdbox - healthy rotation size vs default
On 26.08.2013, at 22:20, Michael Grimm trash...@odo.in-berlin.de wrote: Three mdbox files became deleted and copied to w ones. s/to w ones/to new ones/ Sorry.
Re: [Dovecot] mdbox - healthy rotation size vs default
On Sat, Aug 24, 2013 at 10:47:56AM +0200, Michael Grimm wrote: I am running mdbox_rotate_size = 100m for approx. a year now on a small server (a handful of users, only). All mailboxes are around 1G each with a lot of attachments. I never had an issue so far. How much space are your mdboxes using, compared to to quota usage? I.e. how much space is wasted on deleted messages? (not sure this will be easy to measure, because of compression..) -jf
Re: [Dovecot] mdbox - healthy rotation size vs default
On 23.08.2013, at 14:17, Charles Marcus cmar...@media-brokers.com wrote: On 2013-08-22 9:57 PM, Stan Hoeppner s...@hardwarefreak.com wrote: I'd recommend mdbox as well, with a healthy rotation size. The larger files won't increase IMAP performance substantially but they can make backup significantly quicker. I'm considering migrating to mdbox... wondering what you consider 'healthy' rotation size. I generally try to avoid changing defaults whenever possible, [...] I am running mdbox_rotate_size = 100m for approx. a year now on a small server (a handful of users, only). All mailboxes are around 1G each with a lot of attachments. I never had an issue so far. Don't ask me why I did chose 100m, I cannot remember ;-) Ok, if one of such mdbox files will become corrupt, I will loose a lot of mail, but on the other hand I am running two dovecot servers in parallel (replicator/dsync) and I do take hourly snapshots (ZFS) of my mail storage file system as well. Regards, Michael
[Dovecot] mdbox - healthy rotation size vs default - WAS Re: Dovecot tuning for GFS2
On 2013-08-22 9:57 PM, Stan Hoeppner s...@hardwarefreak.com wrote: On 8/21/2013 4:07 PM, Jan-Frode Myklebust wrote: I would strongly suggest using mdbox instead. AFAIK clusterfs' aren't I'd recommend mdbox as well, with a healthy rotation size. The larger files won't increase IMAP performance substantially but they can make backup significantly quicker. I'm considering migrating to mdbox... wondering what you consider 'healthy' rotation size. I generally try to avoid changing defaults whenever possible, so, do you consider the default size of 2MB too small? I guess though that it depends on usage. Since we get a decent number of large attachments, maybe that is a good reason to bump it up? Thanks, -- Best regards, */Charles/*
Re: [Dovecot] mdbox - healthy rotation size vs default - WAS Re: Dovecot tuning for GFS2
On 8/23/2013 7:17 AM, Charles Marcus wrote: On 2013-08-22 9:57 PM, Stan Hoeppner s...@hardwarefreak.com wrote: On 8/21/2013 4:07 PM, Jan-Frode Myklebust wrote: I would strongly suggest using mdbox instead. AFAIK clusterfs' aren't I'd recommend mdbox as well, with a healthy rotation size. The larger files won't increase IMAP performance substantially but they can make backup significantly quicker. I'm considering migrating to mdbox... wondering what you consider 'healthy' rotation size. I generally try to avoid changing defaults whenever possible, so, do you consider the default size of 2MB too small? I guess though that it depends on usage. Since we get a decent number of large attachments, maybe that is a good reason to bump it up? It's probably better to err large than to err small. Analyze your current maildir directories and make a distribution graph of file sizes. That should give you a good idea of what your rotation size should be. -- Stan
