Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Mikkel
Timo Sirainen skrev: On Wed, 2009-10-14 at 23:59 +0200, Mikkel wrote: In case of mdbox wouldn't you have the very same problem since larger files may be fragmented all over the disk just like many small files in a directory might? I guess this depends on filesystem. But the files would

Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Timo Sirainen
On Thu, 2009-10-15 at 10:55 +0200, Mikkel wrote: In any case if there are expunged messages, files containing them would be recreated (nightly or something). That'll unfragment the files. It would be nice if this recreation interval (nightly, weekly, monthly?) was made tunable. It's

Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Mikkel
Timo Sirainen skrev: On Thu, 2009-10-15 at 10:55 +0200, Mikkel wrote: Some users would have mailboxes a several hundred megabytes and having to recreate thousands of these every night because of a single mail getting expunged a day could result in a huge performance hit. It doesn't matter

Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Timo Sirainen
On Fri, 2009-10-16 at 00:24 +0200, Mikkel wrote: It doesn't matter what the user's full mailbox size is. It matters how many of these max. ~2 MB dbox files have expunged messages. Typically users would expunge only new mails, so typically there would be only a single file that needs to

Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Charles Marcus
On 10/15/2009, Timo Sirainen (t...@iki.fi) wrote: The default is 2 MB, and whoever sets it larger deserves what they get (for better or worse). Just for clarification... what happens when a single message is larger than 2MB? Is it stored just as a single dbox file all by itself?

Re: [Dovecot] Questions regarding dbox migration

2009-10-15 Thread Timo Sirainen
On Thu, 2009-10-15 at 20:16 -0400, Charles Marcus wrote: On 10/15/2009, Timo Sirainen (t...@iki.fi) wrote: The default is 2 MB, and whoever sets it larger deserves what they get (for better or worse). Just for clarification... what happens when a single message is larger than 2MB? Is it

[Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us. Now I wonder what may be the best way of doing so. I'm considering some sort of intermediate migration where the existing Maildir users are changed to single-dbox and then

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Oct 14, 2009, at 7:03 AM, Mikkel wrote: It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us. BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Charles Marcus
On 10/14/2009, Timo Sirainen (t...@iki.fi) wrote: I'm trying to get v2.0.0 out pretty quickly though. v2.0.beta1 should hopefully be out in less than a month. Great news on how fast 2.0 is coming along! I finally got the go ahead to convert my biggest client, but he wants to hold off (political

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Charles Marcus
On 10/14/2009, Timo Sirainen (t...@iki.fi) wrote: BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how much. What are the downsides? Also, I'm guessing maybe there are certain conditions where you definitely don't want

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 12:18 -0400, Charles Marcus wrote: The main problem with it is actually how to make it enough backwards compatible that everyone won't start hating me. I honestly don't see that ever happening Timo - and please, don't go down the same road Microsoft did with Windows.

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 12:18 -0400, Charles Marcus wrote: On 10/14/2009, Timo Sirainen (t...@iki.fi) wrote: BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how much. What are the downsides? Also, I'm guessing maybe

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Charles Marcus
On 10/14/2009 12:24 PM, Timo Sirainen wrote: BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how much. What are the downsides? Also, I'm guessing maybe there are certain conditions where you definitely don't want to

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen skrev: On Oct 14, 2009, at 7:03 AM, Mikkel wrote: It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us. BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen skrev: On Oct 14, 2009, at 7:03 AM, Mikkel wrote: Now the big question is whether multi-dbox and single-dbox are compatible formats. Kind of, but not practically. If a Maildir-dbox migration is made on a system running dovecot v. 1.1, would it then be trivial later changing

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 21:14 +0200, Mikkel wrote: It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us. BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 21:28 +0200, Mikkel wrote: From your comments it appears like dbox and mdbox are quite different in many ways. Is mdbox going to replace dbox completely or are you expecting to keep both formats? My point is: what's going to be the difference between dbox and mbox

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen skrev: The main difference is that mdbox needs to lock files when saving messages. That's not especially nice with NFS. Single-dbox currently locks index files, but it can be made entirely lockless eventually. There are also some other differences like: - in mdbox all messages

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 23:04 +0200, Mikkel wrote: So basically you prefer mdbox but are maintaining dbox because of its almost lockless design which is better for NFS users? Do you consider it to be viable having two different dbox formats or are you planning to keep only one of them in a

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen skrev: On Wed, 2009-10-14 at 21:14 +0200, Mikkel wrote: It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us. BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 23:18 +0200, Mikkel wrote: I don't think I've tried that one. Earlier on I experimented with fsync_disable=yes (which made a huge difference by the way) but that was before I started using mail_nfs_storage=yes and mail_nfs_index=yes I would like to try using

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen wrote: And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors. Well, it does complain once in a while but it has never resulted in data being lost in any way. But I guess your point is that this might

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 23:41 +0200, Mikkel wrote: Timo Sirainen wrote: And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors. Well, it does complain once in a while but it has never resulted in data being

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen wrote: On Wed, 2009-10-14 at 23:41 +0200, Mikkel wrote: Timo Sirainen wrote: And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors. Well, it does complain once in a while but it has never resulted in

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 23:52 +0200, Mikkel wrote: Timo Sirainen wrote: On Wed, 2009-10-14 at 23:41 +0200, Mikkel wrote: Timo Sirainen wrote: And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors. Well, it does

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen skrev: On Wed, 2009-10-14 at 23:04 +0200, Mikkel wrote: So basically you prefer mdbox but are maintaining dbox because of its almost lockless design which is better for NFS users? Do you consider it to be viable having two different dbox formats or are you planning to keep only

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Mikkel
Timo Sirainen wrote: On Wed, 2009-10-14 at 23:52 +0200, Mikkel wrote: But it should be able to heal itself using the backup files in version 2.0, right? That's the theory anyway. :) How often are they created anyway? Whenever dovecot.index file would normally get recreated, the old one

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread Timo Sirainen
On Wed, 2009-10-14 at 23:59 +0200, Mikkel wrote: I'm planning on keeping both of them. And it's not necessarily only because of NFS users. Multi-dbox was done mainly because filesystems suck (mailbox gets fragmented all around the disk). Maybe if filesystems in future suck less,

Re: [Dovecot] Questions regarding dbox migration

2009-10-14 Thread dovecot
At 12:18 PM 10/14/2009, Charles Marcus wrote: Not that you need any help, but I would suggest to just focus on making the conversion tools (mbox/maildir (m)dbox, old config new config, etc) rock solid, and document any incompatibilities really well. It'd be nice to think you were kidding,