Re: [Dovecot] Please advise on very fast search

2011-11-17 Thread Stan Hoeppner
On 11/15/2011 1:02 PM, Timo Sirainen wrote: On Tue, 2011-11-15 at 12:26 -0600, Stan Hoeppner wrote: This is why I recommended mbox in the first place. If your only writes to these mailbox files are appends of new messages, mbox is the best format by far. It's faster at appending than any

Re: [Dovecot] Please advise on very fast search

2011-11-16 Thread Stan Hoeppner
On 11/16/2011 12:15 AM, Alexander Chekalin wrote: Hello, Stan, This is why I recommended mbox in the first place. If your only writes to these mailbox files are appends of new messages, mbox is the best format by far. It's faster at appending than any other format, and it's faster for

Re: [Dovecot] Please advise on very fast search

2011-11-15 Thread Stan Hoeppner
On 11/14/2011 3:16 PM, Alexander Chekalin wrote: Locking issues on mbox is the reason for my long-lasting love affair with maildir, Same reason most others fell in love with it. Many now want to divorce maildir, as the cost of the storage to maintain acceptable performance is now too high.

Re: [Dovecot] Please advise on very fast search

2011-11-15 Thread Timo Sirainen
On Tue, 2011-11-15 at 12:26 -0600, Stan Hoeppner wrote: This is why I recommended mbox in the first place. If your only writes to these mailbox files are appends of new messages, mbox is the best format by far. It's faster at appending than any other format, and it's faster for searching

Re: [Dovecot] Please advise on very fast search

2011-11-14 Thread Alexander Chekalin
Timo, Stan, I've just tested mdbox and find it pretty nice for me, but now I got some questions for you: 1. mdbox uses 'a lot' files (m.1, m.2 ... etc), and the default size if 2Mb. Looks like not even every message can fit into such storage container volume (nowadays we used to see

Re: [Dovecot] Please advise on very fast search

2011-11-14 Thread Alexander Chekalin
Locking issues on mbox is the reason for my long-lasting love affair with maildir, and it's lasts long years. Ok, the life's lessons are like this, learn something and move on with it ;) even if it's new old thing. Thank you for pointing that! What I was doubt about is default rotate size of

Re: [Dovecot] Please advise on very fast search

2011-11-14 Thread Timo Sirainen
On 14.11.2011, at 16.35, Alexander Chekalin wrote: 1. mdbox uses 'a lot' files (m.1, m.2 ... etc), and the default size if 2Mb. Looks like not even every message can fit into such storage container volume (nowadays we used to see messages of 20Mb and even more). The messages are never split

Re: [Dovecot] Please advise on very fast search

2011-11-10 Thread Stan Hoeppner
On 11/9/2011 11:35 PM, Alexander Chekalin wrote: Hello, Stan, in fact the only thing I miss even with my current scheme is permanent ID assigned to the message so I can easily find it despite the IMAP mailbox it is now (so if someone moved the message from one mailbox/folder to another, the

Re: [Dovecot] Please advise on very fast search

2011-11-10 Thread Timo Sirainen
On 10.11.2011, at 6.37, Alexander Chekalin wrote: Are there any ways I can search or parse mboxes or mdboxes not directly and not with IMAP (I'm afraid it slooow in dump parsing)? See doveadm fetch / doveadm search. in fact the only thing I miss even with my current scheme is permanent ID

[Dovecot] Please advise on very fast search

2011-11-09 Thread Alexander Chekalin
Hello, I try to create some kind of mail backup system. What I need is system that will store mail for the whole domain, and allow me to restore messages from/to specified email at that domain. The scheme is pretty simple: on our main mail server the SMTP server itself has a rule to send a

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Robert Schetterer
Am 09.11.2011 14:57, schrieb Alexander Chekalin: Hello, I try to create some kind of mail backup system. What I need is system that will store mail for the whole domain, and allow me to restore messages from/to specified email at that domain. The scheme is pretty simple: on our main mail

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Timo Sirainen
On Wed, 2011-11-09 at 16:57 +0300, Alexander Chekalin wrote: The problem is that when my archive become big (several years), it appears to be painful to find specified message(s). When someone suddenly needs to find his/her old message, it is mostly guesses like 'I think the message was

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Alexander Chekalin
Thanks, Robert, will take a look at. What I'm afraid for is how database storage should be planned (storage, CPU, RAM, scaling when will be over-filled). When dealing with files (I'm using maildir), it is much easy to understand and to fix just about everything. Adding database involves tune

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Timo Sirainen
On Wed, 2011-11-09 at 19:16 +0300, Alexander Chekalin wrote: Will also try to use full text search, but afraid of index size (and I need no search on body, just on headers). It wouldn't be difficult to patch Dovecot to skip indexing message bodies. Of course then you'd need to remember to keep

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Stan Hoeppner
On 11/9/2011 10:16 AM, Alexander Chekalin wrote: Thanks, Robert, will take a look at. What I'm afraid for is how database storage should be planned (storage, CPU, RAM, scaling when will be over-filled). When dealing with files (I'm using maildir) Bingo.^^^ Maildir is very likely

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Stan Hoeppner
On 11/9/2011 10:40 AM, Timo Sirainen wrote: On Wed, 2011-11-09 at 19:16 +0300, Alexander Chekalin wrote: Will also try to use full text search, but afraid of index size (and I need no search on body, just on headers). It wouldn't be difficult to patch Dovecot to skip indexing message

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Alexander Chekalin
Oh, that's the point to consider. But I must confess I'm in love with Maildir for maybe 10 years for that simple fact I can do anything with each and every single message even on disk (=much faster than via IMAP). If I would deal with mbox directly I'd need to parse huge files, b. Are

Re: [Dovecot] Please advise on very fast search

2011-11-09 Thread Alexander Chekalin
Hello, Stan, in fact the only thing I miss even with my current scheme is permanent ID assigned to the message so I can easily find it despite the IMAP mailbox it is now (so if someone moved the message from one mailbox/folder to another, the ID allows to retrieve it fast anyway). You see,