Re: [Dovecot] Please advise on very fast search
Hello, Stan, in fact the only thing I miss even with my current scheme is permanent ID assigned to the message so I can easily find it despite the IMAP mailbox it is now (so if someone moved the message from one mailbox/folder to another, the ID allows to retrieve it fast anyway). You see, what I need is not only find message from|to someone on specified date, I also sometime need to restore that message back to user's original box. As far our mailserver and backup-mailserver are different machines, it is a bit tricky to copy messages between it fast enough. Say, if I need to find and restore all mails from u...@domain.com within 2009 year, and search yields in some 1000's of messages, then use IMAP to copy it over to another server takes some time - and if you consider both search time and restore/copy time the whole process may take "ages". With maildir I can rsync/scp needed files to another host and that's fast way - that's why I stick with maildir. FTS in my case can help (I can search for u...@domain.com, for example), but it also return messages that contains such a string in message body (and that takes index space, too), so I'll need to filter it later, but surely it'll be faster than checking every message in the archive. Yours, Alexander Maildir is very likely a hug factor in your current slow search time. With a maildir search, every mail file must be opened and searched. How many total mail files are opened for each of your searches? Thousands? Tens of thousands? Maildir causes a massive disk IO bottleneck when searching so many files. Run iostat the next time you do one of these searches, and look at the %iowait value. It will likely be very high. If it is, this confirms maildir is a big part of the problem. mbox, and mdbox, would be many many times faster than maildir WRT searching as the total number of files is lower by orders of magnitude. Switching from maildir to mbox/mdbox shifts the workload burden from the disk subsystem to the processor/memory. And I'm sure as with everyone else on the planet today, you have massive spare CPU cycles, but extremely limited spindle throughput. And as Timo suggested, using one of the indexing search plugins would be much faster yet, as long as you keep the indexes updated. -- С уважением, Александр Чекалин Лазурит Калининград +7 909 799 2549 acheka...@lazurit.com
Re: [Dovecot] Please advise on very fast search
Oh, that's the point to consider. But I must confess I'm in love with Maildir for maybe 10 years for that simple fact I can do anything with each and every single message even on disk (=much faster than via IMAP). If I would deal with mbox directly I'd need to parse huge files, b. Are there any ways I can search or parse mboxes or mdboxes not directly and not with IMAP (I'm afraid it slooow in dump parsing)? 10.11.2011, в 3:42, Stan Hoeppner написал(а): > On 11/9/2011 10:16 AM, Alexander Chekalin wrote: >> Thanks, Robert, >> >> will take a look at. >> >> What I'm afraid for is how database storage should be planned (storage, >> CPU, RAM, scaling when will be over-filled). When dealing with files >> (I'm using maildir) > > Bingo.^^^ > > Maildir is very likely a hug factor in your current slow search time. > With a maildir search, every mail file must be opened and searched. How > many total mail files are opened for each of your searches? Thousands? > Tens of thousands? Maildir causes a massive disk IO bottleneck when > searching so many files. Run iostat the next time you do one of these > searches, and look at the %iowait value. It will likely be very high. > If it is, this confirms maildir is a big part of the problem. > > mbox, and mdbox, would be many many times faster than maildir WRT > searching as the total number of files is lower by orders of magnitude. > Switching from maildir to mbox/mdbox shifts the workload burden from > the disk subsystem to the processor/memory. And I'm sure as with > everyone else on the planet today, you have massive spare CPU cycles, > but extremely limited spindle throughput. > > And as Timo suggested, using one of the indexing search plugins would be > much faster yet, as long as you keep the indexes updated. > > -- > Stan
Re: [Dovecot] patching dovecot for sieve/managesieve support, centos 5.6?
- Forwarded Message - From: Scott Lewis To: "dovecot@dovecot.org" Sent: Thursday, 3 November 2011 4:31 PM Subject: patching dovecot for sieve/managesieve support, centos 5.6? Hi all, I am having real trouble when attempting to patch dovecot 1.2 to include the Pidgeonhole sieve support on my CentOS 5.6 x64 mail server. I am relatively new to the programming side of linux, but I am not having a lot of luck when trying to get this thing to compile. Here's what happens: [root@mail ~]# whereis dovecot dovecot: /usr/sbin/dovecot /etc/dovecot.conf /usr/lib/dovecot /usr/libexec/dovecot /usr/share/man/man8/dovecot.8.gz [root@mail dovecot-1.2-sieve-0.1.19]# ./configure --with-dovecot=/usr/lib/dovecot ... checking whether to build static libraries... yes dovecot-config not found from /usr/lib/dovecot, use --with-dovecot=PATH to give path to compiled Dovecot sources or to a directory with the installed dovecot-config file. configure: error: dovecot-config not found -- I get this message regardless of whether I set --with-dovecot as /usr/sbin/dovecot, or /etc, or /usr/libexec/dovecot. I have SquirrelMail 1.4.22 running, and the avelsieve front-end seems happy enough. when I visit https://mail.mydomain.com/src/configtest.php, I get: Avelsieve plugin details: backend = ManageSieve ERROR: I could not determine the capabilities for Sieve Mail Filtering. Perhaps connectivity with ManageSieve server (if backend=Managesieve) is bad? thanks in advance!
Re: [Dovecot] Please advise on very fast search
On 11/9/2011 10:40 AM, Timo Sirainen wrote: > On Wed, 2011-11-09 at 19:16 +0300, Alexander Chekalin wrote: >> Will also try to use full text search, >> but afraid of index size (and I need no search on body, just on headers). > > It wouldn't be difficult to patch Dovecot to skip indexing message > bodies. Of course then you'd need to remember to keep applying the patch > when updating. Also keep in mind that, in general, many/most message headers today are often as large, or larger than, the actual message body, especially for list mail. Just take a look at messages from this for evidence. Thus, I'd think that going out of your way to avoid indexing message bodies wouldn't be worth the effort/headaches involved. -- Stan
Re: [Dovecot] Please advise on very fast search
On 11/9/2011 10:16 AM, Alexander Chekalin wrote: > Thanks, Robert, > > will take a look at. > > What I'm afraid for is how database storage should be planned (storage, > CPU, RAM, scaling when will be over-filled). When dealing with files > (I'm using maildir) Bingo.^^^ Maildir is very likely a hug factor in your current slow search time. With a maildir search, every mail file must be opened and searched. How many total mail files are opened for each of your searches? Thousands? Tens of thousands? Maildir causes a massive disk IO bottleneck when searching so many files. Run iostat the next time you do one of these searches, and look at the %iowait value. It will likely be very high. If it is, this confirms maildir is a big part of the problem. mbox, and mdbox, would be many many times faster than maildir WRT searching as the total number of files is lower by orders of magnitude. Switching from maildir to mbox/mdbox shifts the workload burden from the disk subsystem to the processor/memory. And I'm sure as with everyone else on the planet today, you have massive spare CPU cycles, but extremely limited spindle throughput. And as Timo suggested, using one of the indexing search plugins would be much faster yet, as long as you keep the indexes updated. -- Stan
[Dovecot] Can't delete sub-folder
Hi list, I have problem with deleting sub-folder. I use thunderbird 8.0 as a client and dovecot 2.0.9 with imap protocol as a server. I use dovecot with tb-extra-mailbox-sep parameter. I use mbox mailboxes and thunderbird with *not checked* option 'server supports folders that contain sub-folders and messages' so when I create sub-folder I can choose between 'folders only' and 'messages only'. When I try to delete 'parent' folder which contain 'messages only' folder, error message appear: [ALREADYEXISTS] Mailbox has children, delete them first. Ok, so I delete 'message only' folder first and try to delete parent folder, but the same error appear: [ALREADYEXISTS] Mailbox has children, delete them first. When I list mailboxes from console I see inside 'parent' folder ".imap" folder. When I delete .imap folder from console and then delete parent folder from thunderbird, parent folder is deleted correctly. Which application create this ".imap" hidden folder and how solve this problem. Thanks
Re: [Dovecot] Bus factor of dovecot / GIT
On 11/09/2011 10:29 AM Thomas Koch wrote: > Hi, > > somebody from the kolab groupware project recently explained me that the Bus- > Factor[1] of around 1 would be one of their primary reasons not to use > Dovecot > and stick with Cyrus. And what is their primary reason for the lame excuse? Too many cooks spoil the broth. just my 2¢ Regards, Pascal -- The trapper recommends today: fabaceae.1131...@localdomain.org
Re: [Dovecot] LDAP expired password
Sven Hartge svenhartge.de> writes: > > Nikolaos Milas noa.gr> wrote: > > On 1/4/2011 11:09 πμ, Sven Hartge wrote: > > >> Have a look at the ppolicy slapd.overlay. This will solve your > >> problem. hello all Sorry for the delay in the response I checked the ppolicy overlay but without success. This overlay does not have a single "password expired" attribute to put in the user_filter. currently my dovecot-ldap.conf contain: user_filter (&(|( = uid =% u) (employeeNumber =% u)) (objectClass = inetOrgPerson) (accountStatus = active)) > At my university we introduced our own attribute gifb-status which > contains a "1" if an account is valid, a "0" if it is not (and several > others for different purposes) and our ldap-filters all contain > something like "(&(ou=foobar)(gifb-status=1))". is possible that the only way to do this is to manage a new attribute? how can understand all the people that have configured the mail client to authenticate with imap-dovecot that their passoword has expired? thanks / roby
Re: [Dovecot] Corrupted transaction log file
On 2011-11-04 22:26, Timo Sirainen wrote: Nov 4 15:10:42 mail dovecot: imap (t...@aaaone.net): Error: Corrupted transaction log file /mail/3340444/.TestMails/dovecot.index.log seq 2: indexid changed 1320419300 -> 1320419441 (sync_offset=0) Session A had TestMails open and created with index file whose ID was 1320419300 (that's also UNIX timestamp of its creation time, Fri Nov 4 17:08:20 EET 2011). Session B came and recreated the index files 141 seconds later with ID 1320419441. Either it didn't see A's original index files for some reason or it simply decided to recreate them for some reason. Either way this shouldn't have happened. Turns out this is expected to confuse Session A. The client in question sometimes start the session (B) with this command sequence: DELETE folder CREATE folder APPEND... Any Session A having opened "folder" of course would be surprised that there's a new index file (makes we wish for an IMAP5 where folderID!=displayname) This can be reproduced by hand speaking IMAP with two telnets. Only question left is, why does Dovecot end the log sequence by saying: "Disconnected: IMAP session state is inconsistent, please relogin." ... when it is capable of detecting this and returning "BYE folder deleted under us" and logging the same. /Peter
Re: [Dovecot] 2.1: virtual plugin and lsub
On 06.11.2011 17:03, wrote e-frog: 1 lsub "" "virtual/*" * LSUB () "/" "virtual" * LSUB () "/" "virtual/Lists" * LSUB () "/" "virtual/unread" 1 OK Lsub completed. Just tested this again with 2.1beta1 with the same result.
Re: [Dovecot] default Dovecot-LDA CHGRP/CHMOD
I am familiar with this article http://wiki.dovecot.org/LDA#multipleuids I want to disable setuid-root, but I don't know the default group and permissions. sudo chgrp root /usr/libexec/dovecot/dovecot-lda sudo chmod 00750 /usr/libexec/dovecot/dovecot-lda but my delivery is still failing Nov 9 18:22:39 vl42 postfix/pipe[23039]: DEC54700B1: to=, orig_to=, relay=dovecot, delay=0.05, delays=0.04/0/0/0.01, dsn=4.3.0, status=deferred (temporary failure. Command output: pipe: fatal: pipe_command: execvp /usr/libexec/dovecot/dovecot-lda: Permission denied ) Thoughts? Thanks Chris
Re: [Dovecot] Quota BUG ?
On 11/09/11 20:46, Timo Sirainen wrote:
On Wed, 2011-11-09 at 20:44 +0200, Adrian Minta wrote:
Is the quota in the database 1096 or>2GB? i.e. is the problem with
reading it, or updating it? In general the quota code uses 64bit
integers everywhere so this shouldn't be happening.
In the database quota field is 204800.
In the dict quota value? That looks more like the quota limit, not the
current quota usage.
In the mailbox table.
The 200 value commes from:
user_query = SELECT '/home/%d/%n' as home, 'maildir:/home/%d/%n' as
mail, 150 AS uid, 8 AS gid, CONCAT('*:bytes=', CAST(quota AS CHAR)) AS
quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
This value is easy to change by modifying the above mysql query, but the
1096 value I don't know how to change. Maybe to "instruct" dovecot to
work internal with Kilobytes ?
For instance by altering the query like this:
user_query = SELECT '/home/%d/%n' as home, 'maildir:/home/%d/%n' as
mail, 150 AS uid, 8 AS gid, CONCAT('*:bytes=', quota, 'K') AS
quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
... I get ...
* QUOTA "user" (STORAGE 1096 204800)
--
Best regards,
Adrian MintaMA3173-RIPE, www.minta.ro
Re: [Dovecot] Bus factor of dovecot / GIT
On Nov 9, 2011, at 7:53 AM, Timo Sirainen wrote: > On Wed, 2011-11-09 at 10:29 +0100, Thomas Koch wrote: > >> Somehow related: Since the free software world[2] slowly converges towards >> GIT >> as the "one and only" VCS, have you thought about a switch? It's much more >> likely that somebody checks out your code, looks into it and provides >> patches >> if the VCS is already installed. > > I'm not as much against git anymore as I was when I switched to hg, but > I don't see much benefits in switching to git either. I highly doubt I'd > get even a single patch more if I used git instead of hg. > Hg / Git. Either seems fine to me. My big concern would be that the source history is in multiple accessible places around the world. This is the biggest limitation of SVN to my mind. Could I suggest a google code clone as an additional VCS backup location?
Re: [Dovecot] Quota BUG ?
On Wed, 2011-11-09 at 20:44 +0200, Adrian Minta wrote: > > Is the quota in the database 1096 or>2GB? i.e. is the problem with > > reading it, or updating it? In general the quota code uses 64bit > > integers everywhere so this shouldn't be happening. > > > > > In the database quota field is 204800. In the dict quota value? That looks more like the quota limit, not the current quota usage.
Re: [Dovecot] Quota BUG ?
Is the quota in the database 1096 or>2GB? i.e. is the problem with reading it, or updating it? In general the quota code uses 64bit integers everywhere so this shouldn't be happening. In the database quota field is 204800. -- Best regards, Adrian MintaMA3173-RIPE, www.minta.ro
Re: [Dovecot] Quota BUG ?
On Wed, 2011-11-09 at 20:35 +0200, Adrian Minta wrote: > On 11/09/11 19:59, Timo Sirainen wrote: > > Anyway, here's a guess: the quota is stored in mysql in "integer", which > > has a maximum value of 2 GB. Actually looks like Dovecot wiki pages also > > use "integer". Replace that with "bigint" and recalculate quota and it > > should work. > > > Actually the mysql part is OK: > quota | bigint(20) > > As you can see the value readed from mysql ( 200bytes) is also OK: 200 kilobytes is the quota limit. > * QUOTA "user" (STORAGE 1096 200) > > The problem is the 1096 number calculated internally by dovecot. By > tweaking the SQL querry I coud put any value instead of 200, but I > was unable to change 1096 part. Is the quota in the database 1096 or >2GB? i.e. is the problem with reading it, or updating it? In general the quota code uses 64bit integers everywhere so this shouldn't be happening.
Re: [Dovecot] Quota BUG ?
On 11/09/11 19:59, Timo Sirainen wrote: Anyway, here's a guess: the quota is stored in mysql in "integer", which has a maximum value of 2 GB. Actually looks like Dovecot wiki pages also use "integer". Replace that with "bigint" and recalculate quota and it should work. Actually the mysql part is OK: quota | bigint(20) As you can see the value readed from mysql ( 200bytes) is also OK: * QUOTA "user" (STORAGE 1096 200) The problem is the 1096 number calculated internally by dovecot. By tweaking the SQL querry I coud put any value instead of 200, but I was unable to change 1096 part. -- Best regards, Adrian MintaMA3173-RIPE, www.minta.ro
Re: [Dovecot] Quota BUG ?
On Wed, 2011-11-09 at 14:59 +0200, Adrian M wrote:
> Hello list,
> I believe I found a bug with quota.
> The dovecot quota replies with incorrect values when the mailbox is above 2GB.
>
> du -ks returns 2005764bytes
>
> dovecot IMAP on the same user/directory:
> 1 getquotaroot inbox
> * QUOTAROOT "INBOX" "user"
> * QUOTA "user" (STORAGE 1096 200)* QUOTAROOT "INBOX" "user"
> * QUOTA "user" (STORAGE 1096 200)
> 1 OK Getquotaroot completed.
So it wraps.
> dovecot.conf:
> dict {
>quotadict= mysql:/etc/dovecot/dovecot-dict-quota.conf
> }
> plugin {
> quota = dict:user::proxy::quotadict
> }
>
> dovecot-dict-quota.conf :
> user_query = SELECT '/home/%d/%n' as home, 'maildir:/home/%d/%n' as
> mail, 150 AS uid, 8 AS gid, CONCAT('*:bytes=', CAST(quota AS CHAR)) AS
> quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
This is dovecot-sql.conf, not dovecot-dict-quota.conf (or alternatively
you've named them in a "wrong" way).
Anyway, here's a guess: the quota is stored in mysql in "integer", which
has a maximum value of 2 GB. Actually looks like Dovecot wiki pages also
use "integer". Replace that with "bigint" and recalculate quota and it
should work.
Re: [Dovecot] Dovecot 2.0.15 quota configuration with mbox
On Wed, 2011-11-09 at 10:54 +0100, David Ocana wrote:
> I've been trying to set up dovecot 2.0.15, everything seems to work
> pretty well except for the quota feature. I would like to set a quota
> limit only for the Inbox folder. I configured two namespaces,
> according to some posts from Timo Sirainen
>
> namespace {
> separator = /
> prefix = INBOX/
> location = mbox:/var/empty:INBOX=/mail/%d/%n:INDEX=/var/dovecot/%d/%n
> inbox = yes
> hidden = yes
> }
>
> plugin {
> quota = dirsize:User quota
quota = dirsize:User quota:ns=INBOX/
This limits the quota only to mailboxes in INBOX/ namespace.
> # I've tried with:
> quota_rule = INBOX:storage=819200K
> quota_rule = INBOX/*:storage=819200K
> quota_rule = INBOX/Inbox:storage=819200K
Quota rules don't work in this way. There are no per-mailbox quotas
really, at least in the way you're thinking about.
Re: [Dovecot] Please advise on very fast search
On Wed, 2011-11-09 at 19:16 +0300, Alexander Chekalin wrote: > Will also try to use full text search, > but afraid of index size (and I need no search on body, just on headers). It wouldn't be difficult to patch Dovecot to skip indexing message bodies. Of course then you'd need to remember to keep applying the patch when updating.
Re: [Dovecot] v2.1.beta1 released
On Wed, 2011-11-09 at 18:34 +0300, Odhiambo Washington wrote: > I have today tried compiling this -beta, but I saw so many warning about > linking libraries, which I ignored, Probably plugin warnings which you can safely ignore. > Making all in wiki > make: don't know how to make 2b. Stop Looks like there's some weird file went into wiki docs, which some makes don't like. You could just find the line containing "2b" from doc/wiki/Makefile and delete it.
Re: [Dovecot] Please advise on very fast search
Thanks, Robert, will take a look at. What I'm afraid for is how database storage should be planned (storage, CPU, RAM, scaling when will be over-filled). When dealing with files (I'm using maildir), it is much easy to understand and to fix just about everything. Adding database involves tune it up too, and I'll have more points of 'tune it a bit' In fact work with Dovecot is pretty nice, but I think I can tune it to work faster. I now run it on FreeBSD (on UFS2), maybe I should change OS + FS, but need to test (really hope ZFS disks on SAS drives will help; still find no benchmarks on such a setup). Will also try to use full text search, but afraid of index size (and I need no search on body, just on headers). Anyway thank your for pointing me in right directions! Yours, Alexander
Re: [Dovecot] v2.1.beta1 released
On Wed, Nov 9, 2011 at 01:35, Timo Sirainen wrote: > http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz > http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz.sig > > Here's the first beta release of Dovecot v2.1. This version has already > been tested quite a lot, so I'm not expecting any major bugs. So please > upgrade and see if you can find any problems. I'm optimistic about > getting rc1 released this year and perhaps even v2.1.0. > > Since alpha2 there have been a lot of fixes, especially to imapc > backend. There are probably also some small new features, but nothing > huge. > > As a reminder, here's the largest changes since v2.0: > >* Plugins now use UTF-8 mailbox names rather than mUTF-7: > acl, autocreate, expire, trash, virtual >* auth_username_format default changed to %Lu. If you really want > case sensitive usernames, set it back to empty. >* Solr full text search backend changed to use mailbox GUIDs > instead of > mailbox names, requiring reindexing everything. solr_old backend > can > be used with old indexes to avoid reindexing, but it doesn't > support > some newer features. > >+ imapc (= IMAP client) storage allows using a remote IMAP server to > be used as storage. This allows using Dovecot as a smart (caching) > proxy or using dsync to do migration from remote IMAP server. >+ Mailbox indexing via queuing indexer service (required for Lucene) >+ Lucene full text search (FTS) backend rewritten with support for > different languages >+ FTS finally supports "OR" search operation >+ FTS supports indexing attachments via external programs >+ IMAP FUZZY extension, supported by Lucene and Solr FTS backends >+ Mailbox list indexes >+ Statistics tracking via stats service. Exported via doveadm stats. >+ Autocreate plugin creates/subscribes mailboxes physically only > when > the mailbox is opened for the first time. Mailbox listing shows > the > autocreated mailboxes even if they don't physically exist. >+ Password and user databases now support default_fields and > override_fields settings to specify template defaults/overrides. >- listescape plugin works perfectly now > > I have been running the alpha for some time now. I have today tried compiling this -beta, but I saw so many warning about linking libraries, which I ignored, but it did fail with: Making all in wiki make: don't know how to make 2b. Stop *** Error code 1 Stop in /usr/home/wash/Tools/Dovecot/dovecot-2.1.beta1/doc. *** Error code 1 Stop in /usr/home/wash/Tools/Dovecot/dovecot-2.1.beta1. *** Error code 1 Stop in /usr/home/wash/Tools/Dovecot/dovecot-2.1.beta1. -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. <>
Re: [Dovecot] Please advise on very fast search
On Wed, 2011-11-09 at 16:57 +0300, Alexander Chekalin wrote: > The problem is that when my archive become big (several years), it > appears to be painful to find specified message(s). When someone > suddenly needs to find his/her old message, it is mostly guesses like 'I > think the message was between june and july of 2009, or maybe month or > two before that', so I need to search all mailboxes (with 1000's > messages in each). And it takes really long time. > > > I tried to play with Dovecot indexes, but it won't help too much. They'll help with the dates. > The > bad part is that I need to search for all emails in each message > headers, not only for "From" or "To", since some messages are sent to > maillists soe "To" = list address, not person's personal email. Headers only, not message body? Anyway, some of the full text search backends would support searching from both. I'd recommend using either Solr or with Dovecot v2.1 you can also use Lucene: http://wiki2.dovecot.org/Plugins/FTS
Re: [Dovecot] Doveadm scrip-fu to determine alternate storage needs
On Wed, 2011-11-09 at 15:24 +0100, Maria Arrea wrote: > Timo, maybe you have mispelled the command? Minor changes, I tested that this actually works: doveadm -f tab fetch -A size.physical before 365d The -A parameter requires that you use a userdb that supports iteration. With SQL you need to provide the iteration_query.
Re: [Dovecot] Doveadm scrip-fu to determine alternate storage needs
Timo, maybe you have mispelled the command? > We are using dovecot 2.0.15+mdbox+zlib and we are evaluating to buy > cheap > sata-based iscsi storage / sata-NFS. We want to evaluate how > much space > would be needed in alternate storage BEFORE buying it. I > have been reading > about doveadm altmove. We want to move ALL messages > older than a year to > alternate storage, what doveadm-script-fu could I > do to identify how much > space would go from a user to alternate > storage? doveadm -A -f tab fetch > size.physical before 1y would print all the message sizes. You could then > sum them up with some scripting to produce the total. Unfortunately they are > the uncompressed sizes, but maybe simply dividing by 2 would give > approximate size? :) (You could get the compressed sizes from > dovecot.map.index files with doveadm dump and mapping their contents to the > older than 1 year message UIDs, but this gets difficult.)
Re: [Dovecot] Please advise on very fast search
Am 09.11.2011 14:57, schrieb Alexander Chekalin:
> Hello,
>
> I try to create some kind of mail backup system. What I need is system
> that will store mail for the whole domain, and allow me to restore
> messages from/to specified email at that domain.
>
> The scheme is pretty simple: on our main mail server the SMTP server
> itself has a rule to send a copy of every message to
> 'bac...@backupserver.host', and the backupserver.host domain is placed
> nearby on second server.
>
> The SMTP on second server do simple 'catchall' redirect of all messages
> to the single box. There is also a Dovecot that takes care for remote
> IMAP access to that box. And, finally, I've create some scripts to sort
> all messages in INBOX to folders named after message's date.
>
> So I have a lot of mailboxes inside the catchall box:
> INBOX
> 2011.11.03
> 2011.11.04
> 2011.11.05
> 2011.11.06
> ...etc...
>
> and each folder holds messages for that day. Simply, and works perfectly.
>
> The problem is that when my archive become big (several years), it
> appears to be painful to find specified message(s). When someone
> suddenly needs to find his/her old message, it is mostly guesses like 'I
> think the message was between june and july of 2009, or maybe month or
> two before that', so I need to search all mailboxes (with 1000's
> messages in each). And it takes really long time.
>
>
> I tried to play with Dovecot indexes, but it won't help too much. The
> bad part is that I need to search for all emails in each message
> headers, not only for "From" or "To", since some messages are sent to
> maillists soe "To" = list address, not person's personal email.
>
> Then I tried to index messages on my own, storing info on emails into
> MySQL database ('email' -> 'mailbox', 'message filename'), but soon I
> find out that message files can be renamed by Dovecot.
>
> Could you please advice me how to speed up message search?
>
>
> Sorry for such a long question, hope you can help!
>
> Yours,
> Alexander Chekalin
>
guess youre searching over imap ?
perhaps compression will help for speed up, and many other speed related
stuff, or you need some other idea of indexing
at last if its maildir how fast is "grep" etc...and so on
some ideas here
http://wiki.dovecot.org/HowTo/ReadOnlyArchive etc
anyway , i think you really need another kind of archive solution
in Germany there is a law that you need to archive some kind of business
mails up to 10 years for finance and other review, so there are a lot of
"you can by" solutions now, these have solved the problems you
discovered ( indexing etc )
i was shown i.e
http://www.bytstormail.de which looked fine to me
or
perhaps you might have a look
http://www.archiveopteryx.org/
here too
--
Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
Re: [Dovecot] Doveadm scrip-fu to determine alternate storage needs
On Wed, 2011-11-09 at 14:19 +0100, Maria Arrea wrote: > We are using dovecot 2.0.15+mdbox+zlib and we are evaluating to buy > cheap sata-based iscsi storage / sata-NFS. We want to evaluate how > much space would be needed in alternate storage BEFORE buying it. I > have been reading about doveadm altmove. We want to move ALL messages > older than a year to alternate storage, what doveadm-script-fu could I > do to identify how much space would go from a user to alternate > storage? doveadm -A -f tab fetch size.physical before 1y would print all the message sizes. You could then sum them up with some scripting to produce the total. Unfortunately they are the uncompressed sizes, but maybe simply dividing by 2 would give approximate size? :) (You could get the compressed sizes from dovecot.map.index files with doveadm dump and mapping their contents to the older than 1 year message UIDs, but this gets difficult.)
[Dovecot] Please advise on very fast search
Hello,
I try to create some kind of mail backup system. What I need is system
that will store mail for the whole domain, and allow me to restore
messages from/to specified email at that domain.
The scheme is pretty simple: on our main mail server the SMTP server
itself has a rule to send a copy of every message to
'bac...@backupserver.host', and the backupserver.host domain is placed
nearby on second server.
The SMTP on second server do simple 'catchall' redirect of all messages
to the single box. There is also a Dovecot that takes care for remote
IMAP access to that box. And, finally, I've create some scripts to sort
all messages in INBOX to folders named after message's date.
So I have a lot of mailboxes inside the catchall box:
INBOX
2011.11.03
2011.11.04
2011.11.05
2011.11.06
...etc...
and each folder holds messages for that day. Simply, and works perfectly.
The problem is that when my archive become big (several years), it
appears to be painful to find specified message(s). When someone
suddenly needs to find his/her old message, it is mostly guesses like 'I
think the message was between june and july of 2009, or maybe month or
two before that', so I need to search all mailboxes (with 1000's
messages in each). And it takes really long time.
I tried to play with Dovecot indexes, but it won't help too much. The
bad part is that I need to search for all emails in each message
headers, not only for "From" or "To", since some messages are sent to
maillists soe "To" = list address, not person's personal email.
Then I tried to index messages on my own, storing info on emails into
MySQL database ('email' -> 'mailbox', 'message filename'), but soon I
find out that message files can be renamed by Dovecot.
Could you please advice me how to speed up message search?
Sorry for such a long question, hope you can help!
Yours,
Alexander Chekalin
Re: [Dovecot] Bus factor of dovecot / GIT
On Wed, 2011-11-09 at 10:29 +0100, Thomas Koch wrote: > somebody from the kolab groupware project recently explained me that the Bus- > Factor[1] of around 1 would be one of their primary reasons not to use > Dovecot > and stick with Cyrus. > > What do you think about that? Is the bus factor much higher then 1? I've created a company for Dovecot support, and if all goes well we should have at least one other coder in not too distant future (anyone want a job? :) Also besides me there's already at least Stephan Bosch who has written Dovecot's Sieve/ManageSieve implementations. I don't know if he'd take care of the whole Dovecot if I happened to die right now, but at least he knows the code pretty well. There are also a few big companies that have some people who have done some Dovecot coding. Also the Bus-Factor of Cyrus doesn't seem to be much higher than 1 to me. AFAIK there's only a single person currently developing it actively (plus I guess a few more not-very-active developers from CMU). > Somehow related: Since the free software world[2] slowly converges towards > GIT > as the "one and only" VCS, have you thought about a switch? It's much more > likely that somebody checks out your code, looks into it and provides patches > if the VCS is already installed. I'm not as much against git anymore as I was when I switched to hg, but I don't see much benefits in switching to git either. I highly doubt I'd get even a single patch more if I used git instead of hg. The biggest problem with lack of patches is that few people are interested in coding a mail server. You can see the same with all open source IMAP/SMTP servers (and probably commercial ones too). Nearly always there's only a single guy who has written almost all of it.
[Dovecot] Doveadm scrip-fu to determine alternate storage needs
Hello We are using dovecot 2.0.15+mdbox+zlib and we are evaluating to buy cheap sata-based iscsi storage / sata-NFS. We want to evaluate how much space would be needed in alternate storage BEFORE buying it. I have been reading about doveadm altmove. We want to move ALL messages older than a year to alternate storage, what doveadm-script-fu could I do to identify how much space would go from a user to alternate storage? Regards Maria
[Dovecot] Quota BUG ?
Hello list,
I believe I found a bug with quota.
The dovecot quota replies with incorrect values when the mailbox is above 2GB.
du -ks returns 2005764bytes
dovecot IMAP on the same user/directory:
1 getquotaroot inbox
* QUOTAROOT "INBOX" "user"
* QUOTA "user" (STORAGE 1096 200)* QUOTAROOT "INBOX" "user"
* QUOTA "user" (STORAGE 1096 200)
1 OK Getquotaroot completed.
Is there any way to fix this ?
---
linux system:
dovecot 1.2.15-7
linux 2.6.32-5-amd64
dovecot.conf:
dict {
quotadict= mysql:/etc/dovecot/dovecot-dict-quota.conf
}
plugin {
quota = dict:user::proxy::quotadict
}
dovecot-dict-quota.conf :
user_query = SELECT '/home/%d/%n' as home, 'maildir:/home/%d/%n' as
mail, 150 AS uid, 8 AS gid, CONCAT('*:bytes=', CAST(quota AS CHAR)) AS
quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
Re: [Dovecot] GIT in Debian
Hi Sandro, I've been asking myself whether I should include Debian in the list. You're right that there are also other VCS' used in Debian. Zack's statistics say: arch22 bzr 271 cvs 31 darcs 382 git 5230 hg 63 mtn 13 svn 4843 http://upsilon.cc/~zack/stuff/vcs-usage/ The only open question is, towards which DVCS system the 4843 subversion packages will eventually migrate. I'd bet several rounds of beer on GIT. I believe that the recent migration of 2110 packages from the perl team from svn to git is not yet fully represented in the above numbers. But we shouldn't annoy the dovecot list with Debian details any longer. Regards, Thomas Koch, http://www.koch.ro
[Dovecot] Dovecot 2.0.15 quota configuration with mbox
Hello,
I've been trying to set up dovecot 2.0.15, everything seems to work
pretty well except for the quota feature. I would like to set a quota
limit only for the Inbox folder. I configured two namespaces, according
to some posts from Timo Sirainen
(http://dovecot.org/list/dovecot/2006-July/014530.html) &
(http://www.dovecot.org/list/dovecot/2011-January/056131.html) but I
can't configure the quota plugin to act the way I pretend.
Here's my namespace configuration:
namespace {
separator = /
prefix = INBOX/
location = mbox:/var/empty:INBOX=/mail/%d/%n:INDEX=/var/dovecot/%d/%n
inbox = yes
hidden = yes
}
namespace {
separator = /
prefix =
inbox = no
location = mbox:/mail/%d/MAILBOXES/%n:INDEX=/var/dovecot/%d/%n
}
Quota config:
plugin {
quota = dirsize:User quota
# I've tried with:
quota_rule = INBOX:storage=819200K
quota_rule = INBOX/*:storage=819200K
quota_rule = INBOX/Inbox:storage=819200K
# Works with:
quota_rule = *:storage=819200K
quota_rule = ?:storage=819200K (Same behavior as '*')
}
Using '*' and '?', is the only way I get it to work, but obviously,
quota is calculated summing up all the mailbox folders. I don't know
what else to change, any ideas? Is the two namespace configuration correct?
Thanks in advance.
David
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [Dovecot] Bus factor of dovecot / GIT
Am Mittwoch, 9. November 2011, 10:29:37 schrieb Thomas Koch: Hi, > somebody from the kolab groupware project recently explained me that the > Bus- Factor[1] of around 1 would be one of their primary reasons not to > use Dovecot and stick with Cyrus. The problem is: Even it Timo will be hitten by a bus, everybody could stay on the actual Dovecot-version and everything's fine. Cyrus will IMO never reach the level, that Dovecot already has. :-) Peer P.S.: Timo! Don't leave your house. It's too dangerous for you. -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Bus factor of dovecot / GIT
On 11/09/2011 10:29 AM, Thomas Koch wrote: Somehow related: Since the free software world[2] slowly converges towards GIT as the "one and only" VCS, have you thought about a switch? It's much more likely that somebody checks out your code, looks into it and provides patches if the VCS is already installed. [2] Android, Debian, Drupal, Eclipse, Fedora, Gnome, KDE, Linux Kernel, Perl, PHP (ongoing), PostgreSQL, Qt, Ruby on Rails, X.org (freedesktop.org) I don't know where you inferred that, but for sure Debian is not converging to git; we have that VCS as many others and none is the preferred/superior. Please try to balance what you say with actual facts: backing your reasoning with partial data is misleading for others not knowing the env you're talking about. Regards, -- Sandro Tosi Product Engineer Shared Hosting Products R&D | Dada.pro eml sandro.t...@register.it
[Dovecot] Bus factor of dovecot / GIT
Hi, somebody from the kolab groupware project recently explained me that the Bus- Factor[1] of around 1 would be one of their primary reasons not to use Dovecot and stick with Cyrus. What do you think about that? Is the bus factor much higher then 1? [1] http://en.wikipedia.org/wiki/Bus_factor Somehow related: Since the free software world[2] slowly converges towards GIT as the "one and only" VCS, have you thought about a switch? It's much more likely that somebody checks out your code, looks into it and provides patches if the VCS is already installed. [2] Android, Debian, Drupal, Eclipse, Fedora, Gnome, KDE, Linux Kernel, Perl, PHP (ongoing), PostgreSQL, Qt, Ruby on Rails, X.org (freedesktop.org) Best regards, Thomas Koch, http://www.koch.ro
Re: [Dovecot] Multiple Patitions with with mdbox
> How do you handle >> 10 TB mailstore? ZFS: no need to fsck. GlusterFS: "always-online". On Nov 8, 2011, at 6:19 AM, Peer Heinlein wrote: > > Having > 10 TByte mailstore filesystem-checks takes too much time. > > At the moment we have four different partitions, but I don't like to set > symlinks or LDAP-flags to sort customers and their domains to there > individual mount-point. I'd like to work with mdbox:/mail/%d/%n to calculate > the path automatically. > > How do you handle >> 10 TB mailstore? > > I'm very interested in the feature "alternative mailstore" with mdbox, > because that makes it very easy to use at least TWO filesystems without any > tricky configuration. > > I think I'd love to have alternative mailstores. Why does dbox doesn't > look for its m.*-files in more then two directorys? Sure, looking in 4 > directorys would lead to four disc operations, but maybe it could be very > helpful. > > Peer > -- > > Heinlein Professional Linux Support GmbH > Linux: Akademie - Support - Hosting > http://www.heinlein-support.de > > Tel: 030/405051-42 > Fax: 030/405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin
