Re: [Dovecot] Authentication cache, failure to login after changed password
Tom Sommer wrote:
Timo Sirainen wrote:
On Tue, 2009-01-20 at 09:53 +0100, Tom Sommer wrote:
sql(u...@example.com,127.0.0.1): query: SELECT username as user,
plainpassword as password, nopassword FROM cyrususers WHERE username
= 'u...@example.com' AND password = PASSWORD('SECRET') AND active = 1
dovecot: Jan 20 09:01:18 Info: auth-worker(default):
sql(u...@example.com,127.0.0.1): unknown user
..
It appears the user missed the cache, a SQL lookup is performed
(which returns 1 record, I tested the query directly) - however for
some reason the lookup is set as Unknown User, a state which it then
keeps.
It's most likely set to unknown user because the password=PASSWORD()
check fails and no rows are returned. If you're already returning
plainpassword for Dovecot, why do you do the password check also in the
SQL query? That doesn't allow Dovecot to differentiate between unknown
user and invalid password.
No I ran the query manually afterwards and it returned 1 row.
The reason I'm using plainpassword, PASSWORD() and nopassword, etc. is
because not all users have a plainpassword - yet - as time progress
more and more users will return plainpassword and nopassword=NULL
That's how you fix design flaws without forcing all users to change
passwords :)
auth_cache_negative_ttl seems like a good source for user flaws (login
attempt before account is created = you cant log in for 3600 seconds
even after the account is valid), gonna go with 0 on all servers.
I'm sorry, but I still have problems with this. I got cache_size 1024,
cache_ttl 3600, cache_negative_ttl 0, but if a user changes password in
my SQL, sometimes it requires a restart of dovecot for him to be able to
log in. Dovecot 1.1.14.
Using plain passwords, no nopassword feature, and MySQL for passdb.
Basically my log is:
--- Dovecot starts
- auth failed, 1 attempts
- auth failed, 1 attempts
- auth failed, 1 attempts
- auth failed, 1 attempts
- auth failed, 1 attempts
- auth failed, 1 attempts
- auth failed, 1 attempts
--- I RESTART DOVECOT ---
- Login successful (same password, same everything)
The cache seems to be faulty somehow, I wish there was a way to dump the
contents of the cache to debug this, because somehow I cannot forcefully
reproduce it.
Notice in this case the user never before logged in successfully, so I
dont understand why he would even be in the cache, unless there is
something wrong with cache_negative_ttl.
Thanks
// Tom
Re: [Dovecot] Migration questions...
Curtis Maloney wrote: Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Thanks again! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. Any opinions? Thanks again! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. I used to use ext3, ran into its horrible performance even with directory indexing enabled, went to XFS and never looked back. All of my systems are Debian stable. Reiser3 is part of the kernel so I wouldn't worry about that; Namesys considered it complete and stopped work on it long before the whole murder thing. Both Reiser3 and XFS have worse reputations than ext3, however, I've seen ext3 filesystems hosed beyond repair, too. All my XFS filesystems have battery-backed cache controllers, so it hasn't happened to me yet, hopefully never. ;) One catch with XFS (such as with LVM) to keep in mind is you can't ever shrink it, only grow. ext3 is mature but IMHO completely unsuitable for a busy mail server or any situation where you'll have a bajillion of files in one directory. The exact point at which ext3 will screw you over obviously depends on many factors. But when it happens it'll probably be painful to reformat to something better. ~Seth
[Dovecot] migration and conversion from courier
Dear all, back in January I posted about migration from courier and in particular on the script courier-dovecot-migrate.pl. That script had a bug on BSD (or should I say not-Linux?) type systems*. On FreeBSD (6.1) and MacOSX (10.5) it appended a lot of binary garbage at the end of dovecot-keywords files. On a couple of systems I had not actually run the script with the -- convert option, as I had misunderstood the procedure, so those systems, ironically, are the ones that I migrated better... Now to the issue. My final question at the time was (http://dovecot.org/pipermail/dovecot/2009-January/036292.html): can one delete the dovecot-keywords and rerun the script (leaving alone the dovecot-uidlist)? There is a problem if this cannot be done, as those files have the following structure: 0 flag0 1 flag1 ... m binary garbage [optionally followed by more lines like the last] and if a new keyword is added, the file ends up like (this is copy and paste example): 0 JunkRecorded 1 $NotJunk 2 $Junk 3 Redirected 4 ^AF0# 5 B096# 6 $Forwarded It does not look like a nice thing to me. I doubt though that the script will cope with flags correctly, as in principle will get the flags from an out of date courierimapkeywords, thus missing any new ones. Perhaps the only solution is to have a new script that deletes the garbage lines and renumbers the following ones, so that the above example would become: 0 JunkRecorded 1 $NotJunk 2 $Junk 3 Redirected 4 $Forwarded and then renames all *g files to *e. Thoughts? Giuliano * This bug was fixed back then by Timo who replaced the script on the site http://www.dovecot.org/tools/courier-dovecot-migrate.pl. But there's no note of the change on the site nor was an announcement made to the list, AFAIK.
Re: [Dovecot] Migration questions...
On F 15 May, 2009, at 09:39 , Seth Mattinen wrote: This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. What repercussion has not updating the index on delivery? My systems use exim to deliver and they seem to act normally... Giuliano
Re: [Dovecot] Possibly dumb questions about DC and user/system limits
On May 14, 2009, at 2:17 PM, Stewart Dean wrote: Warning: fd limit 2000 is lower than what Dovecot can use under full load (more than 2054). Either grow the limit or change login_max_processes_count and max_mail_processes settings So I changed the no_size and no_size_hard to 3500 and 4000 respectively in both dovecot and root AIX defines nosize:* *Sets the soft limit for the number of file descriptors a user process may have open at one time. an lsuser dovecot returns: dovecot id=417 pgrp=dovecot groups=dovecot shell=/bin/false daemon=true admin=false ... fsize=2097151 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=3500 nofiles_hard=4000 I kill dovecot and all children and restart itsame error message What am I missing? Does the machine have to be rebooted for the no_size to be updated? AIX should not have to be rebooted for this to happen. For debug, I would change the shell to a real shell; change the fsize to 600 or something small; logout and then login(as root and then su or just as the user) and try to make large files and verify limits. Ping me off list for more debugging unless people consider this on topic. -Jonathan smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 15 May 2009, Giuliano Gavazzi wrote: well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. Huh? exim won't try local deliver unless it has decided it is a local recipient. You won't get overquota status this way, did you mean that? Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBSg1EP3WSIuGy1ktrAQJtVAgAqH3G9syzKPtmLJ1axrGtAAKb6NS5knma 8081aI9LXsOHU/MiPJ0KCzRER80gJAbNhyYlFNaHU2nc9ryOPFO3+HnJ/tkpSqkG 3P8m7PCYa1+pupknU+ZH9YarcA7qxFT/wPHKCvhnlXEqQttvImn0ROL4EUN1waqP QmFivY26FJuhrn4BFm8t/1Unl5AJ5kknajZ7Q/WuNcwhxzewfNF4+CG29VOmJzs9 V87kwc4mpUEWwnkG7YkL4RJWhnBE9dnoZBYaCMR3GyqiL2vS1qGHJwYY6WC0tObQ bym38B7+OV9cOjiU0RYj8ZB4wa9UiGvNEokCzvS1uSjhx9bKl7JyGA== =CSiD -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Richard Hobbs wrote: Curtis Maloney wrote: Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. Do does this mean that it's slightly slower to actually deliver the mail with dovecot (because it's writing two places instead of one), but it saves the files having to be indexed again, so overall potentially faster? And one more question... given that we're going to be using maildir, should i still use dovecot's POP3 server, or whatever the standard one is? I've heard (through google mainly), that dovecot's POP3 server, in terms of performance, is actually quite bad compared to other ones, but what's your take on this? Thanks again, people! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, May 15, 2009 at 11:28:42AM +, Richard Hobbs wrote: [...] Only dovecot 'deliver' will update the index on delivery. Do does this mean that it's slightly slower to actually deliver the mail with dovecot (because it's writing two places instead of one), but it saves the files having to be indexed again, so overall potentially faster? Things get re-indexed on client request if Dovecot sees that index is stale. So you are buying faster response times for clients with somewhat higher server load at mail delivery. I don't know how this piecemeal update of the index stacks up against a complete re-index, but I'd assume it to be more efficient (only having to do new mails instead of whole mailbox). Still, I find the re-index to be almost instantaneous on not-so-small mailboxes (hundreds of MB) and fairly modest hardware, by today's standards. Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFKDVeEBcgs9XrR2kYRAtBGAJ4zHYA23C5SxJRcS5khH5cskZGfSACdFs3r 7tmWdF5ShGkTOiqVXWQIYjs= =swId -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Richard Hobbs wrote: Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. I used to use ext3, ran into its horrible performance even with directory indexing enabled, went to XFS and never looked back. All of my systems are Debian stable. Reiser3 is part of the kernel so I wouldn't worry about that; Namesys considered it complete and stopped work on it long before the whole murder thing. Both Reiser3 and XFS have worse reputations than ext3, however, I've seen ext3 filesystems hosed beyond repair, too. All my XFS filesystems have battery-backed cache controllers, so it hasn't happened to me yet, hopefully never. ;) One catch with XFS (such as with LVM) to keep in mind is you can't ever shrink it, only grow. Trouble is... i've been googling this as well, just now, and loads of people say XFS has the better performance, but loads of other people say ReiserFS has the better performance. We have battery backed up RAID controllers too, in this new system, and the systems are UPSd, so on that basis i'm still none the wiser! lol I appreciate your experience with XFS is a positive one, but even the dovecot web site says XFS might now be a good choice... http://wiki.dovecot.org/MailboxFormat/Maildir What a tough decision! I know it probably won't make too much difference in my situation, but i want this to be a very long-term solution, so want to do it right first time! Any other opinions on XFS vs Reiserfs with Dovecot maildir? Thanks again! Richard. ext3 is mature but IMHO completely unsuitable for a busy mail server or any situation where you'll have a bajillion of files in one directory. The exact point at which ext3 will screw you over obviously depends on many factors. But when it happens it'll probably be painful to reformat to something better. ~Seth __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Seth Mattinen, 2009-05-15 09:39: (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. That's obvious. But dovecot (AFAIK) updates (not rebuilds) indexes when it sees new messages, so it should not matter that much. At least with maildir and not too much messages in poll intervalls, so usual always on imap usage should be ok. It might be even better not to use dovecot-lda for heavy-usage systems, because you save the fork/exec and indexes are updated on demand (i.e. when the folder is polled), not for every single message inserted. With exim, I had also the problem that messages got bounced because the lda crashed occasionally (was a svn version, so I don't complain) and stupid exim treated this as a permanent error.
[Dovecot] Unable to delete folder in public namespace
I use public namespace for group mailboxes. Users are able to create subfolders , but its impossible to delete them. thunderbird 2.0.0.21 complains: Cant' rename mailbox to another storage type, both namespaces (private and public) use same storage type, so I believe that it's a namespace problem. Is there any way to move folders between namespaces? Or maybe sieve script or plugin which intercept move command and do the move to some unsibscribible folder in public namespace.
[Dovecot] Filesystem experience question was Migration questions
Doing a cursory Google scan on journaled Linux filesystems, it seems that the three ground-up journaled FSes: XFS, reiser and JFS all have their separate strong points but all compare favorably. Reiser does a better job with many small files...which would seem to be the reality of maildir formatted inboxes. Any comments on that? Any war stories, that is, any comments on reliability, availability and recoverability of them. Support issues? Possible down the road orphaning? Is FS quotaing support and does it work well with a lot (for me that means 4000) users? Anybody using IBM's open source JFS? I currently use AIX and it's been as dependable as a hammer there. In some 20 years with AIX, I can count on the fingers of one hand the number of times that I've had to use fsck. We had one set of machines that would drop hard 2-3x a week due to a bad main electrical trunk splice2 or 3 years of that and the worst I ever saw was having to BRS once every two months or so. OTOH, I distrust IBM's commitment to the little shop and affordable entry machines: this is fast vanishing in the pSeries/AIX world...or else we wouldn't be moving towards Linux. Thanks! Feel free to contact me offline if you deem this marginally related to DC. -- Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows Stewart Dean, Unix System Admin, Bard College, New York 12504 sd...@bard.edu voice: 845-758-7475, fax: 845-758-7035
Re: [Dovecot] Migration questions...
On F 15 May, 2009, at 12:30 , Steffen Kaiser wrote: On Fri, 15 May 2009, Giuliano Gavazzi wrote: well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. Huh? exim won't try local deliver unless it has decided it is a local recipient. You won't get overquota status this way, did you mean that? No. The problem is the try... or at least, it was when I had a cyrus + exim setup five years ago. With cyrus it was impossible to easily check if the recipient was an existing local user. It required either the use of the mbpath utility or a recipient callout with lmtp over tcp. I suppose that with a virtual users setup this is a moot point, but with system users nothing beats the simplicity of an all exim setup! (If you use exim, that is). Giuliano
[Dovecot] namespace list not working in dovecot 1.2 revision 9027:421393827a81
In my config that has worked for some time I have:
namespace private {
separator = /
prefix = ~/
hidden = yes
list = no # for v1.1+
}
1 list ~/ *
1 OK List completed.
2 list ~/ %
2 OK List completed.
I _think_ that the last time I tested this was 2 weeks ago but it
might have been 1 week ago.
Thanks,
Jonathan
smime.p7s
Description: S/MIME cryptographic signature
Re: [Dovecot] Filesystem experience question was Migration questions
On 5/15/2009, Stewart Dean (sd...@bard.edu) wrote: Reiser does a better job with many small files...which would seem to be the reality of maildir formatted inboxes. Any comments on that? I have been using Reiserfs3 on my mail server (maildir format) for 4 years, with zero problems, even after a few unclean shutdowns. Of course, there are also plenty of horror stories, but honestly most are based on earlier versions of reiserfs before most of the bugs were worked out.. Anyway, that my .02 clad coins worth... -- Best regards, Charles
Re: [Dovecot] FTS Plugin design
Citando Timo Sirainen t...@iki.fi: 1. You notice a non-text/* content-type and initialize text extraction for the MIME part. Like: struct attachment_extract_context * attachment_extract_init(const char *content_type); 2. After this you feed all the input belonging to that MIME part to: int attachment_extract_add(struct attachment_extract_context *ctx, const struct message_block *input); Don't output anything to FTS backend at this point. The attachment_extract_add() would probably just basically write to a temporary file. 3. Finally you'll notice that the MIME part ends (either you get headers for the next MIME part or the entire message ends). Then finish the extraction, which actually executes the whatever conversion binaries: int attachment_extract_finish(struct attachment_extract_context *ctx); 4. Get the resulting text to fts_backend_build_more() somehow. Either some attachment_extract_add_to_fts() which internally adds it or some kind of an iterator that returns the text in smaller blocks. Either would work.. That kind of an API would also make it possible to pretty easily modify in future to not write temporary files for specific content types if it's not required. I tried your approach and I think it is working pretty well. Now I only need to look carefully to the output of external programs and build the XML correctly to send to Solr. Thanks Timo Regards, Rui Carneiro -- Portugalmail, Comunicações S.A. www.portugalmail.net
[Dovecot] Filesystems (was Re: Migration questions...)
Richard Hobbs wrote: Trouble is... i've been googling this as well, just now, and loads of people say XFS has the better performance, but loads of other people say ReiserFS has the better performance. It starts to become a religious argument at some point. We have battery backed up RAID controllers too, in this new system, and the systems are UPSd, so on that basis i'm still none the wiser! lol I appreciate your experience with XFS is a positive one, but even the dovecot web site says XFS might now be a good choice... http://wiki.dovecot.org/MailboxFormat/Maildir What a tough decision! I know it probably won't make too much difference in my situation, but i want this to be a very long-term solution, so want to do it right first time! Any other opinions on XFS vs Reiserfs with Dovecot maildir? Thanks again! Richard. If you have time to play tweak the filesystem: http://everything2.com/index.pl?node_id=1479435 It's a good comparison of etx3, xfs, and reiser from a performance standpoint. Also try this if you want to pull the plug or randomly reboot and see what happens: http://brad.livejournal.com/2116715.html ~Seth
[Dovecot] cmd_append_continue_parsing assertion failure
Dovecot-1.1.14 on Mac OS X. Panic: IMAP(user): file cmd-append.c: line 266 (cmd_append_continue_parsing): assertion failed: (ctx-count == uid2 - uid1 + 1) Trivially reproducible: $ telnet mailserver 143 a login user password b append inbox Backtrace: 0 libSystem.B.dylib 0x7fff8458aa92 __kill + 10 1 libSystem.B.dylib 0x7fff84606a1c abort + 83 2 imap 0x000100069f3e default_fatal_finish + 70 3 imap 0x000100069f8c i_syslog_fatal_handler + 0 4 imap0x0001000696eb i_info + 0 5 imap 0x00011406 cmd_append_continue_parsing + 434 6 imap0x000117a3 cmd_append + 407 7 imap 0x0001677e client_command_input + 33 8 imap 0x0001693f client_command_input + 482 9 imap 0x00016a3e client_handle_input + 239 10 imap0x00016bc2 client_input + 148 11 imap 0x000100070f83 io_loop_handler_run + 272 12 imap0x000100070348 io_loop_run + 55 13 imap0x0001db95 main + 1635 14 imap0x00011010 start + 52
Re: [Dovecot] cmd_append_continue_parsing assertion failure
On Fri, 2009-05-15 at 12:00 -0500, Mike Abbott wrote: Dovecot-1.1.14 on Mac OS X. Panic: IMAP(user): file cmd-append.c: line 266 (cmd_append_continue_parsing): assertion failed: (ctx-count == uid2 - uid1 + 1) Trivially reproducible: $ telnet mailserver 143 a login user password b append inbox Thanks, fixed: http://hg.dovecot.org/dovecot-1.1/rev/b9c3205084a3 signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Possibly dumb questions about DC and user/system limits
On May 14, 2009, at 2:17 PM, Stewart Dean wrote: The only changes was that max_mail_processes went from 1024 to 1280. Now I get a error message when I start DC: Warning: fd limit 2000 is lower than what Dovecot can use under full load (more than 2054). Either grow the limit or change login_max_processes_count and max_mail_processes settings So I changed the no_size and no_size_hard to 3500 and 4000 respectively in both dovecot and root AIX defines nosize:* *Sets the soft limit for the number of file descriptors a user process may have open at one time. I found a way to recreate the problem on my side. Since I start dovecot from inetd, I must verify my shell has the proper ulimits then do stopsrc -s inetd;startsrc -s inetd to pick up the new ulimits. I didn't notice how you started dovecot. Maybe this helps. smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] Possibly dumb questions about DC and user/system limits
Maybe it's a D'uh for everyone else, but the reason I wasn't getting anywhere when I increased the no_size and no_size_hard (and confirmed that it had changed using the lsuser cmd) was that a) it needed to be done against root, not dovecot and b) I had to exit root and re-login for that to be part of root environment instance. Indeed, before going out and coming back, ulimit -n did still show 2000. So lsuser shows the configuration and ulimit shows the current environment. All better until the next time. I was also interested to learn, while I was looking under rocks here and there, that in AIX with a 64 bit kernel there is total number of files that can be in use by all users of ~3,355,236. Wouldn't it be nasty to bump into that! Jonathan Siegle wrote: On May 14, 2009, at 2:17 PM, Stewart Dean wrote: The only changes was that max_mail_processes went from 1024 to 1280. Now I get a error message when I start DC: Warning: fd limit 2000 is lower than what Dovecot can use under full load (more than 2054). Either grow the limit or change login_max_processes_count and max_mail_processes settings So I changed the no_size and no_size_hard to 3500 and 4000 respectively in both dovecot and root AIX defines nosize:* *Sets the soft limit for the number of file descriptors a user process may have open at one time. I found a way to recreate the problem on my side. Since I start dovecot from inetd, I must verify my shell has the proper ulimits then do stopsrc -s inetd;startsrc -s inetd to pick up the new ulimits. I didn't notice how you started dovecot. Maybe this helps. -- Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows Stewart Dean, Unix System Admin, Bard College, New York 12504 sd...@bard.edu voice: 845-758-7475, fax: 845-758-7035
[Dovecot] ext3/ext4 performance issue
Wonder if anyone else has actually noticed this and has some kind of statistics? http://marc.info/?t=12423400331r=1w=2 signature.asc Description: This is a digitally signed message part
[Dovecot] LDAP auth_bind fails to process certain LDAP server responses (Windows AD)
LDAP auth backend in Dovecot 1.1.5 seemingly fails to handle certain paths of interaction with LDAP server, which is Windows AD running Windows 2003 server machine in my case. The symptoms I observe look exactly as were already reported in [1]: authentication backend hangs after logging the fact it has sent the search request, then after a while the IMAP/POP3 connection is teared down by the server due to idle condition. In the mentioned mail thread, it was proposed that if Dovecot doesn't log any LDAP server response, there is no such response, so I dumped the TCP traffic going between the backend and the server, and studied what happens with different settings. The result is that it seems LDAP auth backend doesn't understand the case when the server offerrs one or more referrals which the client might want to check. It seems that the backend tries to do something else after it gets such a result from the LDAP server even if the server returned an entry matching the search request. The described behaviour manifests itself with Windows AD only if I specify the domain name alone as the search base, e.g. dc=mydomain,dc=com. If I also include at least one OU, the server doesn't return its list of referrals and Dovecot LDAP auth backend works OK. I attached two files produced by tcpdump -- one with hung search request, and another with successful one. They are: 1) ldap-hung.pcap.gz Captured with /etc/dovecot/dovecot-ldap.conf settings: hosts = domain007.com dn = saslau...@domain007.com dnpass = secret auth_bind = yes base = dc=domain007, dc=com pass_filter = ((objectClass=person)(sAMAccountName=%u)) Packet #8 is of interest in this dump. 2) ldap-ok.pcap.gz Captured with /etc/dovecot/dovecot-ldap.conf settings: hosts = domain007.com dn = saslau...@domain007.com dnpass = secret auth_bind = yes base = cn=Users, dc=domain007, dc=com pass_filter = ((objectClass=person)(sAMAccountName=%u)) The only difference compared to the first case is the cn=Users prepended to the base DN, which prevents the server from attaching a list of referrals. In both cases search result done is present in the server response. 3) ldap-ldapsearch.pcap.gz This dump captures the conversation of the ldapsearch utility with the same server using the same parameters Dovecot LDAP auth backend uses in the first case. ldapsearch invocation was like this: $ ldapsearch -h domain007.com -x -D saslau...@domain007.com -w secret -a always -b dc=domain007,dc=com '((objectClass=person)(sAMAccountName=kostix))' General configuration info (package from Debian stable): # dovecot --version 1.0.15 # dovecot -n # 1.0.15: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_privileged_group: mail mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: verbose: yes debug: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: passwd 1. http://www.mail-archive.com/dovecot@dovecot.org/msg09174.html ldap-hung.pcap.gz Description: application/gzip ldap-ok.pcap.gz Description: application/gzip ldap-ldapsearch.pcap.gz Description: application/gzip
Re: [Dovecot] ext3/ext4 performance issue
* Timo Sirainen t...@iki.fi: Wonder if anyone else has actually noticed this and has some kind of statistics? http://marc.info/?t=12423400331r=1w=2 I could run stats here, with 12.000 users to see what the average size is... -- Ralf Hildebrandt Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.computerbeschimpfung.de People demand freedom of speech as a compensation for the freedom of thought which they seldom use.
Re: [Dovecot] ext3/ext4 performance issue
On Fri, 2009-05-15 at 22:58 +0200, Ralf Hildebrandt wrote: * Timo Sirainen t...@iki.fi: Wonder if anyone else has actually noticed this and has some kind of statistics? http://marc.info/?t=12423400331r=1w=2 I could run stats here, with 12.000 users to see what the average size is... It would be interesting to have some kind of statistics for cur/ directories and get cur directory size divided by number of files in cur and then see how many users have values that are much higher than average (i.e. how many users have had tons of mails that have since been deleted). signature.asc Description: This is a digitally signed message part
Re: [Dovecot] ext3/ext4 performance issue
on 5-15-2009 11:59 AM Timo Sirainen spake the following: Wonder if anyone else has actually noticed this and has some kind of statistics? http://marc.info/?t=12423400331r=1w=2 One could always run the mailstore on LVM and then you could snapshot the mount and then fsck it while still technically in use. It would probably slow down the filesystem, but it is still live. signature.asc Description: OpenPGP digital signature
Re: [Dovecot] ext3/ext4 performance issue
On Fri, 2009-05-15 at 17:13 -0400, Timo Sirainen wrote:
I could run stats here, with 12.000 users to see what the average size
is...
It would be interesting to have some kind of statistics for cur/
directories and get cur directory size divided by number of files in
cur and then see how many users have values that are much higher than
average (i.e. how many users have had tons of mails that have since been
deleted).
I wrote an ugly script for it, just change the find . path to wherever
your maildirs are and run in some temp directory:
### total.sh
#!/bin/sh
find . -name cur -exec ./diravg.sh {} \; log
count=`cat log|wc -l`
avg=`cat log | awk '{print $2+}'|tr -d '\n'|sed 's/^/(/'|sed s,+\
$,)/$count\n,|bc -l`
echo average: $avg
echo 3x higher:
cat log | perl -ne @l = split(/ /); print if \$l[1] = ($avg*3);
### diravg.sh
#!/bin/sh
dir=$1
files=`/bin/ls $dir|wc -l`
if [ $files = 0 ]; then
files=1
fi
ls -ld $dir|head -1|awk {print \$dir \ (\$5 / $files);};
signature.asc
Description: This is a digitally signed message part
