Re: Courier-IMAP brings down system when searching Entire Message
* Steve Shockley wrote: Marc Balmer wrote: Indeed, I will try to test an update soon. I have a moderately recent Courier-MTA port somewhere that may help. I'd be glad to update it if there's a chance of committal. Sure. Send me a diff and I can probably merge that with what I already have. I am working on version 4.4.1, btw.
Re: Courier-IMAP brings down system when searching Entire Message
* Brad wrote: On Saturday 26 July 2008 00:30:30 Jason Dixon wrote: On Fri, Jul 25, 2008 at 10:53:52PM -0400, [EMAIL PROTECTED] wrote: I understand your frustration but I think there's something else going on here... Agreed. I've been using Courier-IMAP on OpenBSD going back at least 5 years. The last few have seen a *lot* of Mail.app and Thunderbird users beat on my server constantly with nary a problem. As a side note I have wondered why the Courier IMAP port is almost 2.5 years out of date.. there have been a few bug fixes in that period of time that would definitely warant an update. Indeed, I will try to test an update soon. BTW: we are using the OpenBSD/courier-imap combo for real large mailserver and don't see the problems that were described.
Re: Courier-IMAP brings down system when searching Entire Message
On 2008/07/26 00:11, Marcos Laufer wrote: Does anyone has experience with Perdition and/or the built-in proxy aggregator? you should probably also look at nginx if you're going to do this. (dovecot's proxy is good too, but then you might as well just switch to dovecot :-)
Re: Courier-IMAP brings down system when searching Entire Message
[EMAIL PROTECTED] wrote: Okay, I have an interest in this so I tested it quickly just now. /etc/courier/imapd: MAXPERIP=1 Connection 1: % telnet foo.bar.baz 143 Connected to foo.bar.baz. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS LOGINDISABLED] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc. See COPYING for distribution information. Connection 2: % telnet foo.bar.baz 143 Connected to oolong.backend.invalid. Escape character is '^]'. Connection closed by foreign host. % So I apparently cannot reproduce the failure to enforce MAXPERIP. This is 4.3-stable with courier-imap-4.1.1p2-imap_bugs Read my mail carefully. It says that clicking another folder beyond the MAXPERIP *is* honoured. One way or another, this does not apply for searches. Sam is terse. You asked about imapd forking and he answered that. Yes. He said that it will be one, except of in cases of encrypted link, which will make it two. Which is not true in this case. Uwe
Re: Courier-IMAP brings down system when searching Entire Message
Marc Balmer wrote: Indeed, I will try to test an update soon. BTW: we are using the OpenBSD/courier-imap combo for real large mailserver and don't see the problems that were described. Yes, Marc, yes, Jason, neither did I see them on the Dual-Xeon. But the Dual-Xenon with 15k-SCSI is bigger iron, and the search finishes in below a minute easily. Don't get me wrong, my concern is not on the slow little bugger to not finish a search. My concern is how far this is an exploit, a DoS, eventually via dial-up, once it is known. What I seem to observe, is that a client can start a (full text==Entire Message) search at any moment she so desires. And it also seems, that this doesn't fall under the limitations of MAXPERIP. If what I have to assume, an attacker *could* resend a search request with a higher frequency or on a larger mailbox, or (DDoS) to a system with higher load, due to distributed searches (a plurality of searches on different accounts). The critical case seems to be, when a search takes longer than the TCP-timeout of the client. I think, in the cases described in here, this simply was not the case, due to 'big iron' and a somewhat randomised access and search by the independent users. What happens if someone gobbles together a 10- or 20-liner perl script that bombards the daemon with 'Entire Search'-requests of increasing frequency? Let's say, 60 seconds, 30 seconds, 10 seconds, 3 seconds, 2 seconds, 1 second. At one moment in time, our bigger irons will also fail to come up with a full search within these shorter periods. And when the script misses the result returned, it will resend the search. And so forth. Since this search cannot be performed within these few seconds, also this second request will not result in anything, and the script will repeat the search request ad infinitum. If someone wanted to try, do a 'Entire Message' search in Thunderbird (on a slow system with a large folder), and you'll see - surprise, surprise - that once per minute the client will connect and parse a (new) search request: Once per minute the status bar displays Connecting to Sending Login information for a short time, and the number of imapd processes goes up by one. Until exhaustion, maybe. And to me it looks like a possibly exploitable behaviour. The most straightforward solution: limit the number of concurrent searches per username to one. Uwe
Re: Courier-IMAP brings down system when searching Entire Message
* Uwe Dippel wrote: Marc Balmer wrote: Indeed, I will try to test an update soon. BTW: we are using the OpenBSD/courier-imap combo for real large mailserver and don't see the problems that were described. Yes, Marc, yes, Jason, neither did I see them on the Dual-Xeon. But the Dual-Xenon with 15k-SCSI is bigger iron, and the search finishes in below a minute easily. Don't get me wrong, my concern is not on the slow little bugger to not finish a search. My concern is how far this is an exploit, a DoS, eventually via dial-up, once it is known. What I seem to observe, is that a client can start a (full text==Entire Message) search at any moment she so desires. And it also seems, that this doesn't fall under the limitations of MAXPERIP. If what I have to assume, an attacker *could* resend a search request with a higher frequency or on a larger mailbox, or (DDoS) to a system with higher load, due to distributed searches (a plurality of searches on different accounts). It will be one of your users. Looking at the logfiles you will quickly identify him. The critical case seems to be, when a search takes longer than the TCP-timeout of the client. I think, in the cases described in here, this simply was not the case, due to 'big iron' and a somewhat randomised access and search by the independent users. What happens if someone gobbles together a 10- or 20-liner perl script that bombards the daemon with 'Entire Search'-requests of increasing frequency? Let's say, 60 seconds, 30 seconds, 10 seconds, 3 seconds, 2 seconds, 1 second. At one moment in time, our bigger irons will also fail to come up with a full search within these shorter periods. And when the script misses the result returned, it will resend the search. And so forth. Since this search cannot be performed within these few seconds, also this second request will not result in anything, and the script will repeat the search request ad infinitum. If someone wanted to try, do a 'Entire Message' search in Thunderbird (on a slow system with a large folder), and you'll see - surprise, surprise - that once per minute the client will connect and parse a (new) search request: Once per minute the status bar displays Connecting to Sending Login information for a short time, and the number of imapd processes goes up by one. Until exhaustion, maybe. And to me it looks like a possibly exploitable behaviour. The most straightforward solution: limit the number of concurrent searches per username to one. I suggest that you take your concern upstream, i.e. to the courier developers. Here, we more or less only package what they provide us with. Uwe - Marc
Re: Courier-IMAP brings down system when searching Entire Message
Marc Balmer wrote: It will be one of your users. Looking at the logfiles you will quickly identify him. You came into the thread later ... it is me. :) I suggest that you take your concern upstream, i.e. to the courier developers. Here, we more or less only package what they provide us with. I know. Going in circles. I started upstream, with a few posts, but got one single answer: courier-imap does not fork. I showed my 100 processes, all trying to search an item, and all evolved from a single search of a single user. No answer was forthcoming. Only then did I start the thread in here. And in here, Alf pointed to the culprit: re-requesting searches after a specific time. I posted this upstream, but not a single answer. So what it made from this, was that there was no interest, sort of 'works for me'. But 'works for me' still is no security means. If I can DoS a 1.7 GHz/256MB box through dial-up, involuntarily, who would want to argue that larger iron could not. A new term is born: Security through processing power ;), Uwe
Re: Courier-IMAP brings down system when searching Entire Message
On Sat, Jul 26, 2008 at 11:21:30PM +0800, Uwe Dippel wrote: Marc Balmer wrote: Indeed, I will try to test an update soon. BTW: we are using the OpenBSD/courier-imap combo for real large mailserver and don't see the problems that were described. Yes, Marc, yes, Jason, neither did I see them on the Dual-Xeon. But the Dual-Xenon with 15k-SCSI is bigger iron, and the search finishes in below a minute easily. Mine has been on a PowerEdge 1650 with dual 1133 MHz Pentium III processors. Hardly what you'd call big iron. -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net/
Re: Courier-IMAP brings down system when searching Entire Message
Marc Balmer wrote: Indeed, I will try to test an update soon. I have a moderately recent Courier-MTA port somewhere that may help. I'd be glad to update it if there's a chance of committal.
Re: Courier-IMAP brings down system when searching Entire Message
[..] Alf, thanks, and thumbs up. It is not so much of a fault of OpenBSD as it is on courier-imap and thunderbird. I did have to restart Thunderbird to make mailnews.tcptimeout actually use 600 seconds, which is sick otherwise. But after some 2.5 minutes it had nicely found the string in 'Entire Message', and otherwise behaved: no forking, no too high load. It is a nice DoS-combo, what courier-imap in conjunction with Thunderbird offer here, and one doesn't even need local access. Just any remote Thunderbird client will do, and we can't prevent the user from dragdrop some tens of thousands of messages into a folder. Or, even easier, reduce mailnews.tcptimeout, and we get one new process per one second. Sick. This is not a DoS, just a configuration issue. Have you tried changing the class to something else than daemon? [..] Use smaller inboxes Get more iron:) Sorry, no. We need to do better than allowing an easy DoS. The lever must not be iron or size, it needs to be predictable and always reasonable behaviour of the softwares. read above. f.-
Re: Courier-IMAP brings down system when searching Entire Message
Federico G. Schwindt wrote: It is a nice DoS-combo, what courier-imap in conjunction with Thunderbird offer here, and one doesn't even need local access. Just any remote Thunderbird client will do, and we can't prevent the user from dragdrop some tens of thousands of messages into a folder. Or, even easier, reduce mailnews.tcptimeout, and we get one new process per one second. Sick. This is not a DoS, just a configuration issue. Have you tried changing the class to something else than daemon? Which one? I used default, and it doesn't make a difference. Okay, I didn't let it drive up loads of 80. Simply stopped around 7, once some 8 processes were running. Still, I wonder why it won't stop at 4, despite -maxperip=4 If it was a configuration problem, I'd have to create an extra, very restrictive login class for it. And then, the install message should say so. And even then, with maxproc=10 I'd get loads of close to 10, and no result forever. I'm not an email expert, but I'd think that firstly, it should actually limit the sessions per IP, not the number of processes: If you have 10 users on, they might get well 5 processes each. If there was only 1 user, he still ought not get 50 processes. Furthermore, it should limit to one search per user login at a time. Then, whatever the setting of the client, there would be one search coming to a proper end before another could be spawned. Uwe
Re: Courier-IMAP brings down system when searching Entire Message
On Fri, Jul 25, 2008 at 11:11:50PM +0800, Uwe Dippel wrote: Federico G. Schwindt wrote: It is a nice DoS-combo, what courier-imap in conjunction with Thunderbird offer here, and one doesn't even need local access. Just any remote Thunderbird client will do, and we can't prevent the user from dragdrop some tens of thousands of messages into a folder. Or, even easier, reduce mailnews.tcptimeout, and we get one new process per one second. Sick. This is not a DoS, just a configuration issue. Have you tried changing the class to something else than daemon? Which one? I used default, and it doesn't make a difference. Okay, I didn't let it drive up loads of 80. Simply stopped around 7, once some 8 processes were running. Still, I wonder why it won't stop at 4, despite -maxperip=4 If it was a configuration problem, I'd have to create an extra, very restrictive login class for it. And then, the install message should say so. And even then, with maxproc=10 I'd get loads of close to 10, and no result forever. I'm not an email expert, but I'd think that firstly, it should actually limit the sessions per IP, not the number of processes: If you have 10 users on, they might get well 5 processes each. If there was only 1 user, he still ought not get 50 processes. Furthermore, it should limit to one search per user login at a time. Then, whatever the setting of the client, there would be one search coming to a proper end before another could be spawned. I was talking about OpenBSD and cannot fork part. Clearly courier-imap is the one to blame here. f.-
Re: Courier-IMAP brings down system when searching Entire Message
On Fri, Jul 25, 2008 at 11:11:50PM +0800, Uwe Dippel wrote: I'm not an email expert, but I'd think that firstly, it should actually limit the sessions per IP, not the number of processes: You can restrict sessions per IP very easily with PF.
Re: Courier-IMAP brings down system when searching Entire Message
Stuart Henderson wrote: You can restrict sessions per IP very easily with PF. Yes, Stuart, and no. It is never good engineering practice to turn two knobs at the same time. Firstly, when opening a new folder, courier-imap honours its limitation gracefully. Second, it would limit the communication channel between daemon and client. An innocuous 'go to another folder' or even 'stop search' (agree, it doesn't work as of now, but it should!) would be dropped. It would still not deliver search results, because it would spawn N identical searches (N defined by pf). I could try, and I would as well have to look into it myself, because I don't know how long the timed-out earlier requests would still count as 'link', since the client seems to consider them dead after Thunderbirds 60-second mailnews.timeout. Now, thinking of it, if you don't mind following the line of arguments: even worse: - the links are considered 'timed-out', then pf doesn't limit the number of searches - the links are considered 'active by pf, then any request by the client of moving away and going to another folder is dropped Couldn't be worse, could it!? What bugs me as well: I have been discussing the matter on the courier-imap list, but nobody answered, and the upstream writer never wrote anything but that line of 'courier-imap does not fork'. Over the years, I have entrusted by IMAP-clients to courier-imap thinking it was well-written. I start to have my doubts. Can anyone with dovecot on a very tiny box confirm or deny that it acts differently? One needs to have more mails in a box than the daemon can search within the 60 seconds time-out. Then, in the status bar of Thunderbird, it will say something like 'Connecting to imap.example.com ...' once per minute. Let it go for some minutes, and then observe the resource usage on the server. If dovecot is fine, I'll change immediately. :) Uwe
Re: Courier-IMAP brings down system when searching Entire Message
Uwe,
I'm running and old version of courier-imap-1.7.2p0 mostly with POP3
accounts, some imap too.
This patch i found on the courier-imap mailing list some years ago
helped me reduce the average
load of the machine, and was able to accept more accounts. I don't know
if it might be helpfull to you.
Also add imapproxy, that will reduce some load too.
Anyway, i'm having big load again due to much more accounts being added.
Currently it's holding
over 7000 email accounts on a pentium4 2.80ghz. The average load reaches
tops of 50 , and
is usually at 15 to 20. At night it comes down to a load average of 2.
The load average is mostly to disk usage, so i'm planning on upgrading
this to OpenBSD 4.3 as
soon as possible, and run the newest courier-imap software. Once there i
will split accounts on
two servers using Perdition or the courier-imap built-in proxy
aggregator. I will do this as soon i
get new hardware, i'm looking for a 1U IBM Server which i'd like to use
for this purpose.
Does anyone has experience with Perdition and/or the built-in proxy
aggregator? Which
one has best performance?
And, different subject , what 1U IBM Servers are supported by OpenBSD? I
had never had
the chance to get one to test. Reccomendations?
Oh yes, here's the patch i mentioned you
--begin-cut
--- pop3dserver.c Mon Apr 28 15:40:19 2003
+++ /usr/ports/mail/courier-imap/pop3dserver.c Tue Jul 3 16:13:31 2007
@@ -314,6 +314,10 @@
struct msglist **prev_list;
int savesizes=0;
+// PATCH: dont scan every file, ignore rfc like qmail-pop3d
+char *p;
+long s;
+
if (msglist_cnt == 0) return;
if ((msglist_a=(struct msglist **)malloc(
@@ -349,6 +353,13 @@
}
else
{
+
+ // PATCH: dont scan every file, ignore rfc like qmail-pop3d
+ // msglist_a[i]-filename =~
1018021353.83146_0.host.dom,S=9249:2,S
+ if( (p=strstr(msglist_a[i]-filename,,S=))
(s=atol(p+3))0 )
+ msglist_a[i]-size = (off_t)s;
+ else// fallback to slow rfc-conformant scan
+
calcsize(msglist_a[i]);
savesizes=1;
}
--end-cut--
Uwe Dippel escribió:
Stuart Henderson wrote:
You can restrict sessions per IP very easily with PF.
Yes, Stuart,
and no. It is never good engineering practice to turn two knobs at the
same time. Firstly, when opening a new folder, courier-imap honours
its limitation gracefully. Second, it would limit the communication
channel between daemon and client. An innocuous 'go to another folder'
or even 'stop search' (agree, it doesn't work as of now, but it
should!) would be dropped. It would still not deliver search results,
because it would spawn N identical searches (N defined by pf). I could
try, and I would as well have to look into it myself, because I don't
know how long the timed-out earlier requests would still count as
'link', since the client seems to consider them dead after
Thunderbirds 60-second mailnews.timeout.
Now, thinking of it, if you don't mind following the line of
arguments: even worse:
- the links are considered 'timed-out', then pf doesn't limit the
number of searches
- the links are considered 'active by pf, then any request by the
client of moving away and going to another folder is dropped
Couldn't be worse, could it!?
What bugs me as well: I have been discussing the matter on the
courier-imap list, but nobody answered, and the upstream writer never
wrote anything but that line of 'courier-imap does not fork'. Over the
years, I have entrusted by IMAP-clients to courier-imap thinking it
was well-written. I start to have my doubts.
Can anyone with dovecot on a very tiny box confirm or deny that it
acts differently? One needs to have more mails in a box than the
daemon can search within the 60 seconds time-out. Then, in the status
bar of Thunderbird, it will say something like 'Connecting to
imap.example.com ...' once per minute. Let it go for some minutes, and
then observe the resource usage on the server.
If dovecot is fine, I'll change immediately. :)
Uwe
Re: Courier-IMAP brings down system when searching Entire Message
On Saturday 26 July 2008 00:30:30 Jason Dixon wrote: On Fri, Jul 25, 2008 at 10:53:52PM -0400, [EMAIL PROTECTED] wrote: I understand your frustration but I think there's something else going on here... Agreed. I've been using Courier-IMAP on OpenBSD going back at least 5 years. The last few have seen a *lot* of Mail.app and Thunderbird users beat on my server constantly with nary a problem. As a side note I have wondered why the Courier IMAP port is almost 2.5 years out of date.. there have been a few bug fixes in that period of time that would definitely warant an update. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Courier-IMAP brings down system when searching Entire Message
On Mon, Jul 21, 2008 at 12:47:25AM +0800, Uwe Dippel wrote: ... when a user has close to 20.000 mails in the Inbox. It chucks out courier-imap when more processes are involved than allowed: (This is after some 3 hours of increasing load, searching around 1 GB in cur: load averages: 84.11, 83.12, 80.7721:04:01 10218 udippel -140 3504K 4336K sleepinode 0:00 0.88% imapd 5550 udippel -50 3372K 1972K sleepgetblk1:31 0.15% imapd 2566 udippel -140 3368K 2080K sleepinode 0:54 0.15% imapd 2623 udippel -140 3296K 1892K sleepinode 0:49 0.15% imapd 25935 udippel -50 3504K 1872K sleepgetblk0:47 0.15% imapd 21550 udippel -140 3356K 1816K sleepinode 0:27 0.15% imapd 30191 udippel -140 3256K 1944K sleepinode 0:24 0.15% imapd 14507 udippel -140 3284K 2080K sleepinode 0:54 0.10% imapd 28879 udippel -50 3416K 1976K sleepgetblk0:46 0.10% imapd 21472 udippel -140 3368K 1972K sleepinode 1:15 0.05% imapd 8937 udippel -50 3404K 1792K sleepgetblk0:48 0.05% imapd 20352 udippel -140 3492K 1940K sleepinode 0:15 0.05% imapd 11819 udippel -140 3268K 1956K sleepinode 0:14 0.05% imapd 15284 udippel -140 3384K 1820K sleepinode 1:38 0.00% imapd 21101 udippel -140 3320K 1892K sleepinode 1:34 0.00% imapd [,,,] A moment later the thing is done: Mon Jun 30 21:09:35 SGT 2008 /bin/ksh: cannot fork - try again I wrote to the courier-imap list, and Sam Varshavchik wrote: imapd does not fork any processes. There is always a single process for each login (and a second, if encryption is used). I assume Sam knows. But why does OpenBSD fork like rabbits when a user searches 'Entire Message' in a large mailbox? Uwe We had a similar problem on our mailserver and thunderbird as mail client. Basicly, thunderbird has a certain timeout value for searches. If thunderbird doesn't get a result for a search after timeout seconds, it assumes some error and requests the search again. new search - new fork - more load and so on. Other MUAs may have similar settings. Solution: Search and set the timeout value in thunderbird's config editor (IRC mailnews.tcptimeout, defaults to 60) Use smaller inboxes Get more iron:) Alf
Re: Courier-IMAP brings down system when searching Entire Message
Alf Schlichting wrote: We had a similar problem on our mailserver and thunderbird as mail client. Basicly, thunderbird has a certain timeout value for searches. If thunderbird doesn't get a result for a search after timeout seconds, it assumes some error and requests the search again. new search - new fork - more load and so on. Other MUAs may have similar settings. Solution: Search and set the timeout value in thunderbird's config editor (IRC mailnews.tcptimeout, defaults to 60) Alf, thanks, and thumbs up. It is not so much of a fault of OpenBSD as it is on courier-imap and thunderbird. I did have to restart Thunderbird to make mailnews.tcptimeout actually use 600 seconds, which is sick otherwise. But after some 2.5 minutes it had nicely found the string in 'Entire Message', and otherwise behaved: no forking, no too high load. It is a nice DoS-combo, what courier-imap in conjunction with Thunderbird offer here, and one doesn't even need local access. Just any remote Thunderbird client will do, and we can't prevent the user from dragdrop some tens of thousands of messages into a folder. Or, even easier, reduce mailnews.tcptimeout, and we get one new process per one second. Sick. With the larger part of the blame on courier-imap, because it does not seem to honour its own limit of 4 or 5 sessions per IP: # Maximum number of connections to accept from the same IP address MAXPERIP=4 Also, Thunderbird surely needs two dials: mailnews.tcptimeout is pretty much okay with 60 seconds default, but its mistake is to consider a search-still-in-the-making as 'connection failure'. And to spawn a new one. Use smaller inboxes Get more iron:) Sorry, no. We need to do better than allowing an easy DoS. The lever must not be iron or size, it needs to be predictable and always reasonable behaviour of the softwares. Thanks again, would have never found the problem on my own! Uwe
