Re: imapd mem exhausted
On Thu, 30 Oct 2003, Patrick Welche wrote: I just tried the same test of opening an imap connection and APPENDing lots of mail to various new mailboxes in there (Courier - cyrus move), with new cvs code, so the 2.2 line of code, and still: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 3518 cyrus 20 124M 125M select 1:05 2.98% 2.98% imapd I watch the imapd process grow... There must be a leak somewhere, no? I've run loops of APPEND commands in an imapd running under purify and it detects no leaks (though, it found a number of other errors that I am cleaning up now). -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: imapd mem exhausted
I just tried the same test of opening an imap connection and APPENDing lots of mail to various new mailboxes in there (Courier - cyrus move), with new cvs code, so the 2.2 line of code, and still: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 3518 cyrus 20 124M 125M select 1:05 2.98% 2.98% imapd I watch the imapd process grow... There must be a leak somewhere, no? Fatal error: Virtual memory exhausted Cheers, Patrick
imapd mem exhausted
Sep 24 09:50:30 imp imap[2030]: executed Sep 24 09:50:30 imp imapd[2030]: accepted connection Sep 24 09:50:47 imp imapd[2030]: accepted connection ... Sep 24 09:51:15 imp imapd[2030]: accepted connection ... Sep 24 10:26:23 imp imapd[2030]: Fatal error: Virtual memory exhausted That was then a reasonably long lasting connection that was just doing CREATE followed by stacks of APPEND non-stop. There was only me as a user (only 1 imapd). I suppose that is rather different to the usual usage pattern of lots of connections doing a few things. After a few minutes I see: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 19526 cyrus 2035M 37M select 0:10 0.93% 0.93% imapd Is this a memory leak, or are the messages being stored in memory ie., it's meant to happen? Is there a simple way of telling master to give imapd some more memory when it forks? Cheers, Patrick
Re: imapd mem exhausted
On Wed, 24 Sep 2003, Scott Adkins wrote: 3) The problem was characterized as an mmap() problem on Tru64 because our mailboxes.db file is about 27MB in size. However, we are seeing the sizes jump to 30-32MB in size, and our mailboxes.db file is still only about 27MB in size. *shrugs* mailboxes.db + cyrus.index and cyrus.cache file for some large mailboxes? -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: imapd mem exhausted
What version of Cyrus are you using? We are using 2.2b1 here, and are experiencing something similar with memory issues for IMAP. I brought up the discussion a few weeks ago about it and it was characterized as some kind of mmap() weirdness on our Tru64 platform, which I don't really think is the case. I will look in our logs to see if we are seeing a similar issue with a lot of accepted connections as you are. My experiences are as follows with the problem: 1) It seems that the RSS footprint of the process increases when a user makes a connection (from observation). 2) Most of the time, we may see a process jump up to 26 or 27MB in size (and lately, we are now seeing them jump up to 30-32MB in size), but typically go back down to several hundred KB shortly thereafter. If the system starts to slow down for whatever reason, the time for a process to shrink increases, and we have more processes at the bigger RSS size and run the risk of exhausting memory and swap. 3) The problem was characterized as an mmap() problem on Tru64 because our mailboxes.db file is about 27MB in size. However, we are seeing the sizes jump to 30-32MB in size, and our mailboxes.db file is still only about 27MB in size. *shrugs* 4) Restarting the Cyrus server has a dramatic effect on memory usage by putting all the processes back at the base several hundred KB size. It takes quite a while before we see the issue #2 become a problem again (even on a busy server). Anyways, I have been looking at this problem over that last several days and my gut feeling has been leaning towards the on-connection route... which certainly jives with your message below... Scott --On Wednesday, September 24, 2003 12:02 PM +0100 Patrick Welche [EMAIL PROTECTED] wrote: Sep 24 09:50:30 imp imap[2030]: executed Sep 24 09:50:30 imp imapd[2030]: accepted connection Sep 24 09:50:47 imp imapd[2030]: accepted connection Sep 24 09:51:15 imp imapd[2030]: accepted connection Sep 24 10:26:23 imp imapd[2030]: Fatal error: Virtual memory exhausted That was then a reasonably long lasting connection that was just doing CREATE followed by stacks of APPEND non-stop. There was only me as a user (only 1 imapd). I suppose that is rather different to the usual usage pattern of lots of connections doing a few things. After a few minutes I see: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 19526 cyrus 2035M 37M select 0:10 0.93% 0.93% imapd Is this a memory leak, or are the messages being stored in memory ie., it's meant to happen? Is there a simple way of telling master to give imapd some more memory when it forks? Cheers, Patrick -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: imapd mem exhausted
On Wed, Sep 24, 2003 at 10:48:39AM -0400, Scott Adkins wrote: What version of Cyrus are you using? CVS HEAD from yesterday, which is called 2.1.15. We are using 2.2b1 here, and are experiencing something similar with memory issues for IMAP. I brought up the discussion a few weeks ago about it and it was characterized as some kind of mmap() weirdness on our Tru64 platform, which I don't really think is the case. I will look in our logs to see if we are seeing a similar issue with a lot of accepted connections as you are. I actually only had 1 connection - I should have made it clear that those were *all* the accepts in imapd.log (from my ... it looks as though the line is repeated - sorry) - it was just to show that nothing much was going on apart from that 1 imapd running for about 35mins until that one imapd exhausted virtual memory. My experiences are as follows with the problem: 1) It seems that the RSS footprint of the process increases when a user makes a connection (from observation). Yes - and for me, essentially the more messages I append, the greater the increase. It doesn't matter which mailbox, nor whether the selected mailbox changes. It seems to be a per message increase. 2) Most of the time, we may see a process jump up to 26 or 27MB in size (and lately, we are now seeing them jump up to 30-32MB in size), but typically go back down to several hundred KB shortly thereafter. If the system starts to slow down for whatever reason, the time for a process to shrink increases, and we have more processes at the bigger RSS size and run the risk of exhausting memory and swap. 3) The problem was characterized as an mmap() problem on Tru64 because our mailboxes.db file is about 27MB in size. However, we are seeing the sizes jump to 30-32MB in size, and our mailboxes.db file is still only about 27MB in size. *shrugs* Hmm, I'm using NetBSD, and my mailboxes.db is tiny (20k). Remember, I was just testing with 1 user. There are 2 users total in the system. 4) Restarting the Cyrus server has a dramatic effect on memory usage by putting all the processes back at the base several hundred KB size. It takes quite a while before we see the issue #2 become a problem again (even on a busy server). Anyways, I have been looking at this problem over that last several days and my gut feeling has been leaning towards the on-connection route... which certainly jives with your message below... For me it seems there is a memory leak in append :/ Maybe for you each imap connection receives another few messages = they grow? So memory isn't even freed between connections? Cheers, Patrick
Re: imapd mem exhausted
On Wed, Sep 24, 2003 at 10:58:46AM -0400, Rob Siemborski wrote: On Wed, 24 Sep 2003, Scott Adkins wrote: 3) The problem was characterized as an mmap() problem on Tru64 because our mailboxes.db file is about 27MB in size. However, we are seeing the sizes jump to 30-32MB in size, and our mailboxes.db file is still only about 27MB in size. *shrugs* mailboxes.db + cyrus.index and cyrus.cache file for some large mailboxes? Hmm.. Starting from a totally empty user, not even an inbox in sight, and using that 1 imapd process which grew to 37Mb to CREATE and APPEND repeatedly, Oh! # du -ck . (Inside /var/spool/user/userid) 514 ./Sent/2003/08-Aug ... 38104 . 38104 total PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 19526 cyrus 2035M 37M select 0:10 0.93% 0.93% imapd (I killed it just after) Now, why would the total quantity of data written to disk in that session magically tally with the process' size.. I'd better try again in case this is coincidence.. That figure obviously includes not only the message files, but also cyrus.index cyrus.cache for each mailbox, the biggest being -rw--- 1 cyrus mail 409268 Sep 24 11:55 ./Sent/2003/03-Mar/cyrus.cache which isn't much.. Cheers, Patrick
