Re: Too many open files
Rob McMahon wrote: Shelley Waltz wrote: Oddly enough I had the identical issue today while trying to remove a bunch of users. I am running cyrus-imapd-2.3.7-2.el5 on RHEL5. I used cyradm and did sam user.x anyone c dm user.x This worked fine for about 20 users, then I got the following in maillog ... Nov 3 10:35:28 rr imap[16539]: IOERROR: opening quota file /var/lib/imap/quota/x/user.x: Too many open files Nov 3 10:35:28 rr imap[16539]: DBERROR: error fetching user.x: cyrusdb error Nov 3 10:35:28 rr imap[16539]: Deleted mailbox user.x If I look in /var/spool/imap/x/user/ the directory and folders for user x are still there. If I try to remove the mailbox once again using cyradm, it claims the mailbox does not exist. How does one resolve this issue safely? I have the same problem with a few mailboxes myself, and don't know how to clean it up. To avoid the problem in the first place, if you have just a few (tens ? couple of hundred) users beginning with the same prefix, I think you can avoid the problem in the first place by putting e.g. imap cmd=imapd listen=imap prefork=0 maxfds=4096 in /etc/cyrus.conf. I suspect the underlying cause is this chunk from lib/cyrusdb_quotalegacy.c(foreach): /* strip off the qr specific path and replace with pattern */ sprintf(strstr(quota_path, FNAME_QUOTADIR) + strlen(FNAME_QUOTADIR), ?/%s*, prefix); /* search for the quotaroots */ glob(quota_path, GLOB_NOSORT, NULL, globbuf); That pattern on our boxes can match thousands of users. I think it needs to be more careful and go looking for `prefix' and `prefix.*', but I haven't dug into it far enough yet. I changed the maxfds to 4096 and will try removing mailboxes once again. From my previous attempt described above, I still have /var/spool/imap/x/user/x with all the folders from the account in the directory. There appears to be no other trace of the account. Is it safe to just rm the var/spool/imap/x/user/x directory contents? I have a replica using sync_client which shows no trace whatsoever of the account, so it seems a safe bet, yes? thanks S Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
On 04 Nov 2008, at 07:39, Rob McMahon wrote: I suspect the underlying cause is this chunk from lib/cyrusdb_quotalegacy.c(foreach): /* strip off the qr specific path and replace with pattern */ sprintf(strstr(quota_path, FNAME_QUOTADIR) + strlen (FNAME_QUOTADIR), ?/%s*, prefix); /* search for the quotaroots */ glob(quota_path, GLOB_NOSORT, NULL, globbuf); That pattern on our boxes can match thousands of users. I think it needs to be more careful and go looking for `prefix' and `prefix.*', but I haven't dug into it far enough yet. That code's been rewritten in 2.3.13 to read the quota directory instead. You might be able to back port the change: https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/ cyrusdb_quotalegacy.c.diff?r1=1.18;r2=1.19 For the record, it wasn't rewritten because of the bug you identified, but for an unrelated problem. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Shelley Waltz wrote: Oddly enough I had the identical issue today while trying to remove a bunch of users. I am running cyrus-imapd-2.3.7-2.el5 on RHEL5. I used cyradm and did sam user.x anyone c dm user.x This worked fine for about 20 users, then I got the following in maillog ... Nov 3 10:35:28 rr imap[16539]: IOERROR: opening quota file /var/lib/imap/quota/x/user.x: Too many open files Nov 3 10:35:28 rr imap[16539]: DBERROR: error fetching user.x: cyrusdb error Nov 3 10:35:28 rr imap[16539]: Deleted mailbox user.x If I look in /var/spool/imap/x/user/ the directory and folders for user x are still there. If I try to remove the mailbox once again using cyradm, it claims the mailbox does not exist. How does one resolve this issue safely? I have the same problem with a few mailboxes myself, and don't know how to clean it up. To avoid the problem in the first place, if you have just a few (tens ? couple of hundred) users beginning with the same prefix, I think you can avoid the problem in the first place by putting e.g. imap cmd=imapd listen=imap prefork=0 maxfds=4096 in /etc/cyrus.conf. I suspect the underlying cause is this chunk from lib/cyrusdb_quotalegacy.c(foreach): /* strip off the qr specific path and replace with pattern */ sprintf(strstr(quota_path, FNAME_QUOTADIR) + strlen(FNAME_QUOTADIR), ?/%s*, prefix); /* search for the quotaroots */ glob(quota_path, GLOB_NOSORT, NULL, globbuf); That pattern on our boxes can match thousands of users. I think it needs to be more careful and go looking for `prefix' and `prefix.*', but I haven't dug into it far enough yet. Cheers, Rob -- E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037 Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Rob McMahon wrote:
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
and indeed this process has hit the default 256 limit, most of which
seem to be quota files. Other issues were manifesting, IMAP commands
never completing for instance, so I had to kill this process, which in
turn caused the master process to restart. Not good.
I think I've just found an answer to this. Looking at the files open by
one of these processes (I'd upped descriptors to 2048), it appears that
if you try to delete, say, user.a, the IMAP process ends up quota files
for every user beginning with a a ... sounds mad but that is what I've
observed, and if I'm careful about not deleting short names it goes
through okay. I'll try to investigate further. I know this is old
code, and the algorithms have probably changed by now.
Cheers,
Rob
--
E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Oddly enough I had the identical issue today while trying to remove a
bunch of users. I am running cyrus-imapd-2.3.7-2.el5 on RHEL5.
I used cyradm and did
sam user.x anyone c
dm user.x
This worked fine for about 20 users, then I got the following in maillog ...
Nov 3 10:35:28 rr imap[16539]: IOERROR: opening quota file
/var/lib/imap/quota/x/user.x: Too many open files
Nov 3 10:35:28 rr imap[16539]: DBERROR: error fetching user.x: cyrusdb error
Nov 3 10:35:28 rr imap[16539]: Deleted mailbox user.x
If I look in /var/spool/imap/x/user/ the directory and folders for user
x are still there. If I try to remove the mailbox once again using
cyradm, it claims the mailbox does not exist.
How does one resolve this issue safely?
S.Waltz
Rob McMahon wrote:
Rob McMahon wrote:
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open
files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
and indeed this process has hit the default 256 limit, most of which
seem to be quota files. Other issues were manifesting, IMAP commands
never completing for instance, so I had to kill this process, which in
turn caused the master process to restart. Not good.
I think I've just found an answer to this. Looking at the files open by
one of these processes (I'd upped descriptors to 2048), it appears that
if you try to delete, say, user.a, the IMAP process ends up quota files
for every user beginning with a a ... sounds mad but that is what I've
observed, and if I'm careful about not deleting short names it goes
through okay. I'll try to investigate further. I know this is old
code, and the algorithms have probably changed by now.
Cheers,
Rob
--
E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Rob McMahon wrote: Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR: opening quota file /var/imap/quota/V/user.??.Trash: Too many open files Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning: cannot open /etc/hosts.deny: Too many open files and indeed this process has hit the default 256 limit, most of which seem to be quota files. Other issues were manifesting, IMAP commands never completing for instance, so I had to kill this process, which in turn caused the master process to restart. Not good. Has anyone got a handle on this bug ? It doesn't seem like a slow descriptor leak: I've just deleted 250 users in chunks of 20, checking carefully on fd usage between each batch, and then on the final 20, blatt, it did it again. It feels like it's some sort of house-keeping function kicking in. I can go around hunting for, and deleting, quota roots. I can up maxfds to try to alleviate the problem. But, I've got no confidence, I've got 4,000 users to delete, and I want to completely automate it, and rely on the service staying up once the deed is done. Rob -- E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037 Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
I believe this is due to a bug in the quota code that doesn't close the quota file after deleting a mailbox. I consistently get this behavior trying to delete a user that has more than ~250 mailboxes. (ulimit -f is 255) If you delete the user's quota first you shouldn't get this error, at least if you're running into the same bug I've seen. Assuming you've patched cyrus to allow quota roots to be removed, sq user.username remove 0 will delete the quota root. I spent a couple hours trying to track the bug down to no avail. On 8/25/07, Alain Spineux [EMAIL PROTECTED] wrote: Maybe this is because your are accessing multiple mailbox through the same and unique connection. At each mailbox it should open some files related to the mailbox and never close them ! The use is to connect to a unique mailbox and read mails and browse subfolders, and then always use the same related files. As a workaround you could try to logout and login before to switch mailbox. Regards On 8/25/07, Dale Ghent [EMAIL PROTECTED] wrote: On Aug 24, 2007, at 11:49 AM, Rob McMahon wrote: Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC, around 115,000 mailboxes. After deleting just a few, I start getting errors like Must login first, System I/O Error (sorry, I haven't got the exact messages to cut paste). Looking in the logs I see: We (Rob Banz and I) are seeing the same thing on Solaris 10 x86. We started our mass mail migration this past monday and a number of user migrations have failed due to the same error. We haven't found the cause yet, but to get around it for now we cranked up the per-process FD limit and limited the number of requests a imapd is allowed to handle. It looks like a file descriptor leak of some sort. So far that has been a decent work-around. /dale -- Dale Ghent Specialist, Storage and UNIX Systems UMBC - Office of Information Technology ECS 201 - x51705 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- -Adam Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
On Aug 24, 2007, at 11:49 AM, Rob McMahon wrote: Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC, around 115,000 mailboxes. After deleting just a few, I start getting errors like Must login first, System I/O Error (sorry, I haven't got the exact messages to cut paste). Looking in the logs I see: We (Rob Banz and I) are seeing the same thing on Solaris 10 x86. We started our mass mail migration this past monday and a number of user migrations have failed due to the same error. We haven't found the cause yet, but to get around it for now we cranked up the per-process FD limit and limited the number of requests a imapd is allowed to handle. It looks like a file descriptor leak of some sort. So far that has been a decent work-around. /dale -- Dale Ghent Specialist, Storage and UNIX Systems UMBC - Office of Information Technology ECS 201 - x51705 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Maybe this is because your are accessing multiple mailbox through the same and unique connection. At each mailbox it should open some files related to the mailbox and never close them ! The use is to connect to a unique mailbox and read mails and browse subfolders, and then always use the same related files. As a workaround you could try to logout and login before to switch mailbox. Regards On 8/25/07, Dale Ghent [EMAIL PROTECTED] wrote: On Aug 24, 2007, at 11:49 AM, Rob McMahon wrote: Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC, around 115,000 mailboxes. After deleting just a few, I start getting errors like Must login first, System I/O Error (sorry, I haven't got the exact messages to cut paste). Looking in the logs I see: We (Rob Banz and I) are seeing the same thing on Solaris 10 x86. We started our mass mail migration this past monday and a number of user migrations have failed due to the same error. We haven't found the cause yet, but to get around it for now we cranked up the per-process FD limit and limited the number of requests a imapd is allowed to handle. It looks like a file descriptor leak of some sort. So far that has been a decent work-around. /dale -- Dale Ghent Specialist, Storage and UNIX Systems UMBC - Office of Information Technology ECS 201 - x51705 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Alain Spineux aspineux gmail com May the sources be with you Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Too many open files
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
and indeed this process has hit the default 256 limit, most of which
seem to be quota files. Other issues were manifesting, IMAP commands
never completing for instance, so I had to kill this process, which in
turn caused the master process to restart. Not good.
Any suggestions on this ? I could up maxfds to, say, 1024 for imap, but
then would it just hit that limit instead. No other IMAP process seems
to use more than around 16 descriptors.
Cheers,
Rob
--
E-Mail: [EMAIL PROTECTED] PHONE: +44 24 7652 3037
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
On Fri, Aug 24, 2007 at 04:49:10PM +0100, Rob McMahon wrote:
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
Same thing here with cyrus-imapd-2.3.8 on Solaris 10 SPARC.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
Mine was like this:
Aug 23 09:43:46 castor imap[20105]: [ID 136705 local6.error] IOERROR: opening
/imap/spool/K/user/xx/cyrus.header: Too many open files
The problem is intermittent, only happening with certain imapd
processes. I've noticed it when processing large batches of mailboxes
from a perl script. When I rerun the batch, with a different imapd,
the error doesn't appear. It almost never happens in normal IMAP use.
I don't know how to track it down in these circumstances.
--
-Gary Mills--Unix Support--U of M Academic Computing and Networking-
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
syncserver crashes with Too many open files
In implementing the scheme I discussed a few days ago I came across this error. Because syncserver needs to delete all the mailboxes first, it crashes with Too many open files when the user has a lot of messages. I'm unsure if this is a cyrus or system error, but my guess is cyrus. Does syncserver take the maxfds argument? It takes a restart of the imapserver to get it back running properly. syncserver[25602]: Deleted mailbox user.xxx.yyy.zzz syncserver[25602]: Deleted mailbox user.xxx.bbb.ccc etc syncserver[25602]: IOERROR: creating quota file /var/imap/quota/m/user.xxx.NEW: Too many open files syncserver[25602]: DBERROR: error storing user.xxx: cyrusdb error syncserver[25602]: LOSTQUOTA: unable to record free of 35066 bytes in quota (null) syncserver[25602]: Deleted mailbox user.xxx.ddd.eee fff syncserver[25602]: IOERROR: opening quota file /var/imap/quota/m/user.xxx: Too many open files Thank you, Per olof Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
Solved. Instead of user.username, you delete +user.username. Kai Wang wrote: I actually changed the nofile by editing the /etc/security/limits.conf file. It was only 1024. I have increased it to be bigger than the total number of messages the test user has. The test user account contains 760+ folders and 76000+ messages. I restore the user and deleted it again. The problem remains. [EMAIL PROTECTED] ~]# su - cyrus -bash-3.00$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 131072 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 59390 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Zachariah Mully wrote: On Mon, 2006-06-12 at 14:51 -0600, Kai Wang wrote: Greetings. I'm running Cyrus on Redhat AS4. cyrus complains about too many open files when I try to delete a user with large number of mailboxes/messages. I increased /proc/sys/fs/file-max and max number of open files a few times. The errors still happen. Actually the user is successfully deleted. I wonder if I should have any concern about these error messages. [EMAIL PROTECTED] ~]# cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 2) [EMAIL PROTECTED] ~]# uname -a Linux lms2.acs.ucalgary.ca 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] ~]# ulimit -a Under debian with the speedblue.org debs, cyrus imapd runs under the cyrus user, so you'll want to adjust the file-max for that user, not just for root (as you did). Edit /etc/security/limits.conf Z -- Kai Wang System Services Information Technologies, University of Calgary, 2500 University Drive, N.W., Calgary, Alberta, Canada T2N 1N4 Phone (403) 220-2423, Fax (403) 282-9361 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Too many open files
Greetings. I'm running Cyrus on Redhat AS4. cyrus complains about too many open files when I try to delete a user with large number of mailboxes/messages. I increased /proc/sys/fs/file-max and max number of open files a few times. The errors still happen. Actually the user is successfully deleted. I wonder if I should have any concern about these error messages. [EMAIL PROTECTED] ~]# cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 2) [EMAIL PROTECTED] ~]# uname -a Linux lms2.acs.ucalgary.ca 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 16384 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 59390 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [EMAIL PROTECTED] ~]# cat /proc/sys/fs/file-max 2995744 Jun 12 11:50:11 lms2 imap[3320]: IOERROR: opening quota file /var/lib/imap/quota/w/user.wangtest: Too many open files Jun 12 11:50:11 lms2 imap[3320]: DBERROR: error fetching user.wangtest: cyrusdb error Jun 12 11:50:11 lms2 imap[3320]: IOERROR: opening quota file /var/lib/imap/quota/w/user.wangtest: Too many open files Jun 12 11:50:11 lms2 imap[3320]: DBERROR: error fetching user.wangtest: cyrusdb error Jun 12 11:50:11 lms2 imap[3320]: IOERROR: opening quota file /var/lib/imap/quota/w/user.wangtest: Too many open files Jun 12 11:50:11 lms2 imap[3320]: DBERROR: error fetching user.wangtest: cyrusdb error Jun 12 11:50:11 lms2 imap[3320]: IOERROR: opening quota file /var/lib/imap/quota/w/user.wangtest: Too many open files Jun 12 11:50:11 lms2 imap[3320]: DBERROR: error fetching user.wangtest: cyrusdb error Jun 12 11:50:11 lms2 imap[3320]: LOSTQUOTA: unable to remove quota root user.wangtest for user.wangtest: Mailbox does not exist Thanks -- Kai Wang System Services Information Technologies, University of Calgary, 2500 University Drive, N.W., Calgary, Alberta, Canada T2N 1N4 Phone (403) 220-2423, Fax (403) 282-9361 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
On Mon, 2006-06-12 at 14:51 -0600, Kai Wang wrote: Greetings. I'm running Cyrus on Redhat AS4. cyrus complains about too many open files when I try to delete a user with large number of mailboxes/messages. I increased /proc/sys/fs/file-max and max number of open files a few times. The errors still happen. Actually the user is successfully deleted. I wonder if I should have any concern about these error messages. [EMAIL PROTECTED] ~]# cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 2) [EMAIL PROTECTED] ~]# uname -a Linux lms2.acs.ucalgary.ca 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] ~]# ulimit -a Under debian with the speedblue.org debs, cyrus imapd runs under the cyrus user, so you'll want to adjust the file-max for that user, not just for root (as you did). Edit /etc/security/limits.conf Z Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
I actually changed the nofile by editing the /etc/security/limits.conf file. It was only 1024. I have increased it to be bigger than the total number of messages the test user has. The test user account contains 760+ folders and 76000+ messages. I restore the user and deleted it again. The problem remains. [EMAIL PROTECTED] ~]# su - cyrus -bash-3.00$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 131072 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 59390 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Zachariah Mully wrote: On Mon, 2006-06-12 at 14:51 -0600, Kai Wang wrote: Greetings. I'm running Cyrus on Redhat AS4. cyrus complains about too many open files when I try to delete a user with large number of mailboxes/messages. I increased /proc/sys/fs/file-max and max number of open files a few times. The errors still happen. Actually the user is successfully deleted. I wonder if I should have any concern about these error messages. [EMAIL PROTECTED] ~]# cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 2) [EMAIL PROTECTED] ~]# uname -a Linux lms2.acs.ucalgary.ca 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] ~]# ulimit -a Under debian with the speedblue.org debs, cyrus imapd runs under the cyrus user, so you'll want to adjust the file-max for that user, not just for root (as you did). Edit /etc/security/limits.conf Z -- Kai Wang System Services Information Technologies, University of Calgary, 2500 University Drive, N.W., Calgary, Alberta, Canada T2N 1N4 Phone (403) 220-2423, Fax (403) 282-9361 Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
too many open files problem.
Hello guys, cyrus is complaining about too many open files when I try to rename(move between servers) or delete a large number of mailboxes. I have a script that connects to the cyrus master server and issue renames or deletes. My script can be configured to close and reopen the connection after every X renames/deletes but that doesnt seem to help... even if I use 1 different connection for every rename/delete I will get the too many open files error after some time. When moving mailboxes, I get the errors on the backend server Im moving the mailboxes from. cyrus/imap[30082]: IOERROR: creating quota file /imap/conf/quota/H/user.658729.NEW: Too many open files cyrus/imap[30082]: DBERROR: error storing user.658729: cyrusdb error cyrus/imap[30082]: IOERROR: opening quota file /imap/conf/quota/H/user.658729: Too many open files cyrus/imap[30082]: DBERROR: error fetching user.658729: cyrusdb error cyrus/imap[30082]: IOERROR: open on /imap/mboxes/H/user/658729/1.: Too many open files cyrus is started with ulimit -S -n 100 Regards, João Assad --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAPD-2.2 imapd: Too many open files
On Mon, 24 Feb 2003, Mike O'Rourke wrote: What do you want me to try again? Resend the post, or resetting line 622 of master.c to s-maxfds? The strace included in the original post was on the initial instance of an imapd prefork. Initially, I had prefork set to 0 in cyrus.conf, and could never catch the Too many open files error, so I set prefork to 1 and strace was able to attach to the process. Try the current CVS. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: IMAPD-2.2 imapd: Too many open files
On Mon, 24 Feb 2003, Mike O'Rourke wrote: What do you want me to try again? Resend the post, or resetting line 622 of master.c to s-maxfds? The strace included in the original post was on the initial instance of an imapd prefork. Initially, I had prefork set to 0 in cyrus.conf, and could never catch the Too many open files error, so I set prefork to 1 and strace was able to attach to the process. Try the current CVS. -Rob I retrieved the CVS from today, and the too many open files condition is fixed. Thanks. I still have to change hash_insert to cyrus_hash_insert though, :-( but I have a diff at hand to do that with every new download! ;-) Mike.
Re: IMAPD-2.2 imapd: Too many open files
On Tue, 25 Feb 2003, Mike O'Rourke wrote: I still have to change hash_insert to cyrus_hash_insert though, :-( but I have a diff at hand to do that with every new download! ;-) If you want to make all the hash_* functions cyrus_hash_*, and send me a diff, I'll take it and get rid of the problems for good. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: IMAPD-2.2 imapd: Too many open files
Hi All,
I compiled and installed the 2.2 branch on my Alpha Server (SuSE Linux
7.1). When I start cyradm, imtest, or telnet to the imap port, the log gets
these entries:
Feb 23 18:08:36 laudare master[18187]: about to exec /usr/cyrus/bin/imapd
Feb 23 18:08:36 laudare master[18187]: couldn't exec /usr/cyrus/bin/imapd:
Too many open files
Feb 23 18:08:36 laudare master[18178]: unrecognized message for service
'imap':15d0
I was able to catch this with strace (I never know how much of these to
include, so here is all of it):
close(6)= 0
setuid(96)= 0
dup2(9, 3) = 3
dup2(7, 4) = 4
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, 0)= 0
fcntl(4, F_GETFD)= 0
fcntl(3, F_SETFD, 0) = 0
close(7) = 0
close(8) = 0
close(9) = 0
close(10) = 0
close(11) = 0
close(12) = 0
close(13) = 0
close(14) = 0
close(15) = 0
setrlimit(RLIMIT_NOFILE, {rlim_cur=0, rlim_max=0}) = 0
gettimeofday({1046012551, 826501}, NULL) = 0
getxpid() = 10728
rt_sigaction(SIGPIPE, {0x243b430, [], 0}, {SIG_DFL}, 8, 0) = 0
send(5, 183Feb 23 16:02:31 master[1072..., 70, 0) = 70
rt_sigaction(SIGPIPE, {SIG_DFL}, NULL, 8, 0) = 0
send(5, 183Feb 23 16:02:31 master[1072..., 70, 0) = 70
rt_sigaction(SIGPIPE, {SIG_DFL}, NULL, 8, 0) = 0
execve(/usr/cyrus/bin/imapd, [imapd], [/* 51 vars */]) = -1 EMFILE (Too
many open files)
gettimeofday({1046012551, 832044}, NULL) = 0
getxpid() = 10728
rt_sigaction(SIGPIPE, {0x243b430, [], 0}, {SIG_DFL}, 8, 0) = 0
send(5, 179Feb 23 16:02:31 master[1072..., 91, 0) = 91
rt_sigaction(SIGPIPE, {SIG_DFL}, NULL, 8, 0) = 0
write(3, \207\342X, 4) = 4
exit(71)= ?
The setrlimit(RLIMIT_NOFILE, {rlim_cur=0, rlim_max=0}) = 0 seemed a
little low to me, so I went to line 622 of master.c and changed s-maxfds
to a hard 256 and the error went bye-bye. The server seems to be
functioning normally now; although it is a test machine and not at all
heavily loaded, except for some complaints about a DBERROR listing log
files (they simply don't exist and cyrus seems not to want to create them
-- this is a new install).
Re: IMAPD-2.2 imapd: Too many open files
On Sun, 23 Feb 2003, Mike O'Rourke wrote:
The setrlimit(RLIMIT_NOFILE, {rlim_cur=0, rlim_max=0}) = 0 seemed a
little low to me, so I went to line 622 of master.c and changed s-maxfds
to a hard 256 and the error went bye-bye. The server seems to be
functioning normally now; although it is a test machine and not at all
heavily loaded, except for some complaints about a DBERROR listing log
files (they simply don't exist and cyrus seems not to want to create them
-- this is a new install).
Could you try again? It looks like a merge-forward got confused due to
context or something.
-Rob
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456
Research Systems Programmer * /usr/contributed Gatekeeper
