I'm wondering if because the COPY might be taking a long time before
responding, that the client thinks that the server has hung or gone
away. The attached (untested) patch might solve the problem.
Paul Dekkers wrote:
Hi,
From time to time (but mostly at the start of the year ;-)), I notice a
lot of load caused by people archiving their mail-folders. Maybe this is
mostly caused by Thunderbird going mad, but I was wondering if I could
do anything on the server-side to prevent things from going bad. Because
now I see memory (and swap) exhaustion and the side-effects of that
(Linux kernel killing processes)...
One example: someone was moving tens of thousands of messages from 2009
to a new "2009 folder". Apparently Thunderbird was stuck, maybe because
these things don't happen "instantly" moving this number of messages so
the server doesn't finish quickly: but Thunderbird created a lot (~100)
of sessions / imapd-processes for this user, maybe after timeouts.
(I think) Only one process was active doing the link's, it looked like
the others were mostly waiting for a write lock (fortunately), waiting
to do the same thing. (Inspected with strace.) But when the process that
hogged the CPU was killed, the next process took over, until all similar
processes were killed. And the new archive-folder now ended up with
several duplicates, taking about millions instead of tens of thousands.
(We'll have to see how to dedup that, any ideas are appreciated
otherwise I'll write something for that.)
It just happened, but it happened before. This mail-server is not that
busy, <100 users, but it happens at least a few times per year.
Any idea how to prevent things like this? Judging from the man-pages I
don't think I could do this from within cyrus, but that I would have to
prevent from linux's ulimit or so and tune that (sounds like a tough
job)... or could I actually do this with cyrus parameters?
Curious if people have similar experiences :-)
Regards,
Paul
P.S. This specific machine is running Red Hat 4 and a version of Simon's
(s)rpm.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University
Index: append.c
===================================================================
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/append.c,v
retrieving revision 1.109.2.2
diff -u -r1.109.2.2 append.c
--- append.c 28 Dec 2009 21:51:28 -0000 1.109.2.2
+++ append.c 4 Jan 2010 15:36:28 -0000
@@ -828,7 +828,8 @@
struct appendstate *as,
int nummsg,
struct copymsg *copymsg,
- int nolink)
+ int nolink,
+ struct protstream *pout)
{
struct mailbox *append_mailbox = &as->m;
int msg;
@@ -841,6 +842,7 @@
int r, n;
int flag, userflag, emptyflag;
struct body *body = NULL;
+ time_t start, now;
assert(append_mailbox->format == MAILBOX_FORMAT_NORMAL);
@@ -854,7 +856,15 @@
xmalloc(nummsg * sizeof(struct index_record));
/* Copy/link all files and cache info */
- for (msg = 0; msg < nummsg; msg++) {
+ for (start = time(0), msg = 0; msg < nummsg; msg++) {
+ /* Send progress update to client every 30 sec */
+ if (pout && (now = time(0)) > start + 30) {
+ start = now;
+ prot_printf(pout, "* OK copied %d of %d messages\r\n",
+ msg, nummsg);
+ prot_flush(pout);
+ }
+
zero_index(message_index[msg]);
message_index[msg].uid = append_mailbox->last_uid + 1 + as->nummsg;
if (append_mailbox->options & OPT_IMAP_CONDSTORE) {
Index: append.h
===================================================================
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/append.h,v
retrieving revision 1.28.2.2
diff -u -r1.28.2.2 append.h
--- append.h 28 Dec 2009 21:51:28 -0000 1.28.2.2
+++ append.h 4 Jan 2010 15:36:28 -0000
@@ -139,7 +139,8 @@
extern int append_copy(struct mailbox *mailbox,
struct appendstate *append_mailbox,
- int nummsg, struct copymsg *copymsg, int nolink);
+ int nummsg, struct copymsg *copymsg, int nolink,
+ struct protstream *pout);
extern int append_collectnews(struct appendstate *mailbox,
const char *group, unsigned long feeduid);
Index: index.c
===================================================================
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/index.c,v
retrieving revision 1.219.2.10
diff -u -r1.219.2.10 index.c
--- index.c 28 Dec 2009 21:51:33 -0000 1.219.2.10
+++ index.c 4 Jan 2010 15:36:29 -0000
@@ -1814,7 +1814,7 @@
docopyuid = (append_mailbox.m.myrights & ACL_READ);
r = append_copy(mailbox, &append_mailbox, copyargs.nummsg,
- copyargs.copymsg, nolink);
+ copyargs.copymsg, nolink, imapd_out);
if (!r) {
int sharedseen = (append_mailbox.m.options & OPT_IMAP_SHAREDSEEN);
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
