Graceful restart also for imapd.conf
Hi there, Since version 2.0.9 Cyrus master can re-read /etc/cyrus.conf on SIGHUP. The man say : Master rereads its configuration file when it receives a hangup signal, SIGHUP. Services and events may be added, deleted or modified when the configuration file is reread. Any active services removed from the configuration file will be allowed to run until completion. Looking at the code I wonder why we don't do the same to re-reread /etc/imapd.conf. Actually, master only send a SIGHUP to children when the service have been removed. My guess is that we could send a SIGHUP to all services. That way we don't have to wait for all preforked processes to finish before taking into account the new imapd.conf I have made some test and till now all seem working well. The only tricky thing I see is that we don't want to SIGHUP children in the SERVICE_STATE_DEAD state but the actual code already take care of that. So is there any problem that justify to limit the SIGHUP propagation to children for removed service only ? Or could we go a bit further (with very few modification to the code) and allow graceful restart when we modify /etc/imapd.conf. Olivier ROLAND AtoS
Re: Todo lists, 2.4.11 and 2.5 preview
For 2.5 preview: All the 2.4.11 stuff, plus: * per-message annotation support * fix LIST-EXT support to be standards complient * fix SORT=DISPLAYNAME cache format for sure * fix SPECIAL-USE to match the RFC (it changed from the draft I implemented, doh) * finalize mailboxes.db format changes (including fixing the the but that's currently breaking murder on the 'master' branch) * autocreate/autosieve Let me know if it looks like I've missed anything! I just hashed this out with Liinu on the IRC channel, and have sent it here to remind myself :) As announced in this thread : http://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg01729.html, I made good progress in porting our code to 2.5. These events type are today supported : - MessageExpunge - MessageNew - MessageRead - MessageTrash - FlagsSet - FlagsClear - MailboxCreate - MailboxDelete - MailboxRename - vnd.cmu.MessageCopy (IMAP COPY is not defined in the RFC 5423) The code is more RFC compliant as discussed in the thread : - mailboxID event parameter is now an IMAP URI, which contains the UID if referring to a single message. - private event parameters and event types are prefixed by vnd. (today vnd.cmu. but could be discussed) - missing mandatory event parameters are added (like UIDVALIDITY) - FlagsSet and FlagsClear are now fully supported (i.e +flags, -flags and flags which may send several event notifications) The code source is fully redesigned as expected. I have simplified a lot the setting. I want to say that the code of 2.5 is very nice, I appreciate the refactoring around struct index_record. Great work Bron ! I still have some finishing work : JSON default format, some testing on IMAP7 folder name, provide a new connector to notifyd (stomp?, AMQP?) and may be support more event types. I will push soon the code in our github clone of cyrus master. Sébastien PS: I thought to write regression tests with Cassandane. Is this the future Cyrus framework for functionnal testing ?
Re: Cyrus Imap and Automake
Ondřej Surý wrote: On Wed, Aug 3, 2011 at 17:59, Jeroen van Meeuwen (Kolab Systems) vanmeeu...@kolabsys.com wrote: Thomas Jarosch wrote: Hi Дилян, here's some feedback about your build system question. Note: I'm not one of the cyrus core developers. if I rewrite the build system of Cyrus imap 2.4(.10) to use Automake to generate the Makefile.in-files, will the patch be accepted in reasonable time in git/master? Have you considered alternatives to GNU Autotools? We have experience with GNU Autotools in our company projects as well as open source projects for several years now. We have found that it has several shortcomings: 1. Autotools version conflicts You can compile a released source package without any Autotools on your system. But as soon as you a) want to develop b) want to install a patch which modifies the build system (like a new path to a library, something that adds a new file,...). This is often happens as part of packaging for .rpm or .deb. you need Autotools on your machine. If the Autotools version on your machine and the one used to build the release are not compatible you can't build. Installing a different Autotools version on a given distribution without breaking something or fixing a huge list of dependency problems is nearly impossible. I have experience with this... I have quite the experience with .rpm and .deb building myself as well, and while I agree autotools *can* be problematic at times, I recon the Linux distributions are not the biggest of problems - the culprit, I think, is with the number of custom / site-specific builds out there, ranging from Sun Solaris to FreeBSD and who knows what versions of autotools are on these systems. With my fancy debian maintainer hat on - I agree, we learnt how to cope with different versions of autotools, that's the minor thing. I personally I would love to have cyrus projects automakized. It's much easy to mangle :). Between the two of us, Debian and Fedora maintainers, are we both saying yes please, no objections? Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Cyrus Imap and Automake
On Wed, Jul 27, 2011 at 03:58:53PM +0200, ?? wrote: if I rewrite the build system of Cyrus imap 2.4(.10) to use Automake to generate the Makefile.in-files, will the patch be accepted in reasonable time in git/master? Sadly this thread seems to have degenerated again - I think the above question is extremely valid. It takes effort to write such patches, and it is a shame when they sit in bugzilla etc. Hoping for a yes please! from a commiter... Cheers, Patrick
imapd crashes with SIGSEGV in mboxlist.c:221
Dear Cyrus developers,
I have reported earlier the problem to Debain bugtracker ([1]), but the
problem seems to have left without attention.
In my setup imapd causes SEGFAULT with this symptoms:
13:10:57 cyrus/imap[17999]: login: tanja.home [192.168.1.11] dmitry plain+TLS
User logged in
13:10:57 kernel: [60608.311267] imapd[17999]: segfault at 0 ip b723049a sp
bfe0fefc error 6 in libc-2.13.so[b71bc000+153000]
13:10:57 cyrus/master[2365]: process 17999 exited, signaled to death by 11
13:10:57 cyrus/master[2365]: service imap pid 17999 in BUSY state: terminated
abnormally
and then due to the crash the recovery procedure takes place:
13:10:57 cyrus/master[18000]: about to exec /usr/lib/cyrus/bin/imapd
13:10:57 cyrus/imap[18000]: DBERROR db5: /var/lib/cyrus/db/__db.001: No such
file or directory
13:10:57 cyrus/imap[18000]: DBERROR: dbenv-open '/var/lib/cyrus/db' failed:
No such file or directory
13:10:58 cyrus/imap[18000]: DBERROR: init() on berkeley
13:10:58 cyrus/imap[18000]: DBERROR: reading /var/lib/cyrus/db/skipstamp,
assuming the worst: No such file or directory
13:10:58 cyrus/imap[18000]: executed
13:10:58 cyrus/imap[18000]: skiplist: recovered /var/lib/cyrus/mailboxes.db
(66 records, 5412 bytes) in 1 second
13:10:58 cyrus/imap[18000]: skiplist: recovered /var/lib/cyrus/annotations.db
(0 records, 144 bytes) in 0 seconds
13:10:58 cyrus/imap[18000]: accepted connection
Here goes the stack trace and code pointer:
(gdb) bt
#0 0xb711549a in ?? () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#1 0x08076278 in mboxlist_mylookup (name=value optimized out, typep=value
optimized out, pathp=0x0,
partp=0xbf88e398, aclp=0xbf88e39c, tid=0x0, wrlock=0) at mboxlist.c:221
#2 0x08051272 in mlookup (tag=0x89afcb8 5, ext_name=0x8ab3130
INBOX.Sent, name=0xbf88e6e5 user.dmitry.Sent,
flags=0x0, pathp=0x0, partp=0x0, aclp=0x0, tid=0x0) at imapd.c:412
#3 0x08053022 in cmd_select (tag=0x89afcb8 5, cmd=0x89afd28 Select,
name=0x8ab3130 INBOX.Sent)
at imapd.c:2619
#4 0x08060c96 in cmdloop () at imapd.c:1462
#5 0x080620c2 in service_main (argc=1, argv=0x89a7008, envp=0xbf891004) at
imapd.c:691
#6 0x0804dc3b in main (argc=3, argv=0xbf890ff4, envp=0xbf891004) at
service.c:537
(gdb) f 1
#1 0x08076278 in mboxlist_mylookup (name=value optimized out, typep=value
optimized out, pathp=0x0,
partp=0xbf88e398, aclp=0xbf88e39c, tid=0x0, wrlock=0) at mboxlist.c:221
221 memcpy(aclresult, p, acllen);
(gdb) p aclresult
$1 = 0x0
(gdb) p p
$2 = 0xb4bc516a default\tdmitry\tlrswipcda\t
(gdb) p acllen
$3 = 0
(gdb) s
Cannot find bounds of current function
(gdb) l
206 r = mboxlist_getpath(part, name, pathp);
207 if(r) return r;
208 } else {
209 r = mboxlist_getpath(partition, name, pathp);
210 if(r) return r;
211 }
212 }
213
214 /* the rest is ACL; return it if requested */
215 if (aclp) {
216 acllen = datalen - (p - data);
217 if (acllen = aclresultalloced) {
218 aclresultalloced = acllen + 100;
219 aclresult = xrealloc(aclresult, aclresultalloced);
220 }
221 (!) memcpy(aclresult, p, acllen);
222 aclresult[acllen] = '\0';
223
224 *aclp = aclresult;
225 }
226 break;
227
228 case CYRUSDB_AGAIN:
229 return IMAP_AGAIN;
230 break;
231
232 case CYRUSDB_NOTFOUND:
233 return IMAP_MAILBOX_NONEXISTENT;
234 break;
235
The solution is mentioned in [1] as well as attached to this letter.
I wonder is it some corrupted data in my DB that causes the crash? At
the moment I need to patch Cyrus each time the version is updated in
Debian repository which is really annoying.
Also the latest Debian releases (2.2.13p1-15) do not SIGSEGV with patch
applied, but fail with the following message:
cyrus/lmtpunix[30775]: verify_user(user.dmitry) failed: Unknown/invalid
partition
I hope that somebody can help on this maillist.
[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604468
--
With best regards,
Dmitry
Index: cyrus-imapd-2.2-2.2.13p1/imap/mboxlist.c
===
--- cyrus-imapd-2.2-2.2.13p1.orig/imap/mboxlist.c 2011-08-08
02:34:25.330006463 +0200
+++ cyrus-imapd-2.2-2.2.13p1/imap/mboxlist.c2011-08-08 02:34:43.282002740
+0200
@@ -183,7 +183,7 @@
if (*p == ' ') p++;
q = partition;
- while (*p != ' ') { /* copy out partition name */
+ while (*p != ' ' *p != '\t') { /* copy out partition name */
*q++ = *p++;
}
*q = '\0';
Input on patch for ptclient/ldap requested
Hi there, I wanted to ask who is actively using ptclient/ldap, as I have some inhouse patch pending on the canonification using some sort of result_attribute, if you will. We currently have under consideration whether everything, life and the universe should be configurable before the patch is accepted upstream, which is to say (pardon my postfix lingo); - result_attribute_format, - leaf_result_attribute, but also; - group_filter_scope, - group_result_attribute Which is to say, we have a deployment extensively using 'nsroledn' -which functionally behaves like a 'memberOf', and the question then becomes if you want to use the 'cn' attribute for groups -which most often is not enforced to be a unique attribute value for groups, but is automatically unique is the search scope for groups is 'one' and the 'cn' attribute builds the 'rdn'. Long story short, I would like to know of other people who use ptclient/ldap, or have attempted to do so but failed, and the various use-case / deployment scenarios. Thanks in advance, Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Graceful restart also for imapd.conf
G'day Olivier, On 08/08/11 19:39, Olivier ROLAND wrote: Hi there, Since version 2.0.9 Cyrus master can re-read /etc/cyrus.conf on SIGHUP. The man say : Master rereads its configuration file when it receives a hangup signal, SIGHUP. Services and events may be added, deleted or modified when the configuration file is reread. Any active services removed from the configuration file will be allowed to run until completion. Looking at the code I wonder why we don't do the same to re-reread /etc/imapd.conf. Actually, master only send a SIGHUP to children when the service have been removed. My guess is that we could send a SIGHUP to all services. That way we don't have to wait for all preforked processes to finish before taking into account the new imapd.conf I have made some test and till now all seem working well. I haven't seen your code so I don't know how you're dealing with re-reading imapd.conf. But I note that in the code I see there's a bunch of global variables in the imapd process which are set as a side effect of reading imapd.conf. Not handling those properly can have all sorts of unpleasant effects, including memory leaks and pointers to freed memory. You might want to look at the recent commit in the master branch that adds a function config_reset() that handles all that; it was added for unit testing but could be useful when re-reading imapd.conf. http://git.cyrusimap.org/cyrus-imapd/commit/?id=03b4fcc4d956ccbfbbd5220b6d1ca9107707821f The only tricky thing I see is that we don't want to SIGHUP children in the SERVICE_STATE_DEAD state but the actual code already take care of that. So is there any problem that justify to limit the SIGHUP propagation to children for removed service only ? Or could we go a bit further (with very few modification to the code) and allow graceful restart when we modify /etc/imapd.conf. There are some imapd.conf entries which it would probably be a really bad idea to modify partway through a client session, like 'virtdomains'. There are others which are only consulted during imapd process initialisation and would be ignored if the file was re-read later, like 'mupdate_server'. All the code that uses config data was written under the assumption that the data would not change in the lifetime of the process, and in some cases that assumption allowed simplifying short-cuts which would not be valid otherwise. So while I think it would be great to be able to re-read imapd.conf on SIGHUP, there may be quite a lot of work to clean up the fallout. -- Greg.
Re: Todo lists, 2.4.11 and 2.5 preview
On 08/08/11 19:45, Sébastien Michel wrote: I still have some finishing work : JSON default format, some testing on IMAP7 folder name, provide a new connector to notifyd (stomp?, AMQP?) and may be support more event types. I will push soon the code in our github clone of cyrus master. Sébastien PS: I thought to write regression tests with Cassandane. I would be greatly pleased if you did that :) Is this the future Cyrus framework for functionnal testing ? That's my hope and my intention. Plus, we have nothing else that does Cyrus-specific testing. Please let me know if you have any problems or feedback about Cassandane. -- Greg.
