Hi, I've some very annoying bit of a problem with the cyrus imapd server and hope that someone here has a good grasp on it's internals and can tell me what's going on, confirm it's a bug, a matter lacking documentation or something else entirely. I'm currently out of ideas (and freedom to spend more time on this).
Thanks! Problem: The IMAP4 IDLE Command does not work as expected. I.e. - The cyrus imapd server advertised the IDLE CAPABILITY - after sending IDLE it responds with "idling" - when I send an e-mail to myself I see the e-mail arrive and being filed in /var/log/mail.log - but imapd does not ever notify me about the new e-mail. (rather my client times out and recheck after >10 minutes) Version: Cyrus IMAP v2.4.16-Debian-2.4.16-4+deb7u2 (which is the currently latest stable Debian Package) I spent quite a few hours debugging this problem and learned the following: 1. I tried disabling idled and rely on just imapd's polling setting. - I disabled idle in cyrus.conf - i deleted the stale/remaining socket file idlesocket: /var/run/cyrus/socket/idle - I set imapidlepoll: 30 Still nothing happened, imapd never did notify me about my incoming test messages. 2. I re-enabled idled and set CYRUS_VERBOSE=1 - verified that idled was working, socket was reachable, permissions ok On connection to the imapd and start of any IDLE session, mail.log would contain: <logdate> mail cyrus/idled[1477]: imapd[1524]: IDLE_INIT '.' On receipt of a new e-mail, mail.log would contain: <logdate> mail cyrus/idled[1477]: IDLE_NOTIFY 'user.xyz' <logdate> mail cyrus/idled[1477]: IDLE_NOTIFY 'user.uvw' After playing around with strace, I discovered that idled was being correctly notified about mailbox updates, but DID NOT send signals to the idling imapd instances. 3. I had a look at idled.c and imapd.c from the latest stable sources. After glancing over the code, it seems obvious that idled will only notify those instances of imapd about a mailbox update that have registered themselves for that very mailbox. i.e. the logmessage IDLE_INIT '.' should have read IDLE_INIT 'user.xyz' Since it did not, idled does not feel any imapd has registered for updates to that mailbox thus we do not the signals. Why is that ? It seems that imapd purposefully DOES NOT send its mailbox name to idled, if the test "if(imapd_index)" fails. This is where I stopped browsing the code and tried to rebuild / recover the index in case it became corrupted by the recent debian distribution upgrade. I tried running squatter and recover -r user.xyz serveral times as well as restarted cyrus, but to no avail. 4. Looking to the Maillinglist for Help .. any ideas or help is appreciated. :-) config files attached.
# Debian Cyrus imapd.conf # See imapd.conf(5) for more information and more options # Configuration directory configdirectory: /var/lib/cyrus # Directories for proc and lock files proc_path: /run/cyrus/proc mboxname_lockpath: /run/cyrus/lock # Which partition to use for default mailboxes defaultpartition: default partition-default: /var/spool/cyrus/mail # News setup partition-news: /var/spool/cyrus/news newsspool: /var/spool/news # Alternate namespace # If enabled, activate the alternate namespace as documented in # /usr/share/doc/cyrus-doc-2.4/html/altnamespace.html, where an user's # subfolders are in the same level as the INBOX # See also userprefix and sharedprefix on imapd.conf(5) altnamespace: no # UNIX Hierarchy Convention # Set to yes, and cyrus will accept dots in names, and use the forward # slash "/" to delimit levels of the hierarchy. This is done by converting # internally all dots to "^", and all "/" to dots. So the "rabbit.holes" # mailbox of user "helmer.fudd" is stored in "user.elmer^fud.rabbit^holes" unixhierarchysep: no # Rejecting illegal characters in headers # Headers of RFC2882 messages must not have characters with the 8th bit # set. However, too many badly-written MUAs generate this, including most # spamware. Enable this to reject such messages. #reject8bit: yes # Munging illegal characters in headers # Headers of RFC2882 messages must not have characters with the 8th bit # set. However, too many badly-written MUAs generate this, including most # spamware. If you kept reject8bit disabled, you can choose to leave the # crappage untouched by disabling this (if you don't care that IMAP SEARCH # won't work right anymore. #munge8bit: no # Forcing recipient user to lowercase # Cyrus IMAPD is case-sensitive. If all your mail users are in lowercase, it is # probably a very good idea to set lmtp_downcase_rcpt to true. This is set by # default, per RFC2821. This was not set by default in debian versions up to # and including 2.2.12-4. lmtp_downcase_rcpt: yes # Uncomment the following and add the space-separated users who # have admin rights for all services. admins: cyrus # Space-separated list of users that have lmtp "admin" status (i.e. that # can deliver email through TCP/IP lmtp). If specified, this parameter # overrides the "admins" parameter above #lmtp_admins: postman # Space-separated list of users that have mupdate "admin" status, in # addition to those in the admins: entry above. Note that mupdate slaves and # backends in a Murder cluster need to autenticate against the mupdate master # as admin users. #mupdate_admins: mupdateman # Space-separated list of users that have imapd "admin" status, in # addition to those in the admins: entry above #imap_admins: cyrus # Space-separated list of users that have sieve "admin" status, in # addition to those in the admins: entry above #sieve_admins: cyrus # List of users and groups that are allowed to proxy for other users, # seperated by spaces. Any user listed in this will be allowed to login # for any other user. Like "admins:" above, you can have imap_proxyservers # and sieve_proxyservers. #proxyservers: cyrus # No anonymous logins allowanonymouslogin: no # Minimum time between POP mail fetches in minutes popminpoll: 0 # If nonzero, normal users may create their own IMAP accounts by creating # the mailbox INBOX. The user's quota is set to the value if it is positive, # otherwise the user has unlimited quota. autocreatequota: 0 # umask used by Cyrus programs #essential for smooth debian operation to be 077 umask: 077 # Sendmail binary location # DUE TO A BUG, Cyrus sends CRLF EOLs to this program. This breaks Exim 3. # For now, to work around the bug, set this to a wrapper that calls # /usr/sbin/sendmail -dropcr instead if you use Exim 3. #sendmail: /usr/sbin/sendmail # If enabled, cyrdeliver will look for Sieve scripts in user's home # directories: ~user/.sieve. sieveusehomedir: false # If sieveusehomedir is false, this directory is searched for Sieve scripts. sievedir: /var/spool/sieve # notifyd(8) method to use for "MAIL" notifications. If not set, "MAIL" # notifications are disabled. Valid methods are: null, log, zephyr #mailnotifier: zephyr # notifyd(8) method to use for "SIEVE" notifications. If not set, "SIEVE" # notifications are disabled. This method is only used when no method is # specified in the script. Valid methods are null, log, zephyr, mailto #sievenotifier: zephyr # If enabled, the partitions will also be hashed, in addition to the hashing # done on configuration directories. This is recommended if one partition has a # very bushy mailbox tree. hashimapspool: true # Allow plaintext logins by default (SASL PLAIN) allowplaintext: yes # Force PLAIN/LOGIN authentication only # (you need to uncomment this if you are not using an auxprop-based SASL # mechanism. saslauthd users, that means you!). And pay attention to # sasl_minimum_layer and allowapop below, too. sasl_mech_list: PLAIN LOGIN # Allow use of the POP3 APOP authentication command. # Note that this command requires that the plaintext passwords are # available in a SASL auxprop backend (eg. sasldb), and that the system # can provide enough entropy (eg. from /dev/urandom) to create a challenge # in the banner. #allowapop: no # The minimum SSF that the server will allow a client to negotiate. A # value of 1 requires integrity protection; any higher value requires some # amount of encryption. sasl_minimum_layer: 0 # The maximum SSF that the server will allow a client to negotiate. A # value of 1 requires integrity protection; any higher value requires some # amount of encryption. #sasl_maximum_layer: 256 # List of remote realms whose users may log in using cross-realm # authentications. Seperate each realm name by a space. A cross-realm # identity is considered any identity returned by SASL with an "@" in it. # NOTE: To support multiple virtual domains on the same interface/IP, # you need to list them all as loginreals. If you don't list them here, # (most of) your users probably won't be able to log in. #loginrealms: example.com loginrealms: <------redacted--------> # Enable virtual domain support. If enabled, the user's domain will # be determined by splitting a fully qualified userid at the last '@' # or '%' symbol. If the userid is unqualified, and the virtdomains # option is set to "on", then the domain will be determined by doing # a reverse lookup on the IP address of the incoming network # interface, otherwise the user is assumed to be in the default # domain (if set). #virtdomains: userid # The default domain for virtual domain support # If the domain of a user can't be taken from its login and it can't # be determined by doing a reverse lookup on the interface IP, this # domain is used. #defaultdomain: defaultdomain: <------redacted--------> # # SASL library options (these are handled directly by the SASL libraries, # refer to SASL documentation for an up-to-date list of these) # # The mechanism(s) used by the server to verify plaintext passwords. Possible # values are "saslauthd", "auxprop", "pwcheck" and "alwaystrue". They # are tried in order, you can specify more than one, separated by spaces. # # Do note that, since sasl will be run as user cyrus, you may have a lot of # trouble to set this up right. sasl_pwcheck_method: saslauthd allowusermoves: yes # What auxpropd plugins to load, if using sasl_pwcheck_method: auxprop # by default, all plugins are tried (which is probably NOT what you want). #sasl_auxprop_plugin: sasldb # If enabled, the SASL library will automatically create authentication secrets # when given a plaintext password. Refer to SASL documentation sasl_auto_transition: no <------ SSL Certs redacted --------> # The length of time (in minutes) that a TLS session will be cached for later # reuse. The maximum value is 1440 (24 hours), the default. A value of 0 will # disable session caching. #tls_session_timeout: 1440 tls_session_timeout: 0 # The list of SSL/TLS ciphers to allow, in decreasing order of precedence. # The format of the string is described in ciphers(1). The Debian default # selects TLSv1 high-security ciphers only, and removes all anonymous ciphers # from the list (because they provide no defense against man-in-the-middle # attacks). It also orders the list so that stronger ciphers come first. tls_cipher_list: TLSv1+HIGH:!aNULL:@STRENGTH # Require a client certificate for ALL services (imap, pop3, lmtp, sieve). #tls_require_cert: false # Require a client certificate for imap ONLY. #imap_tls_require_cert: false # Require a client certificate for pop3 ONLY. #pop3_tls_require_cert: false # Require a client certificate for lmtp ONLY. #lmtp_tls_require_cert: false # Require a client certificate for sieve ONLY. #sieve_tls_require_cert: false # # Cyrus Murder cluster configuration # # Set the following options to the values needed for this server to # autenticate against the mupdate master server: # mupdate_server # mupdate_port # mupdate_username # mupdate_authname # mupdate_realm # mupdate_password # mupdate_retry_delay ## ## KEEP THESE IN SYNC WITH cyrus.conf ## # Unix domain socket that lmtpd listens on. lmtpsocket: /var/run/cyrus/socket/lmtp # Unix domain socket that idled listens on. idlesocket: /var/run/cyrus/socket/idle # Unix domain socket that the new mail notification daemon listens on. notifysocket: /var/run/cyrus/socket/notify # Syslog prefix. Defaults to cyrus (so logging is done as cyrus/imap etc.) syslog_prefix: cyrus imapidlepoll: 30
# Debian defaults for Cyrus IMAP server/cluster implementation # see cyrus.conf(5) for more information # # All the tcp services are tcpd-wrapped. see hosts_access(5) START { # do not delete this entry! recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r" # this is only necessary if idlemethod is set to "idled" in imapd.conf idled cmd="idled" # this is useful on backend nodes of a Murder cluster # it causes the backend to syncronize its mailbox list with # the mupdate master upon startup #mupdatepush cmd="/usr/sbin/cyrus ctl_mboxlist -m" # this is recommended if using duplicate delivery suppression delprune cmd="/usr/sbin/cyrus expire -E 3" # this is recommended if caching TLS sessions tlsprune cmd="/usr/sbin/cyrus tls_prune" } # UNIX sockets start with a slash and are absolute paths # you can use a maxchild=# to limit the maximum number of forks of a service # you can use babysit=true and maxforkrate=# to keep tight tabs on the service # most services also accept -U (limit number of reuses) and -T (timeout) SERVICES { # --- Normal cyrus spool, or Murder backends --- # add or remove based on preferences imap cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100 imaps cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100 pop3 cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50 pop3s cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50 nntp cmd="nntpd -U 30" listen="nntp" prefork=0 maxchild=100 #nntps cmd="nntpd -s -U 30" listen="nntps" prefork=0 maxchild=100 # At least one form of LMTP is required for delivery # (you must keep the Unix socket name in sync with imap.conf) #lmtp cmd="lmtpd" listen="localhost:lmtp" prefork=0 maxchild=20 lmtpunix cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp" prefork=0 maxchild=20 # ---------------------------------------------- # useful if you need to give users remote access to sieve # by default, we limit this to localhost in Debian sieve cmd="timsieved" listen="localhost:sieve" prefork=0 maxchild=100 # this one is needed for the notification services notify cmd="notifyd" listen="/var/run/cyrus/socket/notify" proto="udp" prefork=1 # --- Murder frontends ------------------------- # enable these and disable the matching services above, # except for sieve (which deals automatically with Murder) # mupdate database service - must prefork at least 1 # (mupdate slaves) #mupdate cmd="mupdate" listen=3905 prefork=1 # (mupdate master, only one in the entire cluster) #mupdate cmd="mupdate -m" listen=3905 prefork=1 # proxies that will connect to the backends #imap cmd="proxyd" listen="imap" prefork=0 maxchild=100 #imaps cmd="proxyd -s" listen="imaps" prefork=0 maxchild=100 #pop3 cmd="pop3proxyd" listen="pop3" prefork=0 maxchild=50 #pop3s cmd="pop3proxyd -s" listen="pop3s" prefork=0 maxchild=50 #lmtp cmd="lmtpproxyd" listen="lmtp" prefork=1 maxchild=20 # ---------------------------------------------- } EVENTS { # this is required checkpoint cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30 # this is only necessary if using duplicate delivery suppression delprune cmd="/usr/sbin/cyrus expire -E 3" at=0401 # this is only necessary if caching TLS sessions tlsprune cmd="/usr/sbin/cyrus tls_prune" at=0401 # indexing of mailboxes for server side fulltext searches # reindex changed mailboxes (fulltext) approximately every other hour squatter_1 cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus squatter -s" period=120 # reindex all mailboxes (fulltext) daily squatter_a cmd="/usr/sbin/cyrus squatter" at=0517 }