[Dovecot] graceful failure when some folders are not available...
Hi folks. Quick question in the hopes that someone knows the answer, before I dig in the code some more. In testing a new setup with some long-term archival mbox-format mailboxes stored on an NFS mount, we've found the following: if the mount is unavailable for any reason, the user cannot log into their email at all. Dovecot says: stat() failed with mbox foo and dies. This is coming from the mbox sync checks. (It's possible the same happens with a maildir folder--I'm just specifying mbox because that's what we've tested with so far). Is there a way to reconfigure this behavior? I could maybe see a fatal abort if the inbox is unavailable, but for other folders it seems rather... presumptuous. I have to think there's already a way to handle this more gracefully in the config and I'm just not seeing it. Also, does anyone know offhand if this behavior is the same for folders that aren't in the default/inbox namespace? That would seem *really* wrong. Any thoughts? Thanks much, -Brian
Re: [Dovecot] imap process consuming 100% CPU (Dovecot 1.0.3)
On Sep 7 2007, Jackie Hunt wrote: Sorry, don't know about the mid: protocol. Not sure how to get to the reference you sent. From your email it looks like gdb has the ability to attach to a running process. We have dbx which I don't believe can do that. It would be nice to confirm our issues are the same though. What I'd really like is to find a way to recreate the problem. Start dbx, and type 'attach pid' at its prompt. -Brian Hayden OIT Internet Services, University of Minnesota
Re: [Dovecot] namespaces and creation of folders that contain folders...
On Aug 14 2007, Timo Sirainen wrote: On Mon, 2007-08-13 at 21:58 -0500, [EMAIL PROTECTED] wrote: Hi Timo and all. imap/cmd-create.c contains the following bit of code: .. So. Am I missing something, or crazy, or is this really a bug? Thanks, It's a bug. Fixed: http://hg.dovecot.org/dovecot-1.0/rev/33690bb286af Excellent! Thanks much. -Brian
Re: [Dovecot] Some of my IMAP Folders vanished in some MUAs
On Aug 3 2007, Charles Marcus wrote: Eh? Folder subscription is such a basic feature/function of IMAP that I would not blame the MUA in this case, I'd blame the user... ;) Well... it's irritating that the usual ways of dealing with SUBSCRIBE and LSUB make subscriptions break if (for example) the client's 'IMAP root' setting varies from one machine to another--even if the different settings are functionally equivalent, say, 'mail' and '~/mail' when the server's base dir is set to user's $HOME. Clients and most servers could do a better job of being intelligent about some of that stuff. I realize you don't want to be too fancy, but... yeah. It's easy to blame the user, but the problem is usually more in the organization's instruction and guidance, which can be a lot harder to handle. No reason not to try to make things easier on everyone involved. (Personally, I just tell people not to use subs. :P ) -Brian
Re: [Dovecot] Some of my IMAP Folders vanished in some MUAs
On Aug 3 2007, Charles Marcus wrote: On that we agree... and certainly, if there *is* something that dovecot can do to make things easier - by all means, tell Timo - as I said (and should be obvious) - he is more than willing to make dovecot better, especially if it can be done without too much trouble (I wasn't kidding or being smart when I asked the OP if he knew of anything that could be done better)... Yep! I know. I actually did bring this up on the list a while back (June 18) and then followed-up with a proposed patch (June 20). Haven't heard anything about it, but I wasn't worried since obviously it's not the highest priority, behind fixes for the 1.0.x versions and 1.1 development and such. :) -Brian
Re: [Dovecot] mbox bug in 1.0.0
On Jun 19 2007, [EMAIL PROTECTED] wrote: On May 13 2007, Timo Sirainen wrote: On Mon, 2007-04-23 at 13:40 -0500, [EMAIL PROTECTED] wrote: Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): mbox /var/mail/user: seq=1 uid=8808 uid_broken=0 originally needed 10 bytes, now needs 23 bytes Anything else than Dovecot accessing these mboxes (besides MDA)? Nope. Dovecot IMAP and deliver. Do you have a core file from that crash? It would show a bit more information. If not, see http://dovecot.org/bugreport.html So, I'm including a backtrace for this. Sorry if you replied earlier and I missed it! Please let me know what else you might need. We're seeing this on about three dozen inboxes (which is a very small percentage, but a large enough number to hear about it :) ). (dbx) where [1] 0xff1c16e8(0x6, 0x0, 0xff1a4d28, 0x, 0xff1e8298, 0x6), at 0xff1c16e8 [2] _getutxline(0x9d8b0, 0x1, 0xb00e0, 0xa8244, 0xff1eb298, 0x0), at 0xff140158 =[3] i_internal_panic_handler(fmt = 0x93b88 mbox %s: seq=%u uid=%u uid_broken=%d originally needed %llu bytes, now needs %u bytes, args = 0xffbfe7d0), line 403 in failures.c [4] i_panic(format = 0x93b88 mbox %s: seq=%u uid=%u uid_broken=%d originally needed %llu bytes, now needs %u bytes, ...), line 183 in failures.c [5] mbox_sync_rewrite(sync_ctx = 0xffbfefd0, mail_ctx = (nil), end_offset = 4448ULL, move_diff = 30LL, extra_space = 4294971774ULL, first_seq = 1U, last_seq = 163U), line 422 in mbox-sync-rewrite.c [6] mbox_sync_do(sync_ctx = 0xffbfefd0, flags = -4198952), line 920 in mbox-sync.c [7] mbox_sync(mbox = 0xc99e0, flags = MBOX_SYNC_UNDIRTY), line 1873 in mbox-sync.c [8] mbox_storage_sync_init(box = 0xc99e0, flags = MAILBOX_SYNC_FLAG_FULL_READ), line 1957 in mbox-sync.c [9] mailbox_sync_init(box = 0xc99e0, flags = MAILBOX_SYNC_FLAG_FULL_READ), line 406 in mail-storage.c [10] imap_sync_nonselected(box = 0xc99e0, flags = MAILBOX_SYNC_FLAG_FULL_READ), line 196 in imap-sync.c [11] _cmd_select_full(cmd = 0xc39e4, readonly = 0), line 39 in cmd-select.c [12] cmd_select(cmd = 0xc39e4), line 92 in cmd-select.c [13] client_handle_input(cmd = 0xc39e4), line 335 in client.c [14] client_handle_input(cmd = 0xc39e4), line 389 in client.c [15] _client_input(context = 0xc39a0), line 432 in client.c [16] io_loop_handler_run(ioloop = 0xb9f08), line 199 in ioloop-poll.c [17] io_loop_run(ioloop = 0xb9f08), line 326 in ioloop.c [18] main(argc = -4195453, argv = 0xb0800, envp = 0xb1be4), line 290 in main.c And in case it helps, a dump from frame [5] which is the last bit before the crash: (dbx) frame 5 Current function is mbox_sync_rewrite 422 i_panic(mbox %s: seq=%u uid=%u uid_broken=%d (dbx) dump first_nonexpunged = 255U move_diff = 30LL ret = 0 mails = 0xcd810 idx = 0 padding_per_mail = 20U first_nonexpunged_idx = 0 expunged_space = 0 sync_ctx = 0xffbfefd0 extra_space = 4294971774ULL mail_ctx = (nil) dest_offset = 0 count = 2U last_seq = 163U orig_prev_msg_uid = 1359U next_move_diff = 10ULL end_offset = 4448ULL __PRETTY_FUNCTION__ = mbox_sync_rewrite start_offset = 0 offset = 4294971774ULL first_seq = 1U next_end_offset = 52ULL If these happen frequently, I could also send a debug patch that writes the broken mbox headers to some temp file. That with the core file would provide enough information to figure out what exactly is the problem. This is still an option, if it would help. -Brian Hayden Internet Services University of MN
[Dovecot] mbox inbox default
HI Timo all. We're an mbox environment, inboxes in /var/mail/$USER (I'll include dovecot -n output at the bottom of this mail). mail_location is set accordingly for deliver. I'd assumed that if the /var/mail/$USER file doesn't exist, dovecot would create it, but we've discovered from experience (and I verified it in mbox-storage.c) that if access(path, R_OK|W_OK) fails--as it does if the file doesn't exist--that dovecot defaults then to delivering in root_dir/inbox. I'm curious why this is. It seems like if the configured INBOX file doesn't exist, dovecot should create it if possible; it's a different case than if, say, the user doesn't have write access to that directory, in which case it of course would be appropriate to fall back to some other location rather than rejecting mail. Is there something I'm missing that makes my idea wrong? Thanks, -Brian Hayden OIT Internet Services University of MN # ./dovecot -n # /etc/opt/dovecot/dovecot.conf syslog_facility: local6 protocols: imap imaps pop3 pop3s listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /etc/opt/openssl/certs/dovecot.pem ssl_key_file: /etc/opt/openssl/private/dovecot.pem shutdown_clients: no login_dir: /var/opt/dovecot/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_log_format_elements: user=%u pid=%p method=%m rip=%r lip=%l %c login_processes_count: 32 login_max_processes_count: 6000 max_mail_processes: 8192 first_valid_uid: 100 mail_location: mbox:~:INBOX=/var/mail/%u:INDEX=/var/mail/.dovecot-index/%1u/%u/ mail_debug: yes mbox_lazy_writes: no mail_executable(default): /opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(imap): /opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): acl mail_plugins(imap): acl mail_plugins(pop3): mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 mail_log_prefix: [%p] %Us(%u): mail_log_max_lines_per_sec: 0 imap_client_workarounds(default): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv namespace: type: private separator: / location: mbox:~:INBOX=/var/mail/%u:INDEX=/var/mail/.dovecot-index/%1u/%u/ inbox: yes namespace: type: private separator: / prefix: mail/ location: mbox:~/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/ hidden: yes namespace: type: private separator: / prefix: ~/mail/ location: mbox:~/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/ hidden: yes namespace: type: private separator: / prefix: ~%u/mail/ location: mbox:~/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/ hidden: yes namespace: type: private separator: / prefix: Shared/ location: maildir:~/Maildir/Shared/:INDEX=/var/mail/.dovecot-index/%1u/%u/shared:CONTROL=/var/mail/.dovecot-index/%1u/%u/shared-control auth default: username_format: %Lu passdb: driver: pam args: session=yes * userdb: driver: passwd plugin: acl: vfile:
Re: [Dovecot] mbox inbox default
On Aug 1 2007, Timo Sirainen wrote: On Wed, 2007-08-01 at 08:58 -0500, [EMAIL PROTECTED] wrote: We're an mbox environment, inboxes in /var/mail/$USER (I'll include dovecot -n output at the bottom of this mail). mail_location is set accordingly for deliver. I'd assumed that if the /var/mail/$USER file doesn't exist, dovecot would create it, but we've discovered from experience (and I verified it in mbox-storage.c) that if access(path, R_OK|W_OK) fails--as it does if the file doesn't exist--that dovecot defaults then to delivering in root_dir/inbox. That access() check is done only if INBOX location isn't explicitly specified. Is this a problem only with deliver or also with imap? Deliver only. On further review, I discovered where the problem lies locally, it was not a Dovecot issue. Sorry to bother. :) Thanks for your help, -Brian
Re: [Dovecot] LSUB/SUBSCRIBE under namespaces
On Jun 18 2007, [EMAIL PROTECTED] wrote:
Hi folks.
Here's our situation: Migrating from UW-IMAP. Have lots (as in, tens of
thousands) of clients set up using '~/mail' as the IMAP root, and using
subscriptions.
The rest of my original message is at the end for context. :) Brief:
Subscriptions in general are given to inconsistent behavior if there is any
difference in client settings from one session to the next. Dovecot writes
subscriptions without the namespace prefix and then prepends the current ns
prefix to every item in the subscription list for LSUB response.
So if the namespace prefix changes session-to-session; or if there are
legacy entries with more full paths in the subs file, LSUB returns
duplicates and paths that do not exist. This leads to very confused users
and in some cases data loss when they (reasonably) try to delete a
duplicate and find they've deleted the only copy of a folder.
In order to avoid this, I made a (very) simple two-part mod. The first bit
simply writes subscriptions with the ns prefix included:
diff cmd-subscribe.c cmd-subscribe.c.orig :
35c35 if (mail_storage_set_subscribed(storage, verify_name, subscribe)
0)
---
if (mail_storage_set_subscribed(storage, mailbox, subscribe) 0)
The second changes LSUB so that it only returns entries from the
subscription list which match the ns prefix, and avoids prepending the ns
prefix to LSUB responses:
diff cmd-list.c cmd-list.c.orig :
138,141c138 } else if (ctx-lsub strncmp(ctx-ns-prefix,
list-name, strlen(ctx-ns-prefix))) { continue; } else if
(!ctx-lsub) {
---
} else {
Now we err on the side of not showing some previously subscribed folders if
the client config changes, or if there is random junk in the subscriptions
file--rather than trying to show everything and returning duplicate and
non-existent mailbox paths in LSUB.
Hopefully Timo (or anyone else), you could offer some thoughts on whether
this is an acceptable strategy, or if I've overlooked something. Thanks
very much,
-Brian Hayden
Internet Services
University of Minnesota
Dovecot mail_location is using '~' for mbox storage. We've disabled full
filesystem access in Dovecot in order to use ACLs for shared folders.
Hence we have a hidden namespace for backwards compatibility with those
clients using '~/mail' (dovecot -n output is at the end of this message
so you can see exact settings). Here's the problem. If the client is not
taking advantage of this hidden namespace--say it's configured to use
plain old 'mail' as the root directory--LSUB and SUBSCRIBE work as I'd
expect. Subscribing to a folder called 'foo' puts this in the user's
subscriptions file:
mail/foo
And this (from Thunderbird's log) is the LSUB exchange:
48203776[ed8ebe0]: 2e03600:server.umn.edu:A:SendData: 6 lsub mail/*
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=26 needmore=0]
48203776[ed8ebe0]: 2e03600:serverl.umn.edu:A:CreateNewLineFromSocket: *
LSUB () / mail/foo
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=38 needmore=0]
(trimmed...)
48203776[ed8ebe0]:
2e03600:bhayden.email.umn.edu:A:CreateNewLineFromSocket:
6 OK Lsub completed.
All is well. However, when the client is configured with '~/mail' as the
root and hence triggers the hidden namespace for such, SUBSCRIBE only
writes a path relative to that namespace prefix. So as opposed the
'mail/foo' in the first example, we get:
foo
written to the subs folder.
This leads to three problems:
1. Folders subscribed to when using this namespace will either disappear
or break in a client using a different IMAP root setting that actually
refers to the same place. For example, if this user moves to a client
(such as both of the webmail offerings on campus) that use 'mail/' as the
prefix, these folders will not be visible.
This is a fairly sizeable population of our users, whom we aren't
necessarily in the position to say too bad to. :)
2. Users moved over from UW-IMAP, who were using '~/mail' for the IMAP
root, have a subscriptions file full of:
~/mail/bar
Dovecot, with the same client configuration (since it's expecting a bare
folder name to which it will prepend the namespace prefix), interprets
this as so:
49036288[fa7e740]: 2eabc00:server.umn.edu:A:CreateNewLineFromSocket: *
LSUB () / ~/mail/~/mail/foobar
Leading to having the folder displayed in the client's list, but giving an
error on SELECT. Now, this one we could theoretically solve by scrubbing
the prefixes out of subs files as we convert; however, that leaves us with
#1 above, where the bare folder name in the subs file depends on (to the
average user) an obscure client config setting, which in practice in a
large heterogeneous environment is unreliable.
3. Users who have been using an IMAP root of 'mail' in some client and
subscribed to folders, who then use a client with the root set to
'~/mail' see what appears to be duplicates of the same folder at the
Re: [Dovecot] mbox bug in 1.0.0
On May 13 2007, Timo Sirainen wrote: On Mon, 2007-04-23 at 13:40 -0500, [EMAIL PROTECTED] wrote: Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): mbox /var/mail/user: seq=1 uid=8808 uid_broken=0 originally needed 10 bytes, now needs 23 bytes Anything else than Dovecot accessing these mboxes (besides MDA)? Nope. Dovecot IMAP and deliver. Do you have a core file from that crash? It would show a bit more information. If not, see http://dovecot.org/bugreport.html I finally got around to generating one. It's 3MB, and was generated from 1.0.0. Would you like a copy, or do you want me to check on some state myself and send you results? If these happen frequently, I could also send a debug patch that writes the broken mbox headers to some temp file. That with the core file would provide enough information to figure out what exactly is the problem. I can do that, too, if necessary. Thanks much for your help, -Brian Hayden Internet Services University of MN
[Dovecot] dovecot-shared being ignored?
So, I'm working on a shared folder. Everything is nice except for one bit. My dovecot-shared file in the Maildir is either being ignored, or doesn't work the way I think it does. Any messages copied into the Maildir using a mail client are mode 660 and owned by the user who did the copy. Perms on the dovecot-shared file are 700 (I'm using Solaris ACLs in addition to the Dovecot ACLs in order to prevent unauthorized access via means that don't pass through Dovecot's IMAP server... ie SFTP/SCP, and so I'm using 700 with a rwx mask and access to the file set per user in the Solaris ACL). The message files being created 660 means they are accessible via SFTP/SCP to anyone in the group, and we have a single one for (nearly) all users. This is not really acceptable, so if anyone can offer guidance on the dovecot-shared that'd be great. I checked the wiki, and it has very minimal information. -Brian
[Dovecot] mbox bug in 1.0.0
Hi folks. We're moving users from rc24 and 27 to 1.0.0. On four mailboxes so far we've seen these errors after the move (quick and dirty anonymizing follows): Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.info] imap-login: Login: user=USER, pid=9046, method=PLAIN, rip=***.***.***.***, lip=***.***.***.***, TLS Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): mbox /var/mail/user: seq=1 uid=8808 uid_broken=0 originally needed 10 bytes, now needs 23 bytes Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): Raw backtrace: 0x7b800 - 0x43434 - 0x3e018 - 0x3fc34 - 0x40080 - 0x6d3cc - 0x29a88 - 0x21324 - 0x214c0 - 0x22f48 - 0x22ec0 - 0x230c0 - 0x81c18 - 0x814e4 - 0x2b5cc - 0x1dc58 Points of interest: 1. I can't reproduce this. It still happens after the user's .imap directories are removed. If I make a copy of the mailbox on another machine running the same version of Dovecot with the same config, the problem doesn't occur. Weird. 2. So far, have only seen it on quite large mailboxes (close to or over 100MB). Most large boxes have not had the problem, but the ones that have are all large. Any thoughts? Dovecot -n output below. -Brian Hayden OIT Internet Services U of Minnesota # dovecot -n # /etc/opt/dovecot/dovecot.conf syslog_facility: local6 protocols: imap imaps pop3 pop3s listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /etc/opt/openssl/certs/dovecot.pem ssl_key_file: /etc/opt/openssl/private/dovecot.pem shutdown_clients: no login_dir: /var/opt/dovecot/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_log_format_elements: user=%u pid=%p method=%m rip=%r lip=%l %c login_processes_count: 32 login_max_processes_count: 4096 max_mail_processes: 8192 first_valid_uid: 100 default_mail_env: mbox:~/:INBOX=/var/mail/%u mail_location: mbox:~/:INBOX=/var/mail/%u mail_full_filesystem_access: yes mbox_lazy_writes: no mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 mail_log_prefix: [%p] %Us(%u): imap_client_workarounds(default): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv namespace: type: private separator: / location: mbox:~/:INBOX=/var/mail/%u inbox: yes hidden: yes auth default: username_format: %Lu verbose: yes debug: yes passdb: driver: pam args: session=yes * userdb: driver: passwd
