Re: [Dovecot] Duplicate E-mail Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 13 Apr 2007, Stewart Dean wrote: When this happens with sendmail, it can be that the queue handler doesn't finish before a new queue handler is kicked off, so each successive queue handler/runner ends up sending the same mail. As Matthias already pointed out, sendmail, too, locks a mail _before_ the queue runner retries a message, so there is yet another possibility (beside the one mentioned by Matthias), if the locking fails, two runners might retry the same message. Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRidJ/y9SORjhbDpvAQJWLQgAkZnHPbZFVFgVGZWyZUiLaNUlHeyVzUc7 aybqxePZS2i1OHO7e3ujzWFSV2pFmT9y7ZmIrp4FzWwBGjSkt1xMlJsOuLNTubmP Ly8ptvVV6jfJUA2eYoe9OpF5O2mGYzjXeHUjD940/2MWn9lAIE1NjatMJBb237C5 yzfUkDFpFHGYL/OEw/ky0H5GUVZlbGU+Q53j9eVqJ+p5vnPt6iAsQkYiiIHvg/mZ PtwcB3EZ3/mwq4/OU+ky/VZx3NDg4aGZpU43ic8ZsHtPgU/WB6tlPE4HEuApRTZ2 DsL7x3BBTK9a5EBk/MF5l/+EyNrmVgqhpJZy45D9AzjchYpbvK9pdA== =Hk2O -END PGP SIGNATURE-
Re: [Dovecot] Duplicate E-mail Problem
On 17.4.2007, at 0.11, Phillip T. George wrote: Oh..hmm. I may have left an important part of the log file out...it was previously grepped, but here it is, with the delete line in the middle: Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP (70.177.64.93): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP (70.177.64.93): deleted: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP (70.168.68.56): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] deleted means it was marked with \Deleted flag. The message still exists in the mailbox so it can be copied. Only after it's expunged, it's really gone. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Duplicate E-mail Problem
On Mon, 2007-04-16 at 13:21 -0500, Phillip T. George wrote: Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.177.64.93): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.168.68.56): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] It appears that the problem is that the user has two email programs running the same filters at the same time. Somehow, this creates a duplicate message. Any clues as to why this might be? Are there any options that may cause this behavior to change? I thought maybe the options maildir_copy_with_hardlinks and/or maildir_copy_preserve_filename may be helpful options...(they're both set to the default of 'no'). The message gets duplicated to Spam of course, but do you mean it also gets duplicated to INBOX? If you meant that they're duplicated in Spam, it just can't be helped. It's perfectly valid for an IMAP client to copy the same message twice to another mailbox. Set up a server-side filtering instead. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Duplicate E-mail Problem
Timo Sirainen wrote: On Mon, 2007-04-16 at 13:21 -0500, Phillip T. George wrote: Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.177.64.93): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.168.68.56): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] It appears that the problem is that the user has two email programs running the same filters at the same time. Somehow, this creates a duplicate message. Any clues as to why this might be? Are there any options that may cause this behavior to change? I thought maybe the options maildir_copy_with_hardlinks and/or maildir_copy_preserve_filename may be helpful options...(they're both set to the default of 'no'). The message gets duplicated to Spam of course, but do you mean it also gets duplicated to INBOX? If you meant that they're duplicated in Spam, it just can't be helped. It's perfectly valid for an IMAP client to copy the same message twice to another mailbox. Set up a server-side filtering instead. Its copied twice to Spam...its not in the INBOX any longer. Well...the duplicates were not a problem before when using courier-imap (though on a different server). I'm not too terribly concerned with the issue any longer, as the issue could very much considered client side. Thanks for the help! If you do have any other suggestions (or if anyone else does) on fixing the issue, I'd still like to know. -Phillip
Re: [Dovecot] Duplicate E-mail Problem
Phillip T. George wrote: Timo Sirainen wrote: On Mon, 2007-04-16 at 13:21 -0500, Phillip T. George wrote: Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.177.64.93): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.168.68.56): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] It appears that the problem is that the user has two email programs running the same filters at the same time. Somehow, this creates a duplicate message. Any clues as to why this might be? Are there any options that may cause this behavior to change? I thought maybe the options maildir_copy_with_hardlinks and/or maildir_copy_preserve_filename may be helpful options...(they're both set to the default of 'no'). The message gets duplicated to Spam of course, but do you mean it also gets duplicated to INBOX? If you meant that they're duplicated in Spam, it just can't be helped. It's perfectly valid for an IMAP client to copy the same message twice to another mailbox. Set up a server-side filtering instead. Its copied twice to Spam...its not in the INBOX any longer. Well...the duplicates were not a problem before when using courier-imap (though on a different server). I'm not too terribly concerned with the issue any longer, as the issue could very much considered client side. Thanks for the help! If you do have any other suggestions (or if anyone else does) on fixing the issue, I'd still like to know. -Phillip Oh..hmm. I may have left an important part of the log file out...it was previously grepped, but here it is, with the delete line in the middle: Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.177.64.93): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.177.64.93): deleted: uid=718, msgid=[EMAIL PROTECTED] Apr 16 12:51:45 east dovecot: IMAP(user.domain.com), IP(70.168.68.56): copy - Spam: uid=718, msgid=[EMAIL PROTECTED] The other important part is that there is not a delete after that, but the mail is not still in the inbox. So instead of copy, delete, copy, its actually of copy, copy, delete -- which is inconsistent with the log file. -Phillip
[Dovecot] Duplicate E-mail Problem
Hello all, Definitely congrats on the 1.0.0 release! Anyway, I'm having a problem, that may or may not be related to Dovecot. Specifically, duplicate (and sometimes triplicate) emails are appearing. These emails have exactly the same queue ID, so its not that they were sent multiple times. These dup emails do not show up to me ever. Postfix is being used as the SMTP, and it hands it over to procmail. I set up procmail's verbose logging feature for that user and on the duplicate emails that he receives, it shows only ONE write to a file for that specific subject of email. Tonight / this morning I'll be tracking down to see if this duplicate for sure happens in IMAP. Note: Its possible that part of the problem is he leaves two IMAP clients open at the same time (Netscape I think). I'm going to verify that with him. Any ideas? I'm NOT trying to blame Dovecot, I'm just trying to explore options. Right now its looking fairly bad for Dovecot thoughI mean, what else is there after the mail is delivered to the appropriate place in the mailbox? Software versions: postfix 2.3.8 procmail 3.22 dovecot 1.0.rc28 Thanks, Phillip
Re: [Dovecot] Duplicate E-mail Problem
When this happens with sendmail, it can be that the queue handler doesn't finish before a new queue handler is kicked off, so each successive queue handler/runner ends up sending the same mail. Background: sendmail, and perhaps postfix, is configured to a) toss stuff it can't handle right away into the queue and then, every so often (interval to be set by admin), it burps the queue. If the interval is too short, the previous queue handler/runner doesn't complete before a new one is invoked and gets it list of things to mail...with the same email on both queue handler/runners incarnations to do list... Phillip T. George wrote: Hello all, Definitely congrats on the 1.0.0 release! Anyway, I'm having a problem, that may or may not be related to Dovecot. Specifically, duplicate (and sometimes triplicate) emails are appearing. These emails have exactly the same queue ID, so its not that they were sent multiple times. These dup emails do not show up to me ever. Postfix is being used as the SMTP, and it hands it over to procmail. I set up procmail's verbose logging feature for that user and on the duplicate emails that he receives, it shows only ONE write to a file for that specific subject of email. Tonight / this morning I'll be tracking down to see if this duplicate for sure happens in IMAP. Note: Its possible that part of the problem is he leaves two IMAP clients open at the same time (Netscape I think). I'm going to verify that with him. Any ideas? I'm NOT trying to blame Dovecot, I'm just trying to explore options. Right now its looking fairly bad for Dovecot thoughI mean, what else is there after the mail is delivered to the appropriate place in the mailbox? Software versions: postfix 2.3.8 procmail 3.22 dovecot 1.0.rc28 Thanks, Phillip -- Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 [EMAIL PROTECTED] voice: 845-758-7475, fax: 845-758-7035
Re: [Dovecot] Duplicate E-mail Problem
Hmm. Thats interesting. The thing is, its only happening for one user that I'm aware of. Also, I probably should of mentioned that I'm using Maildir ... I'm sure that changes things :) -Phillip Stewart Dean wrote: When this happens with sendmail, it can be that the queue handler doesn't finish before a new queue handler is kicked off, so each successive queue handler/runner ends up sending the same mail. Background: sendmail, and perhaps postfix, is configured to a) toss stuff it can't handle right away into the queue and then, every so often (interval to be set by admin), it burps the queue. If the interval is too short, the previous queue handler/runner doesn't complete before a new one is invoked and gets it list of things to mail...with the same email on both queue handler/runners incarnations to do list... Phillip T. George wrote: Hello all, Definitely congrats on the 1.0.0 release! Anyway, I'm having a problem, that may or may not be related to Dovecot. Specifically, duplicate (and sometimes triplicate) emails are appearing. These emails have exactly the same queue ID, so its not that they were sent multiple times. These dup emails do not show up to me ever. Postfix is being used as the SMTP, and it hands it over to procmail. I set up procmail's verbose logging feature for that user and on the duplicate emails that he receives, it shows only ONE write to a file for that specific subject of email. Tonight / this morning I'll be tracking down to see if this duplicate for sure happens in IMAP. Note: Its possible that part of the problem is he leaves two IMAP clients open at the same time (Netscape I think). I'm going to verify that with him. Any ideas? I'm NOT trying to blame Dovecot, I'm just trying to explore options. Right now its looking fairly bad for Dovecot thoughI mean, what else is there after the mail is delivered to the appropriate place in the mailbox? Software versions: postfix 2.3.8 procmail 3.22 dovecot 1.0.rc28 Thanks, Phillip
Re: [Dovecot] Duplicate E-mail Problem
Timo Sirainen wrote: On Fri, 2007-04-13 at 09:10 -0500, Phillip T. George wrote: Anyway, I'm having a problem, that may or may not be related to Dovecot. Specifically, duplicate (and sometimes triplicate) emails are appearing. These emails have exactly the same queue ID, so its not that they were sent multiple times. These dup emails do not show up to me ever. What kind of filenames do they have in maildir? Do you see anything in Dovecot's error logs? The only reason why Dovecot would internally duplicate messages is if it detects multiple filenames with same base name, which aren't hardlinks to each others. And that could happen if the maildir filenames aren't truly unique. You said you're using Procmail, are you sure you're not using MH filenames (http://wiki.dovecot.org/MailboxFormat/Maildir)? If that's not the problem, you can enable mail_log plugin and check if it logs that the client does a duplicate COPY. http://wiki.dovecot.org/Plugins/MailLog Here's a snippet from the procmail log file: procmail: Assigning LASTFOLDER=/home/domain.com/homes/user/Maildir/new/1176426436.9170_1.east.domain.com procmail: Notified comsat: [EMAIL PROTECTED]:/home/domain.com/homes/user/Maildir/new/1176426436.9170_1.east.domain.com I do not see any errors near the time the duplicate messages occurred. In /etc/postfix/main.cf : home_mailbox = Maildir/ mailbox_command = /usr/bin/procmail -a $EXTENSION I'll try enabling the mail_log plugin. Thanks, Phillip
Re: [Dovecot] Duplicate E-mail Problem
Stewart Dean schrieb: When this happens with sendmail, it can be that the queue handler doesn't finish before a new queue handler is kicked off, so each successive queue handler/runner ends up sending the same mail. Background: sendmail, and perhaps postfix, is configured to a) toss stuff it can't handle right away into the queue and then, every so often (interval to be set by admin), it burps the queue. If the interval is too short, the previous queue handler/runner doesn't complete before a new one is invoked and gets it list of things to mail...with the same email on both queue handler/runners incarnations to do list... Postfix can cause duplicate messages in certain circumstances if and only if messages are delivered and the delivery is not reported to postfix, or in adverse conditions for messages to multiple recipients. It will log this however, and Postfix's queue runner isn't stupid (and there's only one at a time). So please don't guess, randomly pointing naked fingers at dressed MTAs :-)
Re: [Dovecot] Duplicate E-mail Problem
Phillip T. George schrieb: Here's a snippet from the procmail log file: procmail: Assigning LASTFOLDER=/home/domain.com/homes/user/Maildir/new/1176426436.9170_1.east.domain.com procmail: Notified comsat: [EMAIL PROTECTED]:/home/domain.com/homes/user/Maildir/new/1176426436.9170_1.east.domain.com I do not see any errors near the time the duplicate messages occurred. In /etc/postfix/main.cf : home_mailbox = Maildir/ mailbox_command = /usr/bin/procmail -a $EXTENSION mailbox_command overrides home_mailbox - if you commented this one out, you'd get procmail out of the loop, but I'm not sure if you can actually do that, as it bypasses filtering and procmailrc of all other users where it's apparently working properly. Personally, I'd say if you don't trust procmail, try maildrop - in particular, it omits the nastiest of all promail misfeatures, namely fallthrough behavior to implicit (thus, hidden!) default mailboxes. http://www.courier-mta.org/ Configuring procmail properly is a subject in itself. I know some typical pitfalls, that the typical procmailrc samples do not address, but the procmail maintainers used to withdraw to their retreat of claiming well, you have the liberty of handling errors, but I haven't bought that reason and I'm not going to. Procmail's defaults are just utterly inadequate.
