Re: Remove "Duplicate" emails (and documentation update)

[@lbutlr]

On 2018-02-23 (18:01 MST), @lbutlr  wrote:
> 
>>> First, even after expunging a message and running doveadm index -u
>>> kremels ?Archive?, subsequent runs still show the same duplicate
>>> messages.
>> 
>> I suspect client side caching.
> 
> No, there is no client side involved. I am executing all of these these 
> commands on the mail server. I expunge the messages, I index (or even 
> force-resync) and the next search shows the same messages even though they 
> are not in the Maildir anymore.

OK, I did another run yesterday and expunged 732 message dupes. I checked for a 
few of them before the expunge and there were files there. I checked after the 
expunge (several minutes after) and the files were NOT there.

They still show up if I do another search.

just now: 
 # doveadm -f table fetch -u kremels 'hdr.message-id guid uid hdr.x-listname 
mailbox' mailbox 'Archive' | sort | awk 'cnt[$1]++{if (cnt[$1]>=2) print 
prev[$1]; print} {prev[$1]=$0}'  | sort -u -k1,1| awk '{print "doveadm expunge 
-u kremels MAILBOX-GUID "$2" UID "$3}' | grep 1514660575.66506_1
hdr.message-id guiduid hdr.x-listname mailbox
doveadm expunge -u kremels MAILBOX-GUID 1514660575.66506_1.mail.covisp.net UID 
60608
 # find ~kremels/Maildir -name "1514660575.66506_1*"
 # doveadm fetch  -u kremels MAILBOX-GUID 1514660575.66506_1.mail.covisp.net 
UID 60608
Fatal: Invalid messageset

So, how do I get expunged messages to not show up in the fetch?

(Yes, I ran doveadm index -u kremels "*") 

# doveadm -Dv index -u kremels "*"
Debug: Loading modules from directory: /usr/local/lib/dovecot
Debug: Module loaded: /usr/local/lib/dovecot/lib20_virtual_plugin.so
Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: Undefined symbol 
"acl_user_module" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: Undefined symbol 
"expire_set_lookup" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: Undefined symbol 
"quota_user_module" (this is usually intentional, so just ignore this message)
Debug: Module loaded: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: Undefined symbol 
"fts_filter_filter" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: Undefined 
symbol "mail_crypt_user_get_public_key" (this is usually intentional, so just 
ignore this message)
doveadm(kremels): Debug: Effective uid=1004, gid=1004, home=/home/kremels
doveadm(kremels): Debug: Namespace inbox: type=private, prefix=, sep=, 
inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir
doveadm(kremels): Debug: maildir++: root=/home/kremels/Maildir, index=, 
indexpvt=, control=, inbox=/home/kremels/Maildir, alt=
doveadm(kremels): Debug: Archive: Mailbox opened because: index
doveadm(kremels): Info: Archive: Cache is already up to date
doveadm(kremels): Debug: Drafts: Mailbox opened because: index
doveadm(kremels): Info: Drafts: Cache is already up to date
doveadm(kremels): Debug: Junk: Mailbox opened because: index
doveadm(kremels): Info: Junk: Cache is already up to date
doveadm(kremels): Debug: Misc.not-to-me: Mailbox opened because: index
doveadm(kremels): Info: Misc.not-to-me: Cache is already up to date
doveadm(kremels): Debug: bind: Mailbox opened because: index
doveadm(kremels): Info: bind: Cache is already up to date
doveadm(kremels): Debug: Sent: Mailbox opened because: index
doveadm(kremels): Info: Sent: Cache is already up to date
doveadm(kremels): Debug: Trash: Mailbox opened because: index
doveadm(kremels): Info: Trash: Cache is already up to date
doveadm(kremels): Debug: dovecot: Mailbox opened because: index
doveadm(kremels): Info: dovecot: Cache is already up to date
doveadm(kremels): Debug: freebsd: Mailbox opened because: index
doveadm(kremels): Info: freebsd: Cache is already up to date
doveadm(kremels): Debug: httpd: Mailbox opened because: index
doveadm(kremels): Info: httpd: Cache is already up to date
doveadm(kremels): Debug: spamassassin: Mailbox opened because: index
doveadm(kremels): Info: spamassassin: Cache is already up to date
doveadm(kremels): Debug: bbedit: Mailbox opened because: index
doveadm(kremels): Info: bbedit: Cache is already up to date
doveadm(kremels): Debug: postfix: Mailbox opened because: index
doveadm(kremels): Info: postfix: Cache is already up to date
doveadm(kremels): 

Re: Remove "Duplicate" emails (and documentation update)

[@lbutlr]

On 2018-02-23 (16:47 MST), Joseph Tam  wrote:
> 
> On Fri, 23 Feb 2018, @lbutlr wrote:
> 
>> $ doveadm -f table fetch -u kremels 'hdr.message-id guid uid
>> hdr.x-listname' mailbox "Archive" | sort| awk 'cnt[$1]++{if
>> (cnt[$1]==2) print prev[$1]; print} {prev[$1]=$0}' |grep -E "[0-9] +$"
>> |awk '{print "doveadm expunge -u kremels MAILBOX-GUID "$2" UID "$3}?
> 
> I was unaware of the syntax "hdr.{header}" -- all the reference materials
> I've seen only refers to "hdr" which returns the entire header block.

the error message from doveadm if you specify an invalid field is:

Available fetch fields: hdr. body. binary. user mailbox 
mailbox-guid seq uid guid flags modseq hdr body body.snippet text text.utf8 
size.physical size.virtual date.received date.sent date.saved 
date.received.unixtime date.sent.unixtime date.saved.unixtime imap.envelope 
imap.body imap.bodystructure pop3.uidl pop3.order refcount storageid

>> First, even after expunging a message and running doveadm index -u
>> kremels ?Archive?, subsequent runs still show the same duplicate
>> messages.
> 
> I suspect client side caching.

No, there is no client side involved. I am executing all of these these 
commands on the mail server. I expunge the messages, I index (or even 
force-resync) and the next search shows the same messages even though they are 
not in the Maildir anymore.

> If {count} is what you expected, then dovecot has the correct information
> and it's likely some client-side caching issue.

I would have needed to check the count before doing this, and I did not.

>> Second, what I really want to do is run this over ALL the mailboxes,
>> except for Junk and Sent but if that is possible I can?t find the right
>> syntax.
> 
> You mean to remove duplicates from any 2 mailboxes, or remove duplicates
> in mailboxes also found in Archive?

I want to find any duplicates (based on msg ID) across all mailboxes, except 
Sent

>   doveadm -f table fetch -u kremels \
>   'hdr.message-id guid uid' \
>   NOT mailbox Archive \
>   NOT mailbox Junk \
>   NOT mailbox Sent \
>   | sort -b >list1

Aha! Didn't know you could use NOT mailbox. That probably solves my issue on 
that score.

-- 
"It's unacceptable to think" - George W Bush 15/Sep/2006