Re: dovecot sieve duplicates detection
On Wed, Apr 25, 2018, at 3:20 PM, Stephan Bosch wrote: > > > Op 23/04/2018 om 22:03 schreef André Rodier: > > On 23/04/18 14:18, Stephan Bosch wrote: > >> > >> > >> Op 11-4-2018 om 23:58 schreef André Rodier: > >>> Hello, > >>> > >>> I have tested the sieve duplicate script with success so far, but I > >>> have > >>> a question. > >> > >> Sieve duplicate script? You mean the Sieve duplicate extension (RFC > >> 7352)? > >> > >>> I would like to know if the "duplicate" sieve flag in Dovecot is global > >>> to all folders, or specific to one folder only. > >> > >> It uses the lda-dupes file in the user's home directory. So, it is > >> not normally related to folders, although the identifier used for > >> duplicate matching could be composed of the mailbox name if you want. > >> > >>> For instance, if I copy an email from one folder to another, and I have > >>> a discard action on duplicate email, is this action will be applied (in > >>> this case, discard) or not. > >> > >> Are you talking about IMAPSieve now? I am not sure "duplicate" is > >> currently even allowed in that context. > >> > >>> If the duplicate is global to all folders, is there a way to restrict > >>> the search in one folder only. > >> > >> You can set the :uniqueid parameter accordingly. > >> > >> Regards, > >> > >> Stephan. > > > > Thank you, Stephan. > > > > Yes, I meant the Sieve duplicate extension. > > > > I am using a program to import email (mbsync), which use the IMAP > > append function. Sometimes, the import fail and I have to restart the > > program. Unfortunately, the same emails are imported again. > > > > I found a fix by using a dovecot IMAP sieve script executed on the > > APPEND action > > (https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve). I wrote > > a custom sieve script that "discard" the ones that are detected as > > "duplicate". It worked very well and the emails were not any more > > imported twice. > > > > However, there was a huge side effect: archiving an email with > > Thunderbird is not working any more, and even lost! I have been able > > to understand the error as this: > > > > 1. When archiving an email with Thunderbird, it is first copied > > (APPEND) into the archive folder, but the original folder is not > > expunged. > > 2. The sieve script detect the email as duplicate, and discard it. > > 3. When the original folder is expunged, the source email is lost... > > > > My conclusion was the duplicate detection function is global to all > > folders. > > > > If I could restrict the detection of duplicates in the current folder > > only, this would let me run the import program again without error. > > Specify the ID used for duplicate checking explicitly using the > :uniqueid argument (https://tools.ietf.org/html/rfc7352#section-3.1). > Using the variables extenion, compose the uniqueid from the message-id > and the mailbox name. > In my experience with dovecot's implementation, you can set the ID only once in a script. If you try to filter duplicates based on multiple IDs, only the first (or last, I don't remember) takes effect. V/r, James Cassell
Re: dovecot sieve duplicates detection
On 25/04/18 20:20, Stephan Bosch wrote: Op 23/04/2018 om 22:03 schreef André Rodier: On 23/04/18 14:18, Stephan Bosch wrote: Op 11-4-2018 om 23:58 schreef André Rodier: Hello, I have tested the sieve duplicate script with success so far, but I have a question. Sieve duplicate script? You mean the Sieve duplicate extension (RFC 7352)? I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. It uses the lda-dupes file in the user's home directory. So, it is not normally related to folders, although the identifier used for duplicate matching could be composed of the mailbox name if you want. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. Are you talking about IMAPSieve now? I am not sure "duplicate" is currently even allowed in that context. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. You can set the :uniqueid parameter accordingly. Regards, Stephan. Thank you, Stephan. Yes, I meant the Sieve duplicate extension. I am using a program to import email (mbsync), which use the IMAP append function. Sometimes, the import fail and I have to restart the program. Unfortunately, the same emails are imported again. I found a fix by using a dovecot IMAP sieve script executed on the APPEND action (https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve). I wrote a custom sieve script that "discard" the ones that are detected as "duplicate". It worked very well and the emails were not any more imported twice. However, there was a huge side effect: archiving an email with Thunderbird is not working any more, and even lost! I have been able to understand the error as this: 1. When archiving an email with Thunderbird, it is first copied (APPEND) into the archive folder, but the original folder is not expunged. 2. The sieve script detect the email as duplicate, and discard it. 3. When the original folder is expunged, the source email is lost... My conclusion was the duplicate detection function is global to all folders. If I could restrict the detection of duplicates in the current folder only, this would let me run the import program again without error. Specify the ID used for duplicate checking explicitly using the :uniqueid argument (https://tools.ietf.org/html/rfc7352#section-3.1). Using the variables extenion, compose the uniqueid from the message-id and the mailbox name. Regards, Stephan. Thank you, I will try this. André
Re: dovecot sieve duplicates detection
Op 23/04/2018 om 22:03 schreef André Rodier: On 23/04/18 14:18, Stephan Bosch wrote: Op 11-4-2018 om 23:58 schreef André Rodier: Hello, I have tested the sieve duplicate script with success so far, but I have a question. Sieve duplicate script? You mean the Sieve duplicate extension (RFC 7352)? I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. It uses the lda-dupes file in the user's home directory. So, it is not normally related to folders, although the identifier used for duplicate matching could be composed of the mailbox name if you want. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. Are you talking about IMAPSieve now? I am not sure "duplicate" is currently even allowed in that context. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. You can set the :uniqueid parameter accordingly. Regards, Stephan. Thank you, Stephan. Yes, I meant the Sieve duplicate extension. I am using a program to import email (mbsync), which use the IMAP append function. Sometimes, the import fail and I have to restart the program. Unfortunately, the same emails are imported again. I found a fix by using a dovecot IMAP sieve script executed on the APPEND action (https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve). I wrote a custom sieve script that "discard" the ones that are detected as "duplicate". It worked very well and the emails were not any more imported twice. However, there was a huge side effect: archiving an email with Thunderbird is not working any more, and even lost! I have been able to understand the error as this: 1. When archiving an email with Thunderbird, it is first copied (APPEND) into the archive folder, but the original folder is not expunged. 2. The sieve script detect the email as duplicate, and discard it. 3. When the original folder is expunged, the source email is lost... My conclusion was the duplicate detection function is global to all folders. If I could restrict the detection of duplicates in the current folder only, this would let me run the import program again without error. Specify the ID used for duplicate checking explicitly using the :uniqueid argument (https://tools.ietf.org/html/rfc7352#section-3.1). Using the variables extenion, compose the uniqueid from the message-id and the mailbox name. Regards, Stephan.
Re: replication and .dovecot.lda-dupes
Op 25-4-2018 om 14:37 schreef Patrick Cernko:
Hi list,
it's been 2 months now since my initial posting (s.b.). I wonder if I
could get at least a "still working on it" statement from the devs or
something like that?
We have plans to make the lda-dupes database a dict. In that case
there's much more flexibility to make this work across replication.
Regards,
Stephan.
On 22.02.2018 16:42, Patrick Cernko wrote:
Hi list,
this question was already posted a few years ago
(https://www.dovecot.org/list/dovecot/2014-November/098585.html). I
already asked the original queriest and he told me, that he never got an
solution or workaround but it was not important enough for him.
When using replication in conjunction with sieve vacations, the
.dovecot.lda-dupes file is not synced with the other server. So when
delivering to both servers (round-robin or randomized), senders might
get more vacation mails than configured as the other server does not
know, that the first one already sent a vacation message.
Is this a bug or intentional? If it is a bug, I hereby ask for a fix,
please.
We are using Dovecot version 2.2.27 on Debian/stretch.
This is dovecot -n (hostnames anonymized):
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.76.1.amd64-smp x86_64 Debian 9.3
auth_verbose = yes
default_vsz_limit = 2 G
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
login_log_format_elements = pid=%p user=<%u> method=%m rip=%r lip=%l
mpid=%e %c
mail_attachment_dir = /IMAP/mail/attachments
mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix
mail_home = /IMAP/mail/mailboxes/%u
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u[%p]): "
mail_max_userip_connections = 0
mail_plugins = " notify replication zlib fts fts_squat"
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_autoindex = yes
fts_squat = partial=4 full=10
mail_replica = tcp:other-server
sieve = file:~/sieve;active=~/.dovecot.sieve
zlib_save = gz
zlib_save_level = 3
}
postmaster_address = <>
protocols = " imap lmtp sieve"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service anvil {
client_limit = 2250
}
service auth {
client_limit = 2447
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service imap-login {
inet_listener imap {
port = 0
}
process_limit = 2047
}
service imap {
process_limit = 2047
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_cert = iterate_filter =
(&(objectClass=posixAccount)(istMailHomeServer=servername))
auth_bind = yes
/etc/dovecot/userdb.override is currently an empty file.
P.S.: Although I do not think, that this has something to do with our
problem, I should mention that we run SIS with system uids. This needs
some permission and posixacl tweaks on /IMAP/mail/attachments to work:
# force all files created in this dir recursively) to have mode 0666
setfacl -d -m group::rwx -m o:rwx /IMAP/mail/attachments
(you have to enable posixacls for the corresponding filesystem!)
# trick dovecot to inherit parent dir's permissions (recursively)
chmod 2777 /IMAP/mail/attachments
Feel free to add these tweaks to the SIS documentation. ;-)
Best Regards,
Best regards,
Re: replication and .dovecot.lda-dupes
Hi list,
it's been 2 months now since my initial posting (s.b.). I wonder if I
could get at least a "still working on it" statement from the devs or
something like that?
On 22.02.2018 16:42, Patrick Cernko wrote:
Hi list,
this question was already posted a few years ago
(https://www.dovecot.org/list/dovecot/2014-November/098585.html). I
already asked the original queriest and he told me, that he never got an
solution or workaround but it was not important enough for him.
When using replication in conjunction with sieve vacations, the
.dovecot.lda-dupes file is not synced with the other server. So when
delivering to both servers (round-robin or randomized), senders might
get more vacation mails than configured as the other server does not
know, that the first one already sent a vacation message.
Is this a bug or intentional? If it is a bug, I hereby ask for a fix,
please.
We are using Dovecot version 2.2.27 on Debian/stretch.
This is dovecot -n (hostnames anonymized):
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.76.1.amd64-smp x86_64 Debian 9.3
auth_verbose = yes
default_vsz_limit = 2 G
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
login_log_format_elements = pid=%p user=<%u> method=%m rip=%r lip=%l
mpid=%e %c
mail_attachment_dir = /IMAP/mail/attachments
mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix
mail_home = /IMAP/mail/mailboxes/%u
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u[%p]): "
mail_max_userip_connections = 0
mail_plugins = " notify replication zlib fts fts_squat"
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_autoindex = yes
fts_squat = partial=4 full=10
mail_replica = tcp:other-server
sieve = file:~/sieve;active=~/.dovecot.sieve
zlib_save = gz
zlib_save_level = 3
}
postmaster_address = <>
protocols = " imap lmtp sieve"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service anvil {
client_limit = 2250
}
service auth {
client_limit = 2447
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service imap-login {
inet_listener imap {
port = 0
}
process_limit = 2047
}
service imap {
process_limit = 2047
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_cert =
Best regards,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
