Re: SIS: dovecot creates duplicated attachments (sometimes)

[Alexander Moisseev]

It's been 2 years since my previous post, but the issue is still present 
(Dovecot 2.3.2).

Long story short, SIS attachment deduplication was broken in Dovecot v2.2.16. 
When it was fixed in v2.2.22 I wrote this script to deduplicate attachments on 
my servers back: 
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck
Sometimes I run the script and it finds new duplicated attachments (no too 
many). If you check these attachments, you find out that the attachments has 
the same hash, the same size, match if compare them byte by byte, were created 
at the same time, but one of them has a different inode number.

For instance, 3 days ago it was:
  Attachments processed: 31535
  Duplicated attachments: 370
  Missed hash files: 1313

And today:
  Attachments processed: 32368
  Duplicated attachments: 381
  Missed hash files: 1317

So, 11 new duplicates and 4 new missed hashes.

Can someone suggest a way to debug this problem somehow?

As a lot of time has passed since my previous post, I repost it below.

On 5/16/2016 9:27 AM, Alexander Moisseev wrote:

SIS deduplication was broken in 2.2.16 and has been fixed with 
https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1
but still sometimes (just several times in a month or so) dovecot creates 
duplicated attachments.

As you can see in directory listings below all attachments was created at 
(about) the same time, but one of them has different inode number.


It easy to discover using 
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck

# dsisck -n

mail_uid=vmail
mail_attachment_dir=/vmail/attachments

==> Checking SIS...

# ln -f 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
# mkdir -p -m 700 ./64/8f/hashes && ln 
./64/8f/648f5cfa27af6d20c8570fdcaeab997663e15105-55430d2cd4432c571cb63d96bafd 
./64/8f/hashes/648f5cfa27af6d20c8570fdcaeab997663e15105
# ln -f 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
# ln -f 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd

--
  Unexpected objects found: 0

  Attachments processed: 43500
  Different attachments with similar hashes skipped: 0

  Attachments deduplicated: 3
  File system blocks freed up: 1800
  Attachment deduplication attempts failed: 0

  Hash files created: 1
  Hash files creation attempts failed: 0

  Hash files re-linked: 0
  Hash files re-link attempts failed: 0

  Orphaned hash files: 0
  Invalid hash files (not a regular file): 0
--

# cd /vmail/attachments


# ls -li ./30/b3/ ./30/b3/hashes/
./30/b3/:
total 1996
7464083 -rw---  1 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bb6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bd6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bf6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c36eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
7464084 drwx--  2 vmail  vmail 512 May 10 09:16 hashes

./30/b3/hashes/:
total 332
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545

# ls -li ./6f/3f/ ./6f/3f/hashes/
./6f/3f/:
total 644
15088414 -rw---  1 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
15088448 drwx--  2 vmail  vmail 512 May 13 11:23 hashes

./6f/3f/hashes/:
total 320
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe

# ls -li ./dd/d9/ ./dd/d9/hashes/
./dd/d9/:
total 748
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-68f7c122fefb3157ad353d96bafd
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
80547 -rw---  1 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd
80549 drwx--  2 vmail  vmail 512 May 10 18:19 hash

Re: /var/run/dovecot permission issues

[Alexander Moisseev]

On 8/17/2017 7:07 PM, Larry Rosenman wrote:

In /usr/local/etc/dovecot/conf.d/90-plugin.conf:



It should be enough to just set permissions as other options are defaults.

/usr/local/etc/dovecot/conf.d/10-master.conf :

service stats {
  fifo_listener stats-mail {
mode = 0666
  }
  fifo_listener stats-user {
mode = 0666
  }
  unix_listener stats {
mode = 0666
  }
}

BTW I'm not sure if write permissions on 'stats-user' and 'stats' listeners are 
required for metrics service.
At least I have no evidence if Dovecot ever tried to write to that listeners.
Probably it is enough to set write permissions on 'stats-mail'.

Re: External mail attachments storage cleanup

[Alexander Moisseev]

On 29.06.2016 13:06, Николай Мананков wrote:

 Hi!

Thanks it worked! But only when I have a specific user instead wldcard (i mean 
-u *@example.org).  Now I think I need to write a script that searches for 
users in the domain , and starts each of them this command . And apparently it 
is necessary to add to the cron job..



Hi,

The script:
doveadm-expunge - iterates over passwd-file databases, expunges messages in 
Junk and Trash folders and purges mailboxes for every user.
https://github.com/moisseev/doveadm-tools/blob/master/bin/doveadm-expunge

The crontab entry:
5   4   *   *   *   /usr/local/bin/doveadm-expunge

SIS: dovecot creates duplicated attachments (sometimes)

[Alexander Moisseev]

SIS deduplication was broken in 2.2.16 and has been fixed with 
https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1
but still sometimes (just several times in a month or so) dovecot creates 
duplicated attachments.

As you can see in directory listings below all attachments was created at 
(about) the same time, but one of them has different inode number.


It easy to discover using 
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck

# dsisck -n

mail_uid=vmail
mail_attachment_dir=/vmail/attachments

==> Checking SIS...

# ln -f 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
 
./30/b3/30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
# mkdir -p -m 700 ./64/8f/hashes && ln 
./64/8f/648f5cfa27af6d20c8570fdcaeab997663e15105-55430d2cd4432c571cb63d96bafd 
./64/8f/hashes/648f5cfa27af6d20c8570fdcaeab997663e15105
# ln -f 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
 
./6f/3f/6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
# ln -f 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
 
./dd/d9/ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd

--
 Unexpected objects found: 0

 Attachments processed: 43500
 Different attachments with similar hashes skipped: 0

 Attachments deduplicated: 3
 File system blocks freed up: 1800
 Attachment deduplication attempts failed: 0

 Hash files created: 1
 Hash files creation attempts failed: 0

 Hash files re-linked: 0
 Hash files re-link attempts failed: 0

 Orphaned hash files: 0
 Invalid hash files (not a regular file): 0
--

# cd /vmail/attachments


# ls -li ./30/b3/ ./30/b3/hashes/
./30/b3/:
total 1996
7464083 -rw---  1 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-7ad0a411d17c31572b7a01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bb6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bd6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-bf6eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c36eae04a67c3157287f01003d96bafd
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545-c56eae04a67c3157287f01003d96bafd
7464084 drwx--  2 vmail  vmail 512 May 10 09:16 hashes

./30/b3/hashes/:
total 332
7464082 -rw---  6 vmail  vmail  337719 May 10 09:16 
30b367c584a123eee59478adf3e4f4c9e1226545

# ls -li ./6f/3f/ ./6f/3f/hashes/
./6f/3f/:
total 644
15088414 -rw---  1 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-72da1f32e38e3557947b3d96bafd
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe-b1935817ed75355754673d96bafd
15088448 drwx--  2 vmail  vmail 512 May 13 11:23 hashes

./6f/3f/hashes/:
total 320
15088422 -rw---  2 vmail  vmail  326656 May 13 11:23 
6f3fa3e4d374a9c80d07af54960ce0e7adb2e0fe

# ls -li ./dd/d9/ ./dd/d9/hashes/
./dd/d9/:
total 748
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-68f7c122fefb3157ad353d96bafd
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-6bf7c122fefb3157ad353d96bafd
80547 -rw---  1 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b-90fd1415e7e5315711283d96bafd
80549 drwx--  2 vmail  vmail 512 May 10 18:19 hashes

./dd/d9/hashes/:
total 248
80548 -rw---  3 vmail  vmail  250640 May 10 18:19 
ddd97aa6f624d4f54968d2c4956fc3a9d796b31b


# doveconf -n
# 2.2.24 (a82c823): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.14 (099a97c)
# OS: FreeBSD 10.3-RELEASE i386
auth_default_realm = example.com
auth_mechanisms = digest-md5 cram-md5 plain apop
doveadm_password =  # hidden, use -P to show it
first_valid_gid = 1000
first_valid_uid = 1000
lda_mailbox_autosubscribe = yes
listen = *
mail_attachment_dir = /vmail/attachments
mail_gid = vmail
mail_home = /vmail/%d/%n
mail_location = mdbox:~/mdbox
mail_plugins = quota zlib acl
mail_shared_explicit_inbox = yes
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext
namespace {
  list = children
  location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%u
  prefix = shared/%%u/
  separator 

Re: sis deduplication broken from 2.2.16 upwards

[Alexander Moisseev]

On 11.03.2016 3:56, Timo Sirainen wrote:

So, after the fix is applied, does dovecot silently delete the
duplicated files, or is there a command that needs to be run manually?


You'd have to do it manually in some way. A script that does something like:

Go through all attachment directories and for each file:
 - Sort files by filename
 - Identify that files A and B the same (beginning of the filename begins with 
same hash), but have a different inode
 - ln A B.tmp && mv B.tmp B



The problem turned out to be a bit more complicated than that.

Finally a came up with that script:
https://github.com/moisseev/doveadm-tools/blob/master/bin/dsisck

It assumes Dovecot should not run.

Re: How to delete Dovecot users when using Single Instance Storage

[Alexander Moisseev]

On 19.03.2016 18:56, Jesus Cea wrote:

In case somebody find this useful:

https://blog.jcea.es/posts/20160211-delete_users_dovecot.html


"The expunge step" didn't work because "all" parameter was broken in 2.2.21. 
It's fixed now: https://github.com/dovecot/core/commit/6971937a6f3e93844dbd43bdbe903628e21a9422
Also as workaround you can replace 'all' with '1:*'.

Here is a script that automates the procedure:
https://github.com/moisseev/doveadm-tools/blob/master/bin/doveadm-mdboxrm

Re: sis deduplication broken from 2.2.16 upwards

[Alexander Moisseev]

On 11.03.16 3:56, Timo Sirainen wrote:



On 11 Mar 2016, at 02:37, Charles Marcus  wrote:

On 3/9/2016 9:02 PM, Timo Sirainen  wrote:

On 08 Mar 2016, at 01:50, Pavel Stano  wrote:


sis attachment deduplication is broken in 2.2.16 upwards.
It is caused by this commit.
https://github.com/dovecot/core/commit/664bf3e236c214aee86294483c379e4fa66c2e63

in src/lib-fs/fs-sis.c function fs_sis_try_link() is comparation of
inodes of hash files.
Because fs_stat() after that commit use fstat() on open fd of temporary
file instead of stat on filename. But that temporary file has differnt
inode.

It not cause any corruption but it will not save any space.
Because every duplicate attachment will be in separate file.

Thanks, fixed: 
https://github.com/dovecot/core/commit/3b39022ea0513363241cf852b7d454c841584ea1


So, after the fix is applied, does dovecot silently delete the
duplicated files, or is there a command that needs to be run manually?


You'd have to do it manually in some way. A script that does something like:

Go through all attachment directories and for each file:
 - Sort files by filename
 - Identify that files A and B the same (beginning of the filename begins with 
same hash), but have a different inode
 - ln A B.tmp && mv B.tmp B

 
I've also found that many of /hashes/ directories have missed.


# ll /tank1/vmail/attachments/1f/1f
total 3300
-rw---  1 vmail  vmail   403976 12 ноя 00:20 
1f1f504c582600a2af94b39c088692aba714fe72-c53b9e1508b14356797d0100d09efc50
-rw---  1 vmail  vmail   403976 12 ноя 00:20 
1f1f504c582600a2af94b39c088692aba714fe72-c93b9e1508b14356797d0100d09efc50
-rw---  1 vmail  vmail   403976 12 ноя 00:20 
1f1f504c582600a2af94b39c088692aba714fe72-f2a777181eb14356807d0100d09efc50
-rw---  1 vmail  vmail   403976 12 ноя 00:20 
1f1f504c582600a2af94b39c088692aba714fe72-f31a5e2917b143567e7d0100d09efc50
-rw---  1 vmail  vmail  2582016  3 ноя 00:20 
1f1f97880e8cddc2dfe3c4ad2654b9da937226b7-94c53d358bd33756d614d09efc50

Is it related to the same bug or there is another issue?
Is it safe to delete attachment files if there is no file with the same hash in 
the /hashes/ directory or there is no /hashes/ directory at all?

Re: [bug][Pigeonhole 0.4.10] Missing msgid

[Alexander Moisseev]

On 22.12.15 2:06, Stephan Bosch wrote:


I am quite confident that this fixes it:
http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/37ddc42c007c

Regards,
Stephan.


Yes, that change fixed the issue. Thanks!

Re: [bug][Pigeonhole 0.4.10] Missing msgid

[Alexander Moisseev]

On 16.12.15 12:32, Stephan Bosch wrote:


That means that LTMP performed that delivery. That would normally only
happen when an error occurs.


No errors. All mail is delivering as expected.


Do you see anything strange in the user log? Does system log show
anything more when mail_debug=yes is active?



There is nothing unusual in the log.
Dec 16 12:43:29 mx dovecot: lmtp(72024): Connect from local
Dec 16 12:43:29 mx dovecot: lmtp(use...@example.com): bD6dNzIycVZYGQEAPZa6/Q: 
sieve: stored mail into mailbox 'INBOX'
Dec 16 12:43:29 mx postfix/lmtp[72023]: 7C9D836: to=, 
relay=mx.mezonplus.ru[private/dovecot-lmtp], delay=0.05, delays=0.02/0/0/0.02, dsn=2.0.0, 
status=sent (250 2.0.0  bD6dNzIycVZYGQEAPZa6/Q Saved)
Dec 16 12:43:29 mx dovecot: lmtp(72024): Disconnect from local: Successful quit
Dec 16 12:43:29 mx postfix/qmgr[70108]: 7C9D836: removed


What is your configuration?

Attached.

mx# doveadm config -n
# 2.2.21 (5345f22): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.10 (d61ff8a5af9e+)
# OS: FreeBSD 10.2-RELEASE i386
auth_default_realm = mezonplus.ru
auth_mechanisms = digest-md5 cram-md5 plain apop
doveadm_password =  # hidden, use -P to show it
first_valid_gid = 1000
first_valid_uid = 1000
lda_mailbox_autosubscribe = yes
listen = *
mail_attachment_dir = /vmail/attachments
mail_gid = vmail
mail_home = /vmail/%d/%n
mail_location = mdbox:~/mdbox
mail_plugins = quota zlib acl
mail_shared_explicit_inbox = yes
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart
namespace {
  list = children
  location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%u
  prefix = shared/%%u/
  separator = /
  subscriptions = no
  type = shared
}
namespace inbox {
  inbox = yes
  location =
  mailbox Archives {
special_use = \Archive
  }
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
auto = subscribe
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix =
  separator = /
}
passdb {
  args = scheme=plain username_format=%n 
/usr/local/etc/dovecot/dovecot.auth/%d.passwd
  driver = passwd-file
}
plugin {
  acl = vfile
  acl_shared_dict = file:/vmail/shared-mailboxes.db
  antispam_backend = spool2dir
  antispam_spam = Junk
  antispam_spool2dir_notspam = /tmp/spamspool/%%020lu-%u-%%05luh
  antispam_spool2dir_spam = /tmp/spamspool/%%020lu-%u-%%05lus
  antispam_trash = Trash;train_ham;train_prob;train_spam
  quota = dict:User quota::file:%h/dovecot-quota
  quota_rule = *:storage=2G
  quota_rule2 = Trash:storage=+10%%
  quota_rule3 = Spam:storage=+20%%
  quota_status_nouser = DUNNO
  quota_status_success = DUNNO
  sieve_after = /usr/local/etc/dovecot/sieve/sieve.after
  sieve_before = /usr/local/etc/dovecot/sieve/sieve.before
  sieve_vacation_min_period = 0
  zlib_save = gz
  zlib_save_level = 3
}
postmaster_address = postmas...@mezonplus.ru
protocols = imap lmtp sieve pop3
quota_full_tempfail = yes
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener auth-userdb {
user = vmail
  }
}
service config {
  unix_listener config {
mode = 0600
user = vmail
  }
}
service imap-login {
  inet_listener imaps {
port = 0
  }
  process_limit = 200
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
  }
  user = vmail
}
service pop3-login {
  inet_listener pop3s {
port = 0
  }
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  unix_listener /var/spool/postfix/private/quota-status {
user = postfix
  }
}
ssl_cert = 

[bug][Pigeonhole 0.4.10] Missing msgid

[Alexander Moisseev]

Pigeonhole 0.4.10 doesn't log msgid for some messages.

Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): EbLlL2gLcVYCBgEAPZa6/Q: sieve: 
msgid=<20151216065756.be1df1f...@cp.bitrix24.com>: forwarded to 

Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): EbLlL2gLcVYCBgEAPZa6/Q: 
sieve: stored mail into mailbox 'INBOX'
Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): FrLlL2gLcVYCBgEAPZa6/Q: sieve: 
msgid=<20151216065756.c77bb1f...@cp.bitrix24.com>: forwarded to 

Dec 16 09:57:57 mx dovecot: lmtp(us...@example.com): FrLlL2gLcVYCBgEAPZa6/Q: sieve: 
msgid=<20151216065756.c77bb1f...@cp.bitrix24.com>: stored mail into mailbox 
'bitrix'

2.2.21: doveadm expunge is broken

[Alexander Moisseev]

# doveadm expunge -u u...@example.com MAILBOX Trash ALL
Fatal: expunge: To avoid accidents, each branch in search query must contain something 
else besides MAILBOX (e.g. just add "all" if you want everything)

--
Alexander

Re: users of dbox format

[Alexander Moisseev]

27.02.2015 18:14, Paolo Cravero пишет:


mdbox is THE reason why I am trying Dovecot. With mailboxes of several (tens
of) GB with several k of messages I hope mdbox will speedup backups.


What backup method are you using?

I am doing one-way synchronization to mdbox+SIS at separate ZFS filesystem of 
the backup server and make snapshots to have backup history.

I used to backup with rsync previously. After was I switched to doveadm-backup 
the backup speed increased dramatically.

Re: users of dbox format

[Alexander Moisseev]

27.02.2015 17:42, Andreas Kasenides пишет:

I am interested in finding out your experiences with using the dbox format 
(especially mdbox) if you use this format.


I am using mdbox+SIS on small production server about 2 months. It works well 
for me. But I have had some difficulties using doveadm tools in particular 
situations. Mostly because of some parts of documentation wasn't clear enough 
for me.


I am contemplating changing my maildir setup to mdbox but I still need to make 
a case for it
against maildir which has become a de-facto standard and provides sort of a 
secure basis in case of
software changes.
 
You can convert from mdbox to maildir at any time. Also you can use both formats on the same server simultaneously.


--
Alexander

Re: curious when certain patches might become part of a release

[Alexander Moisseev]

13.02.2015 20:24, Benjamin Connelly пишет:

We ran in to this same problem others discussed in this thread:

http://dovecot.org/list/dovecot/2014-November/098927.html

and have also applied the patches (31262a892ba7 and 80ed82a93c1a) and it
seems to have stopped the panics. We usually just use the FreeBSD ports
tree to install software, so I'm looking forward to the time when we can
go back to that. Will those patches be part of the next release?


you have to ask this question the FreeBSD maintainer of the Dovecot
package.


But the FreeBSD ports tree is up to the latest 2.2.15 -- yet we were
still experiencing the panics, and had to apply those patches. So I'm
wondering more about what version release of Dovecot itself will include
those mbox sync patches. . .

  Ben


Do you mind to open a PR in the FreeBSD Bugzilla?

--
Alexander

Re: curious when certain patches might become part of a release

[Alexander Moisseev]

13.02.2015 21:38, Benjamin Connelly пишет:

The bug wasn't specific to FreeBSD (or the FreeBSD
port.) It's apparently fixed by 31262a892ba7 and 80ed82a93c1a


Of course it is not a port bug. I minded you can ask port maintainer to include 
those patches into the port.

Re: How to Delete an user and Purge attachements with a common directory mail_attachment_dir with SIS for all users.

[Alexander Moisseev]

On 12.02.2015 12:00, TN wrote:

 Hello Alexander,

 Thank you for your answer, it is very helpfull for me.

Just a little addon with your process to complet the deletion an user :

# doveadm expunge -d -u johndoe mailbox '*' all
# doveadm -v purge -u johndoe  (this command 
force attachment deletion immediatly )
# rm -rf /var/mail/johndoe

Thank you

PS: Sorry to not reply to the dovecot list because i setup a digest for all 
answers.



Hi,

Many thanks for your reply. It's a *very important* addition actually.

There is lack of documentation on SIS. So I have done some experimenting and 
found out that the command
# doveadm purge -u johndoe
*is mandatory*. Without this step attachements will stay in mail_attachment_dir 
forever.

Man doveadm-purge(1) states doveadm-purge - Remove messages with refcount=0 from 
mdbox files, but actually it also removes attachments from mail_attachment_dir.

--
Alexander

Re: How to Delete an user and Purge attachements with a common directory mail_attachment_dir with SIS for all users.

[Alexander Moisseev]

10.02.2015 18:36, TN пишет:

What is the best way to delete an user mailbox (ex: /var/mail/johndoe ) 
when all attachments  (for all users) are in a common directory with SIS 
deduplication (ex: mail_attachment_dir = /var/mail/attachments ) ?
Trying to delete user mailbox directory (rm) and do the command  : doveadm -v 
purge -u johndoeleave  all johndoe's attachments orphelin.



You should always expunge mailbox *before* deleting it, if you are using SIS.

# doveadm expunge -d -u johndoe mailbox '*' all
# rm -rf /var/mail/johndoe

https://www.mail-archive.com/dovecot@dovecot.org/msg41683.html

--
Alexander

[Bug ?] IMAP keywords are not preserved during mailbox conversion

[Alexander Moisseev]

IMAP keywords (Thunderbird Tags) are not preserved during Maildir to mdbox 
conversion.

For mailbox conversion I am using the command:
# doveadm -o mail_location=maildir:~/Maildir sync -u u...@example.com 
mdbox:~/mdbox

After conversion all Thunderbird Tags are lost.

Is it a bug or expected behavior?

--
Alexander
# doveconf -n
# 2.2.15: /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6 (3e924b1b6c5c+)
# OS: FreeBSD 10.1-RELEASE i386
auth_default_realm = example.com
auth_mechanisms = digest-md5 cram-md5 apop plain
doveadm_password = 
first_valid_gid = 1000
first_valid_uid = 1000
lda_mailbox_autosubscribe = yes
listen = *
mail_attachment_dir = /vmail/attachments
mail_gid = vmail
mail_home = /vmail/%d/%n
mail_location = mdbox:~/mdbox
mail_plugins = quota zlib
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave duplicate
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
auto = subscribe
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix =
  separator = /
}
passdb {
  args = scheme=plain username_format=%n 
/usr/local/etc/dovecot/dovecot.auth/%d.passwd
  driver = passwd-file
}
plugin {
  antispam_backend = mailtrain
  antispam_mail_notspam = learn_ham
  antispam_mail_sendmail = /usr/local/bin/rspamc
  antispam_mail_sendmail_args = -h;localhost:11334
  antispam_mail_spam = learn_spam
  antispam_spam = Junk
  antispam_trash = Trash
  quota = dict:User quota::file:%h/dovecot-quota
  quota_rule = *:storage=2G
  quota_rule2 = Trash:storage=+10%%
  quota_rule3 = Spam:storage=+20%%
  quota_status_nouser = DUNNO
  quota_status_success = DUNNO
  sieve_after = /usr/local/etc/dovecot/sieve/sieve.after
  sieve_before = /usr/local/etc/dovecot/sieve/sieve.before
  sieve_vacation_min_period = 0
  zlib_save = gz
  zlib_save_level = 3
}
postmaster_address = postmas...@example.com
protocols = imap pop3 lmtp sieve
quota_full_tempfail = yes
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener auth-userdb {
user = vmail
  }
}
service config {
  unix_listener config {
mode = 0600
user = vmail
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
  }
  user = vmail
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  unix_listener /var/spool/postfix/private/quota-status {
user = postfix
  }
}
ssl_cert = /etc/ssl/certs/mx.example.com.crt
ssl_key = /etc/ssl/private/mx.example.com.key
userdb {
  args = username_format=%n /usr/local/etc/dovecot/dovecot.auth/%d.passwd
  driver = passwd-file
}
verbose_proctitle = yes
protocol imap {
  mail_plugins = quota zlib antispam imap_quota imap_zlib
}
protocol lmtp {
  mail_plugins = quota zlib sieve
}
#

dsync-remote: Error: Failed to set attribute

[Alexander Moisseev]

1. I am looking for backup solution for mdbox with SIS and experimenting with 
dsync.
I am planning to do it like that:
doveadm backup [deSISing] = transmitting to remote host = dsync-server 
[SISing]

Is there more convenient method?

2. I have dovecot 2.2.15 on both hosts. The local host is a mail server. 
Dovecot on the remote host neither running nor configured (I have created an 
empty dovecot.conf in the config dir).

When I tried to back up mailbox from local to remote host with command

doveadm backup -P -u u...@domain.tld \
ssh -c blowfish -i /tmp/vmail/.ssh/id_dsa vmail@remotehost \
doveadm \
-o mail_location=mdbox:/tmp/vmail/domain.tld/user/mdbox \
-o mail_attachment_dir=/tmp/vmail/attachments \
dsync-server

I got error messages:

dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute 
vendor/vendor.dovecot/pvt/server/sieve/files/roundcube: Internal attributes 
cannot be changed directly
dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute 
vendor/vendor.dovecot/pvt/server/sieve/files/main: Internal attributes cannot 
be changed directly
dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute 
vendor/vendor.dovecot/pvt/server/sieve/files/test: Internal attributes cannot 
be changed directly
dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute 
vendor/vendor.dovecot/pvt/server/sieve/default: Internal attributes cannot be 
changed directly

What do that errors mean? Does dsync try to replicate sieve scripts?

--
Alexander

Re: [Dovecot] May 05 07:20:21 imap: Warning: Time jumped forwards 16 seconds

[Alexander Moisseev]

Are you got this warnings only after OS reboot?

--
Alexander Moisseev

Re: [Dovecot] Time just moved backwards error even with ntpd

[Alexander Moisseev]

On some FreeBSD 6.2 systems I was solved this problem just by adding to crontab:
@hourly/sbin/sysctl `/sbin/sysctl -e machdep.adjkerntz` /dev/null 21

For some explanation, see:
http://lists.freebsd.org/pipermail/freebsd-hackers/2005-August/013383.html

Quote follows:

... It seems that CMOS clock is updated using resettodr(9) function. There
seem to be only a few occasions when this function is called:

1. clock_settime(ClOCK_REALTIME) // through kern_time.c:settime()
2. settimeofday() // through kern_time.c:settime()
3. machdep.adjkerntz sysctl is set

I believe that ntpd calls *settime-family* functions *only* if time offset
is *larger* than *128ms* (or maybe I am thinking about ntpdate), but
normally it uses ntp_adjtime() to adjust time keeping. Obviosuly, a
system running ntpd with good enough hardware clock and good enough
connection to good enough ntp server(s) would use the latter method all
of the time after some initial stabilization period. But it seems that
time changes are *never propagated to CMOS* in this case, ...

Alexander

[Dovecot] Maildir POP3 UID larger than next_uid bug

[Alexander Moisseev]

dovecot 1.0.0; Free BSD 6.2; x86; UFS; no NFS
IMAP not uses at all, POP3 only

Problem description:
POP3 client can't fetch second e-mail message from Maildir if first POP3 
connect occurs before any message placed in Maildir.

Details:

1. POP3-client first time tries to receive mail:

May 14 16:17:36 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], 
method=PLAIN, rip=192.168.0.100, lip=192.168.0.3
May 14 16:17:36 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out 
top=0/0, retr=0/0, del=0/0, size=0

dovecot creates Maildir for that user:

/var/mail/virtual/mydomain.ru/user/Maildir/
/cur
/new
/tmp
dovecot.index
dovecot.index.cache
dovecot.index.log

2. Postfix MTA receives message for that user and places it at
/var/mail/virtual/mydomain.ru/user/Maildir/new/1179145269.V43I3f5a4M116738.host.mydomain.local

3. POP3-client tries to receive mail:

May 14 16:23:09 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], 
method=PLAIN, rip=192.168.0.100, lip=192.168.0.3
May 14 16:23:09 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out 
top=0/0, retr=1/786, del=1/1, size=770


dovecot creates file dovecot-uidlist with next_uid=1 (!!!):

1 1179145056 1
1 1179145269.V43I3f5a4M116738.host.mydomain.local:2,

Message fetched successfully.

4. POP3-client tries to receive mail:

May 14 16:23:29 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], 
method=PLAIN, rip=192.168.0.100, lip=192.168.0.3
May 14 16:23:29 host dovecot: POP3([EMAIL PROTECTED]): UID larger than next_uid in 
file /var/mail/virtual/mydomain.ru/user/Maildir/dovecot-uidlist (1 = 1)
May 14 16:23:29 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out 
top=0/0, retr=0/0, del=0/0, size=0

File dovecot-uidlist disappears.

5. Postfix MTA receives message for that user and places it at
/var/mail/virtual/mydomain.ru/user/Maildir/new/1179145534.V43I3f5a4M116827.host.mydomain.local

6. POP3-client tries to receive mail:

May 14 16:47:20 host dovecot: pop3-login: Login: user=[EMAIL PROTECTED], 
method=PLAIN, rip=192.168.0.100, lip=192.168.0.3
May 14 16:47:20 host dovecot: POP3([EMAIL PROTECTED]): Disconnected: Logged out 
top=0/0, retr=0/0, del=0/0, size=0

Dovecot creates file dovecot-uidlist:
1 1179145056 2
1 1179145534.V43I3f5a4M116827.host.mydomain.local

Dovecot moves message from new to cur directory and adds flag :2,:

/var/mail/virtual/mydomain.ru/user/Maildir/cur/1179145534.V43I3f5a4M116827.host.mydomain.local:2,

Message not fetched. And POP3 client can't fetch it at all.

All next messages fetches normally as they arrives.

# /usr/local/etc/dovecot.conf
protocols: pop3 pop3s
ssl_cert_file: /etc/ssl/certs/host.mydomain.ru.crt
ssl_key_file: /etc/ssl/private/host.mydomain.ru.key
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable: /usr/local/libexec/dovecot/pop3-login
login_greeting: POP3 server ready
verbose_proctitle: yes
first_valid_gid: 0
mail_extra_groups: mail
mail_location: maildir:~/Maildir
mail_executable: /usr/local/libexec/dovecot/pop3
mail_plugin_dir: /usr/local/lib/dovecot/pop3
pop3_uidl_format: %08Xu%08Xv
pop3_client_workarounds: outlook-no-nuls oe-ns-eoh
auth default:
 mechanisms: plain login digest-md5 cram-md5 apop ntlm
 default_realm: mydomain.ru
 passdb:
   driver: pam
 passdb:
   driver: passwd-file
   args: /etc/auth/%d/passwd
 userdb:
   driver: passwd
 userdb:
   driver: static
   args: uid=5000 gid=5000 home=/var/mail/virtual/%d/%n
 socket:
   type: listen
   client:
 path: /var/spool/postfix/private/auth
 mode: 432
 user: postfix
 group: postfix
   master: