Re: object storage
> Timo Sirainen wrote: > Our typical projects have several million user accounts. I think 100k > users is around the minimum. Interesting. So che choice for object storage is driven by the number of accounts rather than the amount of data stored? For example I am heading towards 10 TB online with "just" 10k users. It wouldn't be worth? Paolo
Re: imap_logout_format improvements / ideas
> # %i - total number of bytes read from client
> # %o - total number of bytes sent to client
> # %{fetch_hdr_count} - Number of mails with mail header data sent to
> client
> # %{fetch_hdr_bytes} - Number of bytes with mail header data sent to
> client
> # %{fetch_body_count} - Number of mails with mail body data sent to client
> # %{fetch_body_bytes} - Number of bytes with mail body data sent to client
> # %{deleted} - Number of mails where client added \Deleted flag
> # %{expunged} - Number of mails that client expunged
> # %{trashed} - Number of mails that client copied/moved to the
> # special_use=\Trash mailbox.
>
> Any thoughts on what else would be useful?
Number of mails that have been \Seen ?
I have been running Dovecot with full debug logging so I am not aware if the
same information is already available elsewhere in the log at production
loglevel.
Paolo
Different mdbox_rotate_size for primary and alternate storage
Hello. In order to speed up backups of very very old messages I would like to set two different limits for mdbox_rotate_size. Like, 50M for primary storage and 100M or larger for alternate storage. There is no mention in docs or such a possibility, so I assume it is not possible. Is that correct? While I am at it, is it possible to configure primary storage as maildir (sturdy indexes) and altstorage as mdbox (more delicate indexes)? Thanks, Paolo
Re: acl and lazy_expunge plugins are incompatible?
Hello.
Before going into deep testing of 2.2.16 (clean install tbd), I would like
to know if someone is using succesfully both ACL and Lazy_expunge plugins.
Perhaps I had a config error?
Thanks,
Paolo
> Il 3 marzo 2015 alle 16.36 Paolo Cravero ha
> scritto:
>
>
> Hello list.
> Following two previous unanswered requests
> (http://www.dovecot.org/list/dovecot/2014-August/097449.html and
> http://www.dovecot.org/list/dovecot/2014-May/096261.html), I fell into the
> same problem: ACL and lazy_expunge plugins do not work together.
>
> I had lazy_expunge enabled working OK. Then I activated ACL plugin, and
> gave
> dovecot.2 visibily over paolo.cravero's Inbox: the IMAP connection fails
> on
> server-side. Relevant debug log is:
>
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: Quota
> grace: root=User quota bytes=576716800 (10%)
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: dict
> quota: user=paolo.crav...@csi.it,
> uri=file:/maildata1/home/csi.it/p/paolo.cravero/dovecot-quota,
> noenforcing=0
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: fs:
> root=/maildata1/main/csi.it/p/paolo.cravero,
> index=/maildata1/indexes/csi.it/p/paolo.cravero, indexpvt=, control=,
> inbox=, alt=/maildata1/altstorage/csi.it/p/paolo.cravero
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl:
> initializing backend with data: vfile
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl: acl
> username = paolo.crav...@csi.it
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl:
> owner
> = 1
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl
> vfile:
> Global ACLs disabled
> Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Fatal:
> lazy_expunge: Unknown namespace: '.EXPUNGED/'
>
>
> Removing the lazy_expunge, ACLs do work as expected and I can
> add/remove/use
> them.
>
>
> Is there a known compatibility issue? Or there's something wrong in my
> config and someone else is running with both plugins enabled?
>
> Thanks,
> Paolo
>
> PS: output of dovecot -n:
>
> # 2.2.15: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32-431.el6.x86_64 x86_64 Red Hat Enterprise Linux Server
> release 6.5 (Santiago) ext3
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> imap_client_workarounds = tb-extra-mailbox-sep
> lmtp_save_to_detail_mailbox = yes
> mail_attachment_dir = /maildata1/attachments
> mail_attachment_hash = %{sha256}
> mail_debug = yes
> mail_gid = 500
> mail_location =
> mdbox:/maildata1/main/%d/%1n/%n:ALT=/maildata1/altstorage/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n
> mail_plugins = quota mail_log notify lazy_expunge acl
> mail_uid = 500
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
> copy include variables body enotify environment mailbox date ihave
> duplicate
> mbox_write_locks = fcntl
> mdbox_rotate_size = 50 M
> namespace {
> hidden = no
> list = yes
> location =
> mdbox:/maildata1/main/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n:MAILBOXDIR=expunged
> prefix = .EXPUNGED/
> subscriptions = no
> }
> namespace {
> list = children
> location =
> mdbox:/maildata1/main/%%d/%%1n/%%n:ALT=/maildata1/altstorage/%%d/%%1n/%%n:INDEX=/maildata1/indexes/%%d/%%1n/%%n
> prefix = user/%%u/
> separator = /
> type = shared
> }
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
> }
> mailbox Junk {
> auto = subscribe
> special_use = \Junk
> }
> mailbox Sent {
> auto = subscribe
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> auto = subscribe
> special_use = \Trash
> }
> prefix =
> type = private
> }
> passdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> plugin {
> acl = vfile
> acl_shared_dict = file:/maildata1/db/shared-mailboxes.db
> lazy_expunge = .EXPUNGED/
> mail_log_events = delete undelete expunge copy mailbox_delete
> mailbox_rename flag_change append
> mail_log_fields = uid box msgid size flags vsize
> quota = dict:User quota::file:%h/dovecot-quota
> quota_grace = 10%%
> quota_rule = *:storage=10G
> quota_rule2 = Trash:storage=+100M
> quota_warning = storage=95%% quota-warning 95 %u
> quota_warning2 = storage=80%% quota-warning 80 %u
> }
> protocols = imap pop3 lmtp sieve
> service auth {
> unix_listener auth-userd
Re: Overriding dovecot.conf from Userdb Extras
Il 9 marzo 2015 alle 4.17 "E.B." ha scritto: > I thought I read that anything from dovecot.conf can be overridden in a > userdb lookup. Or a passdb lookup with "userdb_" prefix. > > But I tried for fun change log_path but it never worked. Is that because > logging is special, already started logging before it comes to the > passdb/userdb lookups? So are there some dovecot.conf settings > that cannot be overridden? To my understanding only these extra parameters can be tweaked through the userdb/passdb: http://wiki2.dovecot.org/PasswordDatabase/ExtraFields http://wiki2.dovecot.org/UserDatabase/ExtraFields (+ mail and quota_rule) Paolo
Re: Rebuilding SIS attachment links from log
Daniel, please help me understand, since I met your same problem on 2015-02-26 as you did. > 2. For each "failed: > read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, > a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists > i. If attachmentHash is missing display such for possible Is this a FEATURE of Dovecot SIS? Or a known bug described somewhere? Loosing detached attachments, I mean. > Can this be done via "pure" BASH? Need sed/awk as well? Well, you need a way to strip out the "-userHash" part from the string, so a little bit of sed and regex is needed. You also need to work out source and destination files. Still, personally I think the problem is somewhere else and should be fixed. Else, stop using SIS and let the SAN do the deduplication. Am I wrong? Paolo
acl and lazy_expunge plugins are incompatible?
Hello list.
Following two previous unanswered requests
(http://www.dovecot.org/list/dovecot/2014-August/097449.html and
http://www.dovecot.org/list/dovecot/2014-May/096261.html), I fell into the
same problem: ACL and lazy_expunge plugins do not work together.
I had lazy_expunge enabled working OK. Then I activated ACL plugin, and gave
dovecot.2 visibily over paolo.cravero's Inbox: the IMAP connection fails on
server-side. Relevant debug log is:
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: Quota
grace: root=User quota bytes=576716800 (10%)
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: dict
quota: user=paolo.crav...@csi.it,
uri=file:/maildata1/home/csi.it/p/paolo.cravero/dovecot-quota, noenforcing=0
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: fs:
root=/maildata1/main/csi.it/p/paolo.cravero,
index=/maildata1/indexes/csi.it/p/paolo.cravero, indexpvt=, control=,
inbox=, alt=/maildata1/altstorage/csi.it/p/paolo.cravero
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl:
initializing backend with data: vfile
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl: acl
username = paolo.crav...@csi.it
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl: owner
= 1
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Debug: acl vfile:
Global ACLs disabled
Mar 3 15:59:02 tst-msg03 dovecot: imap(doveco...@csi.it): Fatal:
lazy_expunge: Unknown namespace: '.EXPUNGED/'
Removing the lazy_expunge, ACLs do work as expected and I can add/remove/use
them.
Is there a known compatibility issue? Or there's something wrong in my
config and someone else is running with both plugins enabled?
Thanks,
Paolo
PS: output of dovecot -n:
# 2.2.15: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.el6.x86_64 x86_64 Red Hat Enterprise Linux Server
release 6.5 (Santiago) ext3
auth_mechanisms = plain login
disable_plaintext_auth = no
imap_client_workarounds = tb-extra-mailbox-sep
lmtp_save_to_detail_mailbox = yes
mail_attachment_dir = /maildata1/attachments
mail_attachment_hash = %{sha256}
mail_debug = yes
mail_gid = 500
mail_location =
mdbox:/maildata1/main/%d/%1n/%n:ALT=/maildata1/altstorage/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n
mail_plugins = quota mail_log notify lazy_expunge acl
mail_uid = 500
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave duplicate
mbox_write_locks = fcntl
mdbox_rotate_size = 50 M
namespace {
hidden = no
list = yes
location =
mdbox:/maildata1/main/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n:MAILBOXDIR=expunged
prefix = .EXPUNGED/
subscriptions = no
}
namespace {
list = children
location =
mdbox:/maildata1/main/%%d/%%1n/%%n:ALT=/maildata1/altstorage/%%d/%%1n/%%n:INDEX=/maildata1/indexes/%%d/%%1n/%%n
prefix = user/%%u/
separator = /
type = shared
}
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
type = private
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
acl = vfile
acl_shared_dict = file:/maildata1/db/shared-mailboxes.db
lazy_expunge = .EXPUNGED/
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
mail_log_fields = uid box msgid size flags vsize
quota = dict:User quota::file:%h/dovecot-quota
quota_grace = 10%%
quota_rule = *:storage=10G
quota_rule2 = Trash:storage=+100M
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
}
protocols = imap pop3 lmtp sieve
service auth {
unix_listener auth-userdb {
group = vmail
user = vmail
}
}
service lmtp {
inet_listener lmtp {
address = 10.102.42.114
port = 24
}
process_min_avail = 2
user = vmail
}
ssl_cert =
Re: mdbox attachment errors
> Il 27 febbraio 2015 alle 18.18 Hardy Flor: > copy file "f6f4f3b882bf3488af632389d4aaba8adc332b12" from backup to > "/var/mail/attachments/f6/f4/hashes/f6f4f3b882bf3488af632389d4aaba8adc332b12" > > and create hardlink to > "/var/mail/attachments/f6/f4/f6f4f3b882bf3488af632389d4aaba8adc332b12-60ab750a1aa4b554da169db5accb" Sure. That fix restores access to the mailbox/message/attachment. What if the file disappeared before the backup process copied it? What if 1% of 20'000 users open a ticket a day for the same reason? But ... why did it disappear at all? I got the same error on a test environment, where I am learning Dovecot (see my message sent 26/2/2015 at 14:48 CET). Nobody else has access to my server. I have not manually deleted files in the SIS partition. With a little script I ran through my mdbox m.* files, I extracted attachment hash-filenames and searched for them on the disk: 10 were missing. In some cases there is no hashes directory at all. Not even the 2nd level base path ( /attachments/a/b/hash stops at /attachments/a ). WHY??? Note that depending on how the IMAP client accesses the mailbox folder, you may be unable to see the whole folder at all! That's the case with Open-Xchange. Again: which tests can be run in order to confirm the behaviour and possibly a bug? Paolo Cravero
Re: users of dbox format
Andreas, > I am interested in finding out your experiences with using the dbox > format (especially mdbox) if you use this format. mdbox is THE reason why I am trying Dovecot. With mailboxes of several (tens of) GB with several k of messages I hope mdbox will speedup backups. Also SIS for attachments sounds very good, but still doesn't follow the altstorage rules (while messages go to altstorage, the extracted attachment stays in the SIS partition). > I am contemplating changing my maildir setup to mdbox but I still need > to make a case for it My current interest is to keep live mailbox data in Maildir format and "archive" in the altstorage in mdbox after 30 days or so. I will experiment that configuration next week. Nevertheless I am still puzzled by the lost data I got yesterday in my mdbox+SIS+altstorage combination (see my message and the one from Daniel Miller). I am also interested to hear real life stories. 73 Paolo Cravero
Re: Dovecot & LDAP Take #2: Authentication failed and logging
This is the user DN: > cn=Klara Fall,ou=People,dc=[domainname],dc=de According to your Dovecot configuration > auth_bind_userdn = cn=%u,ou=People,dc=**[domainname]**,dc=de if you login with "klarafall" it will be expanded into cn=klarafall,ou=People,dc=[domainname],dc=de which is not the correct DN for Mrs Klara. So if you login with "Klara Fall" it should work, but that will probably mess up the things on Dovecot filesystem. I am strongly against setting a static DN when dealing with LDAP authentication. LDAP servers are optimized to serve search requests, so let yours do the job. Allow Dovecot to lookup the correct DN based on the attribute you supply (uid) and then authenticate. This should be achieved if you comment out the auth_bind_userdn line. Paolo Cravero
mdbox + sis + altstorage: error in attachments-connector (file not found)
Hello, first time here.
I am playing with Dovecot because of its mdbox storage, SIS feature and
altstorage ability. I authenticate against LDAP (passdb), but I don't fetch
operational data from LDAP (mail_locations are static from config files and
"user_attrs = =uid=500, =gid=500, =home=/maildata1/home/%d/%1n/%n").
On a daily basis I run "doveadm altmove -A savedbefore 5d"
Thunderbird apparently works 100%, but only apparently. When accessing one
of those mailboxes through Open-Xchange IMAP I got an error for the first
time. Then I could reproduce it with Thunderbird as well, I simply had to
open the message mentioned in the log:
Feb 26 14:06:02 tst-msg03 dovecot: imap(paolo.crav...@csi.it): Error:
read(attachments-connector(/maildata1/altstorage/csi.it/p/paolo.cravero/storage/m.35))
failed:
read(/maildata1/attachments/40/68/40683f07fcbf1f95f09b273e472b3c7067f43210641baf6e3cce333a27b36e11-4a235a29fd05db54fd681117ea21[base64:19
b/l]) failed:
open(/maildata1/attachments/40/68/40683f07fcbf1f95f09b273e472b3c7067f43210641baf6e3cce333a27b36e11-4a235a29fd05db54fd681117ea21)
failed: No such file or directory (uid=5940, box=INBOX)
Feb 26 14:06:02 tst-msg03 dovecot: imap(paolo.crav...@csi.it): Error:
read(attachments-connector(/maildata1/altstorage/csi.it/p/paolo.cravero/storage/m.35))
failed:
read(/maildata1/attachments/40/68/40683f07fcbf1f95f09b273e472b3c7067f43210641baf6e3cce333a27b36e11-4a235a29fd05db54fd681117ea21[base64:19
b/l]) failed:
open(/maildata1/attachments/40/68/40683f07fcbf1f95f09b273e472b3c7067f43210641baf6e3cce333a27b36e11-4a235a29fd05db54fd681117ea21)
failed: No such file or directory (uid=5940, box=INBOX)
Feb 26 14:06:02 tst-msg03 dovecot: imap(paolo.crav...@csi.it): Disconnected:
Internal error occurred. Refer to server log for more information.
[2015-02-26 14:06:02] in=182 out=1266
This happens on many messages sitting in the altstorage that had an
attachment. The detached attachment file does not exist. Sometimes not even
the path is there (anymore?).
At some point in time I have run "doveadm purge" on the mailbox in error,
but I am not sure that is the culprit.
Have I hit a bug or done something wrong? Please suggest tests to verify
where I stand.
Config dump follows.
Paolo
# 2.2.15: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.el6.x86_64 x86_64 Red Hat Enterprise Linux Server
release 6.5 (Santiago) ext3
auth_mechanisms = plain login
disable_plaintext_auth = no
imap_client_workarounds = tb-extra-mailbox-sep
lmtp_save_to_detail_mailbox = yes
mail_attachment_dir = /maildata1/attachments
mail_attachment_hash = %{sha256}
mail_debug = yes
mail_gid = 500
mail_location =
mdbox:/maildata1/main/%d/%1n/%n:ALT=/maildata1/altstorage/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n
mail_plugins = quota mail_log notify lazy_expunge
mail_uid = 500
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave duplicate
mbox_write_locks = fcntl
mdbox_rotate_size = 50 M
namespace {
hidden = no
list = yes
location =
mdbox:/maildata1/main/%d/%1n/%n:INDEX=/maildata1/indexes/%d/%1n/%n:MAILBOXDIR=expunged
prefix = .EXPUNGED/
subscriptions = no
}
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
lazy_expunge = .EXPUNGED/
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
mail_log_fields = uid box msgid size flags vsize
quota = dict:User quota::file:%h/dovecot-quota
quota_grace = 10%%
quota_rule = *:storage=10G
quota_rule2 = Trash:storage=+100M
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
}
protocols = imap pop3 lmtp sieve
service auth {
unix_listener auth-userdb {
group = vmail
user = vmail
}
}
service lmtp {
inet_listener lmtp {
address = 10.102.42.114
port = 24
}
process_min_avail = 2
user = vmail
}
ssl_cert =
