Re: [Dovecot] Attachements with SIS : shauld all the same files share the same inode ?
Hi timo, thanks for your answer
Le 05/11/2011 17:23, Timo Sirainen a écrit :
On Mon, 2011-10-17 at 09:19 +0200, Yann Dupont wrote:
104 instance of -rw--- 104 vmail vmail 346584 16 oct. 17:47
6e2df299fa96daec9b4735d07c494046429be4d6-ffd0d1
The files are hardlinked, so it seems ok, but I also see 9 others files
in the same directory :
-rw--- 1 vmail vmail 346584 16 oct. 17:47
6e2df299fa96daec9b4735d07c494046429be4d6-0519df2a94fc9a4e1f418a6b99b6
-rw--- 1 vmail vmail 346584 16 oct. 17:47
6e2df299fa96daec9b4735d07c494046429be4d6-1790b92994fc9a4e1a418a6b99b6
as you can see, same prefix, If I compare them with diff, they are all
the same files.
I guess you have mail_attachment_fs = sis posix (default)? Sounds like
yes that's the case.
there could be a bug where SIS does comparison of files but for some
reason thinks they are different.
Well I can't try anymore on this, because this particular attachment is
not there. In fact this server keeps only 2 weeks of mails for 5000+
accounts (a form of backup for the main servers).
I'll take a new one :
-
rw--- 103 vmail vmail 571037 7 nov. 13:40
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c0c2230ecdd1b74ec9168a6b99b6
-rw--- 103 vmail vmail 571037 7 nov. 13:40
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c1c1290ecdd1b74ec5168a6b99b6
-rw--- 1 vmail vmail 571037 7 nov. 13:40
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6
So
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6 has
only one link,
but
diff
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c1c1290ecdd1b74ec5168a6b99b6 f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6
tell nothing : they are same files.
You could try if you can merge them with:
mkdir /tmp/queue
touch /tmp/queue/6e2df299fa96daec9b4735d07c494046429be4d6-foo
doveadm sis deduplicate /var/attachments /tmp/queue
hum I didn't get what you want to do ? Does putting a 0 byte file there
force a complete rescan ?
Tried it :
the 0 byte length
/tmp/queue/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-foo disappear from
the /tmp/queue, but
f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6 still
has one link.
A strace reveal this
stat(/vmail/attachments-deduplic, {st_mode=S_IFDIR|0700, st_size=8192,
...}) = 0
open(/tmp/queue, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8
getdents(8, /* 3 entries */, 32768) = 112
stat(/tmp/queue/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-foo,
{st_mode=S_IFREG|0644, st_size=0, ...}) = 0
link(/vmail/attachments-deduplic/f7/c9/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-foo,
/vmail/attachments-deduplic/f7/c9/hashes/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667)
= -1 ENOENT (No such file or directory)
mkdir(/vmail/attachments-deduplic/f7/c9/hashes, 0700) = -1 EEXIST
(File exists)
unlink(/tmp/queue/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-foo) = 0
getdents(8, /* 0 entries */, 32768) = 0
close(8)= 0
Tried with a copy of the file but this doesn't work either :
stat(/vmail/attachments-deduplic, {st_mode=S_IFDIR|0700, st_size=8192,
...}) = 0
open(/tmp/queue, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8
getdents(8, /* 3 entries */, 32768) = 144
stat(/tmp/queue/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6,
{st_mode=S_IFREG|0600, st_size=571037, ...}) = 0
write(2, doveadm(root): Fatal: /tmp/queue..., 170doveadm(root): Fatal:
/tmp/queue/f7c9746ed59fd838cbbfca08cde3c33b6b7ab667-c67f0e29bad1b74e7e158a6b99b6
is not a valid sis-queue file, is the queue directory correct?
How can I help further ?
Thanks,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : yann.dup...@univ-nantes.fr
[Dovecot] Failing to share folders when listescape is enabled (2.0.14)
Hello,
How I can share a folder with dots when the listescape plugin is enabled?
In this example user001 is sharing two foders named docs-abc and docs-a.b.c
to user002.
The first folder (without dots) is seen by user002 but the second is not found
by dovecot because it search a system folder named .docs-a.b.c but the system
folder is .docs-a\2eb\2ec
Nov 8 11:03:52 imap2 dovecot: imap(user002): Debug: acl vfile: file
/var/virtual-maildir/user001/.docs-a.b.c/dovecot-acl not found
drwx-- 2 vmail vmail 4096 2011-11-08 10:54 cur
drwx-- 5 vmail vmail 4096 2011-11-08 10:57 .docs-a\2eb\2ec
drwx-- 5 vmail vmail 4096 2011-11-08 10:57 .docs-abc
-rw--- 1 vmail vmail 20 2011-11-08 10:57 dovecot-acl-list
-rw--- 1 vmail vmail 248 2011-11-08 10:55 dovecot.index.log
-rw--- 1 vmail vmail 96 2011-11-08 10:54 dovecot.mailbox.log
-rw--- 1 vmail vmail 51 2011-11-08 10:55 dovecot-uidlist
-rw--- 1 vmail vmail 8 2011-11-08 10:57 dovecot-uidvalidity
-r--r--r-- 1 vmail vmail 0 2011-11-08 10:54 dovecot-uidvalidity.4eb8fc5b
drwx-- 5 vmail vmail 4096 2011-11-08 10:57 .Drafts
-rw--- 1 vmail vmail 15 2011-11-08 10:54 maildirsize
drwx-- 2 vmail vmail 4096 2011-11-08 10:54 new
drwx-- 5 vmail vmail 4096 2011-11-08 10:54 .Sent
drwx-- 5 vmail vmail 4096 2011-11-08 10:54 .Spam
-rw--- 1 vmail vmail 23 2011-11-08 10:54 subscriptions
drwx-- 2 vmail vmail 4096 2011-11-08 10:54 tmp
drwx-- 5 vmail vmail 4096 2011-11-08 10:54 .Trash
# cat /var/virtual-maildir/user001/subscriptions
Trash
Sent
Drafts
Spam
docs-abc
docs-a\2eb\2ec
# telnet localhost 143
Trying ::1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
AUTH=PLAIN] Dovecot ready.
. login user001 X
. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA ACL RIGHTS=texk] Logged in
. create docs-abc
. OK Create completed.
. create docs-a.b.c
. OK Create completed.
. setacl docs-abc user002 lrsw
. OK Setacl complete.
. setacl docs-a.b.c user002 lrsw
. OK Setacl complete.
. subscribe docs-abc
. OK Subscribe completed.
. subscribe docs-a.b.c
. OK Subscribe completed.
. logout
* BYE Logging out
. OK Logout completed.
Connection closed by foreign host.
# telnet localhost 143
Trying ::1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
AUTH=PLAIN] Dovecot ready.
. login user002 XX
. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA ACL RIGHTS=texk] Logged in
. subscribe shared/user001/docs-abc
. OK Subscribe completed.
. subscribe shared/user001/docs-a.b.c
. NO Mailbox doesn't exist: shared/user001/docs-a.b.c
. logout
* BYE Logging out
. OK Logout completed.
Connection closed by foreign host.
/-/
# dovecot -n
# 2.0.14: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.34.7-0.7-xen x86_64 openSUSE 11.3 (x86_64)
auth_debug = yes
auth_verbose = yes
base_dir = /var/run/dovecot/
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_proxy_port = 24244
doveadm_socket_path = localhost:24244
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_debug = yes
mail_fsync = always
mail_gid = 5000
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = acl quota notify expire listescape
mail_uid = 5000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date
mmap_disable = yes
namespace {
inbox = yes
list = yes
location =
prefix =
separator = /
subscriptions = yes
type = private
}
namespace {
list = children
location = maildir:/var/virtual-maildir/%%n:INDEX=~/shared.%%n
prefix = shared/%%n/
separator = /
subscriptions = no
type = shared
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
acl = vfile
acl_shared_dict = file:/var/maildir/vol00/shared-mailboxes
autocreate = Trash
autocreate2 = Sent
autocreate3 = Drafts
autocreate4 = Spam
autosubscribe = Trash
autosubscribe2 = Sent
autosubscribe3 = Drafts
autosubscribe4 = Spam
expire = Trash
expire2 = Trash/*
expire5 = Spam
expire6 = Spam/*
expire_dict = proxy::expire
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
flag_change append
mail_log_fields = uid box msgid
Re: [Dovecot] Failing to share folders when listescape is enabled (2.0.14)
On 8.11.2011, at 12.20, mailing lists wrote: How I can share a folder with dots when the listescape plugin is enabled? Sorry, doesn't work in v2.0 and I don't think it's possible to fix it without major changes. Those major changes are done in v2.1 code tree though, and it works there.
[Dovecot] Performance-Tuning
Hi, I have 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Thanks for any hints an tricks, Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
What is the setup on the EVA, FC or iSCSI? Sent from my BlackBerry® wireless device -Original Message- From: Peer Heinlein p.heinl...@heinlein-support.de Sender: dovecot-boun...@dovecot.org Date: Tue, 8 Nov 2011 14:50:25 To: dovecot@dovecot.org Subject: [Dovecot] Performance-Tuning Hi, I have 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Thanks for any hints an tricks, Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
On 08.11.2011 14:50, Peer Heinlein wrote: *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Hi Peer, We have switched our mailbox storage format from maildir to mdbox! Maildir is a disaster. (too many small files) After the migration to mdbox the performance has improved significantly. Conclusion: mdbox is great and much better performance than maildir! I would also recommend ext4. Best regards, Morten
Re: [Dovecot] Performance-Tuning
We are very happy with mdbox+zlib+ext4 + iSCSI SAN (HP Lefthand in our setup) If you have CPU to spare, consider using zlib with mdbox. You are trading CPU power (cheap) to get fewer IOPS (IOPS count is expensive). Mdbox has halved our backup windows (2,8 TB uncompressed mailboxes, 2 TB compressed) and backup software is happier because there are few (100.000+ files with mdbox) to backup instead of several millions (Maildir) Regards Javier Hi, I have 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? Thanks for any hints an tricks, Peer
Re: [Dovecot] Performance-Tuning
* Morten Stevens mstev...@imt-systems.com: We have switched our mailbox storage format from maildir to mdbox! I wonder how I can incrementally change over from Maildir to mdbox? I can of course use dsync to mirror Maildir: to mdbox:, but how can I make dovecot look at Maildir FIRST and (if that fails) at mdbox? (or vice versa). That would allow for a smooth transition... -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
[Dovecot] Multiple Patitions with with mdbox
Having 10 TByte mailstore filesystem-checks takes too much time. At the moment we have four different partitions, but I don't like to set symlinks or LDAP-flags to sort customers and their domains to there individual mount-point. I'd like to work with mdbox:/mail/%d/%n to calculate the path automatically. How do you handle 10 TB mailstore? I'm very interested in the feature alternative mailstore with mdbox, because that makes it very easy to use at least TWO filesystems without any tricky configuration. I think I'd love to have n alternative mailstores. Why does dbox doesn't look for its m.*-files in more then two directorys? Sure, looking in 4 directorys would lead to four disc operations, but maybe it could be very helpful. Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
Am Dienstag, 8. November 2011, 15:15:39 schrieb Javier de Miguel Rodríguez: Hi, If you have CPU to spare, consider using zlib with mdbox. You are trading CPU power (cheap) to get fewer IOPS (IOPS count is expensive). Hey. This point is great. I hadn't realized that. Sure. zlib will save IOPS and 2x6-CPUs aren't a problem. Good point -thanks. compressed) and backup software is happier because there are few (100.000+ files with mdbox) to backup instead of several millions (Maildir) Yes, that#s the main reason why I want to switch to mbox. At the moment our roundtrip-time for the backup is 24h... Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
Other important thing to consider is message expunging. With mdbox you are delaying the I/O associated with deleting e-mails. We have a nightly cronjob that expunge messages from mdboxes. If you have en EVA (wich one? 4.400? 6.400? ) you also can consider RAID 1+0 or SSD for indexes. Indexes are hammered in mdbox. Regards Javier Am Dienstag, 8. November 2011, 15:15:39 schrieb Javier de Miguel Rodríguez: Hi, If you have CPU to spare, consider using zlib with mdbox. You are trading CPU power (cheap) to get fewer IOPS (IOPS count is expensive). Hey. This point is great. I hadn't realized that. Sure. zlib will save IOPS and 2x6-CPUs aren't a problem. Good point -thanks. compressed) and backup software is happier because there are few (100.000+ files with mdbox) to backup instead of several millions (Maildir) Yes, that#s the main reason why I want to switch to mbox. At the moment our roundtrip-time for the backup is 24h... Peer
Re: [Dovecot] Performance-Tuning
Am Dienstag, 8. November 2011, 15:16:12 schrieb Ralf Hildebrandt: Hi, I wonder how I can incrementally change over from Maildir to mdbox? If you have double diskspace: Just use dsync mirror in the background to prepare the change. After that it's just a short downtime to migrate incremental the last changes, or it's just a question of a short login-script. if [ -d ~/Maildir ] ; then dsync mirror voodoo-magic rm -R ~/Maildr fi I can of course use dsync to mirror Maildir: to mdbox:, but how can I make dovecot look at Maildir FIRST and (if that fails) at mdbox? (or vice versa). I wonder about that problem too. Even the last-last-last-quick sync would be so much IO, that I can't handle it in realtime in the morning at 9 a.m. Looks like a nightly downtime for the last incremental run. It would be MUCH easier if Dovecot could read maildir: or mdbox: from LDAP attributes. In this case the whole migration process could be split up into groups. Unfortunately we have shared folders and I don't know a way to read the *remote* mailbox-format from LDAP... So having users with maildir and mdbox mixed up will break their shared folders... Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
[Dovecot] dsync and zlib
Will dsync also use zlib-compression when copying mails from one side to another? Wouldn't this be a good way to compress existing mails? http://wiki2.dovecot.org/Plugins/Zlib says, there's no way to do that. But dsync should respect the zlib-Plugin... Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
On 8.11.2011, at 16.16, Ralf Hildebrandt wrote: * Morten Stevens mstev...@imt-systems.com: We have switched our mailbox storage format from maildir to mdbox! I wonder how I can incrementally change over from Maildir to mdbox? I can of course use dsync to mirror Maildir: to mdbox:, but how can I make dovecot look at Maildir FIRST and (if that fails) at mdbox? (or vice versa). That would allow for a smooth transition... If you don't have shared folders (as explained in previous mail) and you can have per-user mail_location in the userdb, this should be pretty easy. The man page for dsync lists the steps that can be used for online migration.
Re: [Dovecot] Performance-Tuning
On 8.11.2011, at 16.03, Morten Stevens wrote: We have switched our mailbox storage format from maildir to mdbox! Maildir is a disaster. (too many small files) After the migration to mdbox the performance has improved significantly. Conclusion: mdbox is great and much better performance than maildir! I would also recommend ext4. You don't happen to have any specific numbers/graphs that can be used to compare maildir vs. mdbox in the same hardware? I'd be interested in seeing those, such as a graph of disk iops spanning a month before/after mdbox switch.
Re: [Dovecot] Performance-Tuning
Hi there, I never tried it, but it should be possible to provide the mail_location from the user repsoitory (LDAP, SQL, whatever) Actually this works :-) Our userdb looks similar to: account1:xyz:000:000::/account1s/home/dir::userdb_mail=maildir:/account1s/home/dir/Maildir account2:xyz:000:000::/account2s/home/dir::userdb_mail=mdbox:/ account2s/home/dir http://wiki2.dovecot.org/UserDatabase/ExtraFields Concerning Maildir backups: what about a backup-to-disc-to-tape scheme using snapshots for the to-disc part and something like perpetual incrementals afterwards for the top-tape (secondary store) ? Regards Dirk -- Dirk Jahnke-Zumbusch Deutsches Elektronen-Synchrotron DESY IT Information Fabrics Member of the Helmholtz Association D-22603 HamburgNotkestrasse 85 / 22607 Hamburg T: +49-40-899.81760 F: +49-40-899.41760 dirk.jahnke-zumbu...@desy.de So you can keep your global config, and use a script to convert one mailbox after another, and add a mail_location extra userdb field in the user repository to overwrite the global setting on a per-user-basis. Regards, Oliver
Re: [Dovecot] Performance-Tuning
On 8.11.2011, at 15.50, Peer Heinlein wrote: At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). A few more ideas for Maildir if you haven't done yet: - maildir_very_dirty_syncs = yes - pop3_no_flag_updates = yes - Switching to dict file quota instead of Maildir++ quota.
Re: [Dovecot] dsync and zlib
On 8.11.2011, at 16.36, Peer Heinlein wrote: Will dsync also use zlib-compression when copying mails from one side to another? As long as the zlib plugin is globally enabled and zlib_save setting is set. Wouldn't this be a good way to compress existing mails? http://wiki2.dovecot.org/Plugins/Zlib says, there's no way to do that. But dsync should respect the zlib-Plugin... Well .. You could think of compressing existing mails with dsync the same as migrating from maildir to migrating to maildir with zlib enabled :) I'll add a note to the wiki page.
Re: [Dovecot] Multiple Patitions with with mdbox
On 8.11.2011, at 16.19, Peer Heinlein wrote: Having 10 TByte mailstore filesystem-checks takes too much time. At the moment we have four different partitions, but I don't like to set symlinks or LDAP-flags to sort customers and their domains to there individual mount-point. I'd like to work with mdbox:/mail/%d/%n to calculate the path automatically. Why not symlinks? You could use e.g. /mail/%16Hu/%d/%n as part of the path which would give you one hex letter when hashing the username (0..9, a..f). So then you could create symlinks like: /mail/0 - /storage1 /mail/1 - /storage2 ..etc.. I'm very interested in the feature alternative mailstore with mdbox, because that makes it very easy to use at least TWO filesystems without any tricky configuration. I think I'd love to have n alternative mailstores. Why does dbox doesn't look for its m.*-files in more then two directorys? Sure, looking in 4 directorys would lead to four disc operations, but maybe it could be very helpful. The current implementation uses a simple message flag to tell when/where to move the mail, so the code would have to get much more complicated with more alt paths.
[Dovecot] another sizeof tweak
Line 776 of dovecot-2.0.15/src/lib-storage/index/maildir/maildir-sync-index.c
reads:
memcmp(old_rec, new_rec, sizeof(old_rec)) != 0) {
Should that be sizeof(*old_rec)?
Re: [Dovecot] another sizeof tweak
On 8.11.2011, at 18.43, Mike Abbott wrote:
Line 776 of dovecot-2.0.15/src/lib-storage/index/maildir/maildir-sync-index.c
reads:
memcmp(old_rec, new_rec, sizeof(old_rec)) != 0) {
Should that be sizeof(*old_rec)?
Yep, looks like was fixed in v2.1 already. I'll fix it for v2.0 too.
Re: [Dovecot] Performance-Tuning
Quoting Peer Heinlein p.heinl...@heinlein-support.de: It would be MUCH easier if Dovecot could read maildir: or mdbox: from LDAP attributes. In this case the whole migration process could be split up into groups. Unfortunately we have shared folders and I don't know a way to read the *remote* mailbox-format from LDAP... So having users with maildir and mdbox mixed up will break their shared folders... May not work for you, but... The way I did this when I migrated was to run two dovecot instances, and have perdition software on a front-end (could be on the same machine instead of a front-end, I just happen to have a front-end machine to do it). Perdition will query ldap for the info per user/connection, and send the connection to the correct dovecot instance based on the ldap lookup. Worked for me, your milage may vary... -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns!
[Dovecot] Dovecot alias and mailbox problem
Hi All, My Postfix / Dovecot / MySQL is running fine but I have the following issue: When I create the following mailboxes t...@example.com and te...@example.com it is working fine. I can create an alias te...@examples.com = te...@example.com that is also working without a problem, only when I create t...@example.com = te...@example.com email is only delivered to test1@ instead to both test@ and test1@ I believe it is properly just a setting but please help me out here ! Let me know if you need any more information before I post configs etc. Best regards, Jeroen
Re: [Dovecot] Dovecot alias and mailbox problem
On 08-11-11 19:34, Jeroen Grusewski wrote: Hi All, My Postfix / Dovecot / MySQL is running fine but I have the following issue: When I create the following mailboxes t...@example.com and te...@example.com it is working fine. I can create an alias te...@examples.com = te...@example.com that is also working without a problem, only when I create t...@example.com = te...@example.com email is only delivered to test1@ instead to both test@ and test1@ I believe it is properly just a setting but please help me out here ! Let me know if you need any more information before I post configs etc. You probably want to create the alias as t...@example.com = te...@example.com,t...@example.com. Depending on how you setup postfix, this should deliver to both mailboxes (and not create an infinite loop). Note that aliasing is handled within Postfix, not in Dovecot, so if you have further questions, you're probably on the wrong list. -- Regards, Tom
Re: [Dovecot] Performance-Tuning
Am Dienstag, 8. November 2011, 17:53:32 schrieb Eric Rostetter: May not work for you, but... The way I did this when I migrated was to run two dovecot instances, and have perdition software on a front-end (could be on the same machine instead of a front-end, I just happen to have a front-end machine to do it). You could do that with Dovecot, too. Perdition will query ldap for the info per user/connection, and send the connection to the correct dovecot instance based on the ldap lookup. Worked for me, your milage may vary... The problem is: You're running in problems with shared folders. You can't read your neighbors storage-engine from ldap. It's easy to read the user's storage engine from ldap. So there's no need to use perdition for that :-) But you can't read or proxy the storage engine from somebody who shared you his folders. That's my problem :-( Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Re: [Dovecot] Performance-Tuning
Quoting Peer Heinlein p.heinl...@heinlein-support.de: The problem is: You're running in problems with shared folders. You can't read your neighbors storage-engine from ldap. Yes, but I didn't have any shared folders, so it worked. Your milage may vary, as I said... :) -- Eric Rostetter The Department of Physics The University of Texas at Austin Go Longhorns!
Re: [Dovecot] another sizeof tweak
On Tue, 2011-11-08 at 18:46 +0200, Timo Sirainen wrote:
On 8.11.2011, at 18.43, Mike Abbott wrote:
Line 776 of
dovecot-2.0.15/src/lib-storage/index/maildir/maildir-sync-index.c reads:
memcmp(old_rec, new_rec, sizeof(old_rec)) != 0) {
Should that be sizeof(*old_rec)?
Yep, looks like was fixed in v2.1 already. I'll fix it for v2.0 too.
Actually it doesn't matter in v2.0 since this is mailbox list indexing
code, which is forcibly disabled (and broken) in v2.0.
[Dovecot] v2.1.beta1 released
http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz.sig Here's the first beta release of Dovecot v2.1. This version has already been tested quite a lot, so I'm not expecting any major bugs. So please upgrade and see if you can find any problems. I'm optimistic about getting rc1 released this year and perhaps even v2.1.0. Since alpha2 there have been a lot of fixes, especially to imapc backend. There are probably also some small new features, but nothing huge. As a reminder, here's the largest changes since v2.0: * Plugins now use UTF-8 mailbox names rather than mUTF-7: acl, autocreate, expire, trash, virtual * auth_username_format default changed to %Lu. If you really want case sensitive usernames, set it back to empty. * Solr full text search backend changed to use mailbox GUIDs instead of mailbox names, requiring reindexing everything. solr_old backend can be used with old indexes to avoid reindexing, but it doesn't support some newer features. + imapc (= IMAP client) storage allows using a remote IMAP server to be used as storage. This allows using Dovecot as a smart (caching) proxy or using dsync to do migration from remote IMAP server. + Mailbox indexing via queuing indexer service (required for Lucene) + Lucene full text search (FTS) backend rewritten with support for different languages + FTS finally supports OR search operation + FTS supports indexing attachments via external programs + IMAP FUZZY extension, supported by Lucene and Solr FTS backends + Mailbox list indexes + Statistics tracking via stats service. Exported via doveadm stats. + Autocreate plugin creates/subscribes mailboxes physically only when the mailbox is opened for the first time. Mailbox listing shows the autocreated mailboxes even if they don't physically exist. + Password and user databases now support default_fields and override_fields settings to specify template defaults/overrides. - listescape plugin works perfectly now
Re: [Dovecot] v2.1.beta1 released
On 11/8/2011 11:35 PM, Timo Sirainen wrote: http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz http://dovecot.org/releases/2.1/beta/dovecot-2.1.beta1.tar.gz.sig Here's the first beta release of Dovecot v2.1. This version has already been tested quite a lot, so I'm not expecting any major bugs. So please upgrade and see if you can find any problems. I'm optimistic about getting rc1 released this year and perhaps even v2.1.0. Ok, I guess it is time for a Pigeonhole release for Dovecot v2.1. I'm a bit swamped this week, but next weekend I should be able to compose one. For now you can use the dovecot-2.0-pigeonhole hg and patch it with http://hg.rename-it.nl/pigeonhole-0.2-dovecot-2.1-patches/file/tip/pigeonhole-0.2-dovecot-2.1.patch to make it compile against v2.1. Regards, Stephan.
Re: [Dovecot] Performance-Tuning
On 11/8/2011 7:50 AM, Peer Heinlein wrote: I have 11 TB hard used Mailstorage, saved als maildir in ext3 on HP EVA. That's a lot of mail (likely a large user base--not given), on a filesystem not designed for such, on a decent SAN controller--LUN RAID configuration not given. I always wanted to make some mesurements about several influences to the performance (switch to ext4, switch to mdbox), but I never had enough time to do that. If you're going to switch filesystems, for this size dataset and concurrent workload, you're moving in the wrong direction. At the moment I *need* more speed, we have too much waitI/O on the system and I already used all other performance and tuning-tricks (separated cache, noatime, fsync and all that stuff). EXT3/4 are not designed, nor optimized, for high concurrency workloads. I have to change my setup, maybe somebody else here have hard facts: *) Is ext4 faster? How much faster? Simulated maildir workload test on 2.6.35-rc5, 128 threads (No data published for newer kernels): http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5_Mail_server_simulation._num_threads=128.html As you can see EXT4 shows a small gain over EXT3, ~20%. If you really want high performance it's time to move to XFS, properly configured to match the underlying RAID characteristics of the LUN(s) you're mounting. You'll prefer kernel 2.6.39+, 2.6.36 at minimum, so you get the delayed logging feature (2.6.35 had delayed logging but had problems in other areas). I'll assume with a 10TB mail store that you're seeing greater than 128 concurrent user operations regularly. As you can see from the graph, XFS will give you ~50% greater ops/s than EXT4 and ~90% greater than EXT3--yes, almost double that of EXT3. As the concurrency increases, so will this performance gap, as XFS was designed from day 1 for high concurrency workloads. This is a simulated mail server benchmark. However you should see similar gains with Dovecot. The XFS delayed logging feature will dramatically reduce the number of physical IOs required for journal writes (i.e. metadata IO), as will delayed allocation, a feature of XFS since its inception in 1994. EXT4 was the first of its lineage to gain delayed allocation, some 10+ years later, after Ted T'so studied the XFS code. In short, if you want an 'enterprise caliber' production Linux filesystem tailor made for high IO concurrency, XFS is it. JFS yields similar performance, but hasn't been actively developed for 8 years or so. XFS has substantial ongoing feature and fix development. *) Is it faster because of the ext4 kernel-module (which can be used on ext3 to) or because of the ext4 filesystem layout? AIUI, the bulk of the EXT4 performance advantage over EXT3 is the delayed allocation logic. The new EXT4 extent based on disk layout yields little in the way of additional performance, but much in free space management, fragmentation mitigation, etc. *) Is mdbox really faster? I'd like to have mdbox to have better performance in running my backup-processes. But does it bring some performance boosts to? mdbox will substantially decrease physical IOs to your storage back end due to dramatically less metadata operations compared to maildir. You've stated you currently have a storage IOPS bottleneck, so I'd have to assume mdbox will seriously increase your overall performance. Good old mbox will do so as well, but everyone shuns it for various reasons, some valid, some not so valid. If you have an appropriate LUN available (sufficient size and spindle speed/count of member disks), properly create an XFS filesystem on it (read much before creating it), and moved to mdbox atop that, I think you'll be really surprised by how much you gain from simply changing filesystems and mailbox storage formats. If you double the size of the LUN you could potentially carry twice as many users with, fewer IOPS than you're seeing now, on essentially the same hardware platform. -- Stan
