Re: [Dovecot] LDAP Lookup not returning value in maxStorage
On Wed, 28 Mar 2012 09:39:37 +1300 Bruce, Andrew wrote: On 28 March 2012 09:36, Bruce, Andrew abr...@tumnus.co.nz wrote: On 27 March 2012 19:14, Nikita Koshikov koshi...@gmail.com wrote: On Tue, 27 Mar 2012 13:57:04 +1300 Bruce, Andrew wrote: Hi there, We're setting up a Dovecot virtual email setup - we've got everything working perfect with LDAP logins authenticating against AD and so forth, but we're having issues with retrieving the maxStorage value from AD (this is a pre-setup field in AD that we'd like to use to set per user quotas). In our LDAP lookup, we have the maxStorage entry listed under user_attrs for the quota (user_attrs = maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see it trying to get the entry, but it fails with: Mar 27 13:19:27 auth: Debug: ldap(username@site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=((objectClass=person)(| (userPrincipalName=username@site) (|(mail=username@site)(samAccountName=username@site fields=maxStorage Mar 27 13:19:27 auth: Debug: ldap(username@site,192.168.1.5): no fields returned by the server At this point, we then see the default quota applied. Try to change your quota rule to be like: maxStorage=quota_rule=*:bytes=%$ ^ And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x If we change the name of the field from maxStorage to instanceType we see the value show up in the logs and passed through to the quota system and applied successfully: Mar 27 11:09:01 auth: Debug: ldap(username@site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=((objectClass=person)(| (userPrincipalName=username@site) (|(mail=username@site)(samAccountName=username@site fields=instanceType Mar 27 11:09:01 auth: Debug: ldap(username@site,192.168.1.5): result: instanceType(quota_rule=*:storage=%$M)=*:storage=4M Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 username@site quota_rule=*:storage=4M Which seems a bit weird. If we use ldapsearch and pass it the same search string and look for the field maxStorage, we clearly see the field and the value being returned. The result looks the same if we also lookup instanceType. We're using Dovecot 2.0.9. Does anyone have any idea as to why we can't use this field? Thanks, Andrew Tried your suggestion Nikita, no joy unfortunately. It still looks like the value never gets returned from the LDAP server to Dovecot. It definitely has something in the field (equivalent of 10GB, but in bytes as suggested) and I changed the user_attrs also, but still get the same no fields returned by the server error message. Modifying the user_attrs to lookup from a different field (instanceType) definitely works. What exact version are you using - perhaps it's a problem with our copy of 2.0.9. Thanks, Andrew Show your full dovecot-ldap.conf file, also what port do you using ? maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) And show exact result of ldapsearch tool, binding under user from dovecot-ldap.conf + debug for this user when it trying to login and 'doveadm -D quota get -u $user' for this one. Also ensure that your search query returns only 1 result. We are using dovecot 2.0.19 now, but all versions of dovecot 2.0 branch was there in the past. I'm updating server since version 2.0.1 - no problem found.
Re: [Dovecot] dbox vs. mdbox
Timo Sirainen tss at iki.fi writes: 4. Are there real-world benchmarks showing measurable differences between maildir, sdbox mdbox? Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. The environment will be virtually the same as this is running in a virtualized environment. I'm keeping the same storage and storage network, same host hardware and about the same VM specs. Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information. My ideia is to install a fresh server and replicate the production maildir on it to build a test source disk that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly
Re: [Dovecot] Namespace, prefix questions
On 3/27/2012 10:40 AM, Steve Campbell wrote:
We've got some users who are using Outlook Express version 6. The
client allows me to specify the root folder, but not a prefix or
namespace. I'm still struggling with some users on our new server that
have crazy imap folder layouts, so I've got a few questions.
When I specify the root folder, does that bypass any namespace/prefix
definitions on the imap server?
On some clients, like Thunderbird, I have the option of specifying
namespace OR prefix. How do these differ? I thought that the prefix
was the name of the namespace.
It appears that I have to delete and re-create the account on these OE
6 clients to make the list of folders show properly. Does that sound
right?
This all came about because one of these OE 6 users was not able to
use their imap folders (server errors). Turns out it was one of the
users that had their folders directly under ~. So I moved them to
~/mail, created a .subscriptions file from their .mailboxlist file and
tried everything in the world to get the folders to list properly.
Only after specifying the root folder as ~/mail after recreating the
account and restarting OE did it show properly and the folders
remained listed. My default config has this setup as the
mail_location parm, but blanks as the root folder don't seem to work
in this situation. I'm also wondering where I specify the list,
hidden and other parms that are usually set in namespace blocks.
dovecot -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final)
disable_plaintext_auth = no
listen = *
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
mbox_write_locks = fcntl
namespace {
hidden = yes
inbox = yes
list = yes
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix =
separator = /
type = private
}
namespace {
hidden = yes
list = no
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix = #mbox/
separator = /
type = private
}
namespace {
hidden = yes
list = no
location =
prefix = mail/
separator = /
type = private
}
namespace {
hidden = yes
list = yes
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix = ~/mail/
separator = /
type = private
}
namespace {
hidden = yes
list = no
location =
prefix = ~%u/mail/
separator = /
type = private
}
passdb {
driver = pam
}
protocols = pop3 imap
ssl_cert = /etc/pki/dovecot/certs/dovecot.pem
ssl_key = /etc/pki/dovecot/private/dovecot.pem
userdb {
driver = passwd
}
protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
}
Thanks
steve campbell
Thanks
steve campbell
After googling a bit, it seems that all 3 can come into play in the same
or different meanings. Seems that prefix and namespace mean the same
thing. Root folder can mean the same as above, but can also stand alone
as an individual pointer to a personal folder that differs from from
what the imap server uses.
It's still not clear to me, but at least I'm getting an idea of what may
or may not work. Still not sure why the null or blank prefixed namespace
doesn't take precedence when nothing is set in the client.
steve
[Dovecot] Sieve fileinto and year/month folders.
Hi there,
I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve.
One thing used by some of powerusers are archiving mail automatically
with autocreated folders based on year + month.
Is there any good way to make that with sieve...
One example
require fileinto;
if address :is [From, To] dovecot@dovecot.org {
fileinto INBOX.mls.%Y.%m.dovecot;
}
This will fill any mails into INBOX.mls.2012.03.dovecot uppon receiving...
I don't know if some sieve guru can tell me how to do that... ?
Kind regards,
Xavier
Re: [Dovecot] Sieve fileinto and year/month folders.
Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef:
Hi there,
I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve.
One thing used by some of powerusers are archiving mail automatically
with autocreated folders based on year + month.
Is there any good way to make that with sieve...
One example
require fileinto;
if address :is [From, To] dovecot@dovecot.org {
fileinto INBOX.mls.%Y.%m.dovecot;
}
This will fill any mails into INBOX.mls.2012.03.dovecot uppon
receiving...
I don't know if some sieve guru can tell me how to do that... ?
require [variables,date,fileinto,mailbox];
# Extract date info
if currentdate :matches year * { set year ${1}; }
if currentdate :matches month * { set month ${1}; }
# Archive Dovecot mailing list items by year and month.
# Create folder when it does not exist.
if header :is list-id dovecot.dovecot.org {
fileinto :create INBOX.mls.${year}.${month}.dovecot;
}
The above also uses a more reliable way to detect the Dovecot mailinglist.
Regards,
Stephan
Re: [Dovecot] dsync redesign
On 3/27/2012 3:57 PM, Jeff Gustafson wrote: We do have a FC system that another department is using. The company dropped quite a bit of cash on it for a specific purpose. Our department does not have access it to. People are somewhat afraid of iSCSI around here because they believe it will add too much latency to the overall IO performance. They're a big believer in locally attached disks. Less features, but very good performance. If you use a software iSCSI initiator with standard GbE ports, block IO latency can become a problem, but basically in only 3 scenarios: 1. Slow CPUs or not enough CPUs/cores. This is unlikely to be a problem in 2012, given the throughput of today's multi-core CPUs. Low CPU throughput hasn't generally been the cause of software iSCSI initiator latency problems since pre-2007/8 with most applications. I'm sure some science/sim apps that tax both CPU and IO may have still had issues. Those would be prime candidates for iSCSI HBAs. 2. An old OS kernel that doesn't thread IP stack, SCSI encapsulation, and/or hardware interrupt processing amongst all cores. Recent Linux kernels do this rather well, especially with MSI-X enabled, older ones not so well. I don't know about FreeBSD, Solaris, AIX, HP-UX, Windows, etc. 3. System under sufficiently high CPU load to slow IP stack and iSCSI encapsulation processing, and or interrupt handling. Again, with today's multi-core fast CPUs this probably isn't going to be an issue, especially given that POP/IMAP are IO latency bound, not CPU bound. Most people running Dovecot today are going to have plenty of idle CPU cycles to perform the additional iSCSI initiator and TCP stack processing without introducing undue block IO latency effects. As always, YMMV. The simply path is to acquire your iSCSI SAN array and use software initiators on client hosts. In the unlikely event you do run into block IO latency issues, you simply drop an iSCSI HBA into each host suffering the latency. They run ~$700-900 USD each for single port models, and they eliminate block IO latency completely, which is one reason they cost so much. They have an onboard RISC chip and memory doing the TCP and SCSI encapsulation processing. They also give you the ability to boot diskless servers from LUNs on the SAN array. This is very popular with blade server systems, and I've done this many times myself, albeit with fibre channel HBAs/SANs, not iSCSI. Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. We thought ZFS would provide us with a nice snapshot and backup system (with zfs send). We never got that far once we discovered that ZFS doesn't work very well in this context. Running rsync on it gave us terrible performance. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands. It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput
Re: [Dovecot] Sieve fileinto and year/month folders.
On 2012-03-28 8:50 AM, Xavier Beaudouin wrote:
Hi there,
I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve.
One thing used by some of powerusers are archiving mail automatically
with autocreated folders based on year + month.
Is there any good way to make that with sieve...
One example
require fileinto;
if address :is [From, To] dovecot@dovecot.org {
fileinto INBOX.mls.%Y.%m.dovecot;
}
This will fill any mails into INBOX.mls.2012.03.dovecot uppon
receiving...
I don't know if some sieve guru can tell me how to do that... ?
Kind regards,
Xavier
I don't remember where I found out how to do this, but the following is
what I use:
require [fileinto, imap4flags, date, variables];
if currentdate :matches month * { set month ${1}; }
if currentdate :matches year * { set year ${1}; }
fileinto :flags \\seen ${year}-${month};
Deliver/lmtp is set to allow creation of folders.
-Greg
signature.asc
Description: OpenPGP digital signature
Re: [Dovecot] Sieve fileinto and year/month folders.
On 2012-03-28 9:04 AM, Stephan Bosch wrote:
Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef:
Hi there,
I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve.
One thing used by some of powerusers are archiving mail automatically
with autocreated folders based on year + month.
Is there any good way to make that with sieve...
One example
require fileinto;
if address :is [From, To] dovecot@dovecot.org {
fileinto INBOX.mls.%Y.%m.dovecot;
}
This will fill any mails into INBOX.mls.2012.03.dovecot uppon
receiving...
I don't know if some sieve guru can tell me how to do that... ?
require [variables,date,fileinto,mailbox];
# Extract date info
if currentdate :matches year * { set year ${1}; }
if currentdate :matches month * { set month ${1}; }
# Archive Dovecot mailing list items by year and month.
# Create folder when it does not exist.
if header :is list-id dovecot.dovecot.org {
fileinto :create INBOX.mls.${year}.${month}.dovecot;
}
The above also uses a more reliable way to detect the Dovecot
mailinglist.
Regards,
Stephan
Stephan,
Is the mailbox extension the one that lets fileinto use :create?
I've had a hard time trying to find a useful sieve reference that I can
understand.
Thank you,
-Greg
signature.asc
Description: OpenPGP digital signature
Re: [Dovecot] Sieve fileinto and year/month folders.
Op 3/28/2012 6:26 PM, Gregory Finch schreef:
On 2012-03-28 9:04 AM, Stephan Bosch wrote:
require [variables,date,fileinto,mailbox];
# Extract date info
if currentdate :matches year * { set year ${1}; }
if currentdate :matches month * { set month ${1}; }
# Archive Dovecot mailing list items by year and month.
# Create folder when it does not exist.
if header :is list-id dovecot.dovecot.org {
fileinto :create INBOX.mls.${year}.${month}.dovecot;
}
The above also uses a more reliable way to detect the Dovecot
mailinglist.
Stephan,
Is the mailbox extension the one that lets fileinto use :create?
I've had a hard time trying to find a useful sieve reference that I can
understand.
Thank you,
Yes:
http://tools.ietf.org/html/rfc5490#section-3.2
You can find links to specifications of the various Sieve extensions
implemented for Pigeonhole here:
http://pigeonhole.dovecot.org
Regards,
Stephan.
Re: [Dovecot] dsync redesign
On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. I'm a new employee at the company. I was a bit surprised they were not using iSCSI. They claim they just can't risk the extra latency. I believe that you are right. It seems to me that offloading snapshots and backups to an iSCSI SAN would improve things. The problem is that this company has been burned on storage solutions more than once and they are a little skeptical that a product can scale to what they need. There are some SAN vendor names that are a four letter word here. So far, their newest FC SAN is performing well. I think having more, small, iSCSI boxes would be a good solution. One problem I've seen with smaller iSCSI products is that feature sets like snapshotting are not the best implementation. It works, but doing any sort of automation can be painful. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: How does this work? I've always had Linux create a snapshot. Would the SAN doing a snapshot without any OS buy-in cause the filesystem to be saved in an inconsistent state? I know that ext4 is pretty good at logging, but still, wouldn't this be a problem? http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. I will take a look. I might have some convincing to do. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. There is one more that I had never used before coming on board here: ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My understanding is that they were using it on the backup machines with the front end dovecot machines using ext4. I'm told the metadata issue is a ZFS thing and they have the same problem on Solaris/Nexenta. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. If they'd tested it properly, and relatively recently, I would think they'd have already replaced EXT4 on your Dovecot server. Unless others factors prevented such a migration. Or unless I've misunderstood the size of your maildir workload. I don't know the entire history of things. I think they really wanted to use ZFS for everything and then fell back to ext4 because it performed well enough in the cluster. Performance becomes an issue with backups using rsync. Rsync is faster than Dovecot's native dsync by a very large margin. I know that dsync is doing more than rsync, but still, seconds compared to over five minutes? That is a significant difference. The problem is that rsync can't get a perfect backup. ...Jeff
Re: [Dovecot] Need fast Maildir to mdbox conversion
On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir-mdbox convertion yesterday, using the attached script. I only converted a little over 1 easy accounts (accounts with simple folder names, as I expect to run into problems once we start hitting accounts with trailing dot or broken latin1/utf8 characters in the folder names). I might agree it wasn't quick, but that really doesn't matter as the only downtime for the user is that he's potentially kicked out during the userdb update. I looked over your script. I plan on doing some trial runs with it. I think the trick where you re-run the sync and then boot the user off the connection should work pretty well. I hadn't totally fleshed out the scripting on the conversion since there is a lot more I need to do with the database and configuration files first. It appears I can use your script as a starting point for our configuration. ...Jeff -jf We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process.
Re: [Dovecot] Need fast Maildir to mdbox conversion
On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox transcoding tool, since none appear to exist. To keep it simple, I/O to/from remote mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's worth of RAM (if one of the directions involves a remote connection), and so far when processing MIX, Maildir, and Mbox files, it's extremely fast. This sounds interesting. If it could so [sm]dbox, it would be very, very useful to large installations. ...Jeff
Re: [Dovecot] dsync redesign
On 27.3.2012, at 1.14, Michescu Andrei wrote: This being said and acknowledged here are my 2 cents: I think that the current '1 brain / 2 workers' seems to be the correct model. The the client connects to the server and pushes the local changes and after retrieves the updated/new items from the server. The brain considers first server as the local storage and the second server as server storage. This design makes it too easy to design it in a way that adds extra roundtrips = extra latency. It also kind of hides other problems as well. For example now dsync can way too easily just fail if something unexpected happens during dsync (e.g. mailbox gets renamed/deleted). And there are of course some bugs that I don't really understand why some people are seeing them at all. For the split design, come to the same conclusion of the state is very race-condition prone. It's race-condition prone with the brain design as well. dsync can't just lock the mailbox during its sync, since the sync can take a long time. With a brainless design it's clear from the beginning that there are race conditions and they need to be dealt with.
Re: [Dovecot] dsync redesign
On 23.3.2012, at 23.25, Timo Sirainen wrote: and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: http://dovecot.org/tmp/dsync-redesign-problem.txt It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. Step #3 was more difficult than I first realized. I spent last two days figuring out a way to make it work, and looks like I finally did. I didn't update the document yet, but I wrote a test program: http://dovecot.org/tmp/test-dsync.c Step #2 should be easy enough. Step #4 I think I'll forget about and just implement a per-mailbox dsync lock. The main reason I wanted to get rid of locks was because a per-user lock can't work with shared mailboxes. But a per-mailbox lock is okay enough. Note that #3 allows the two dsyncs to run in parallel and send duplicate changes, just not modifying the same mailbox at the same time (which would duplicate mails due to two transactions adding the same mails).
Re: [Dovecot] Merge mails from two mail_locations
On 27.3.2012, at 13.11, Patrick Westenberg wrote: recently I had some trouble with my ocfs2 cluster and it unmounted itself from /var/mail. Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. You can use doveadm import to copy mails from one location to another.
Re: [Dovecot] fts-solr not indexing body content
On 27.3.2012, at 10.12, Bård Johannessen wrote: 2012/3/26 Timo Sirainen t...@iki.fi: Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This should fix it: http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 Nope; exactly same result; body field contains just the empty line. Always? It worked in my tests, and I don't see anything else wrong in the code..
Re: [Dovecot] dsync is SLOW compared to rsync
On 22.3.2012, at 23.25, Jeff Gustafson wrote: [root@n24 bu]# time dsync backup -u testu...@domain.com \ mdbox:/home/bu/testuser real1m9.519s user1m7.592s sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox
Re: [Dovecot] Namespace, prefix questions
On 27.3.2012, at 17.40, Steve Campbell wrote:
We've got some users who are using Outlook Express version 6. The client
allows me to specify the root folder, but not a prefix or namespace. I'm
still struggling with some users on our new server that have crazy imap
folder layouts, so I've got a few questions.
When I specify the root folder, does that bypass any namespace/prefix
definitions on the imap server?
Like you noticed, these are pretty much the same things, since most clients
don't understand about namespaces. And Dovecot doesn't select a namespace for
clients. They mostly affect mailbox listing.. Like when a client asks Dovecot
to list mailboxes under foo/, then Dovecot checks if a foo/ namespace exists.
On some clients, like Thunderbird, I have the option of specifying namespace
OR prefix. How do these differ? I thought that the prefix was the name of
the namespace.
I have no idea how Thunderbird handles them differently.
It appears that I have to delete and re-create the account on these OE 6
clients to make the list of folders show properly. Does that sound right?
Not really.
This all came about because one of these OE 6 users was not able to use their
imap folders (server errors). Turns out it was one of the users that had
their folders directly under ~. So I moved them to ~/mail, created a
.subscriptions file from their .mailboxlist file and tried everything in the
world to get the folders to list properly. Only after specifying the root
folder as ~/mail after recreating the account and restarting OE did it show
properly and the folders remained listed. My default config has this setup as
the mail_location parm, but blanks as the root folder don't seem to work in
this situation. I'm also wondering where I specify the list, hidden and
other parms that are usually set in namespace blocks.
..
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
mbox_write_locks = fcntl
namespace {
hidden = yes
inbox = yes
list = yes
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix =
separator = /
type = private
}
All of your namespaces have hidden=yes. There should be (at least) one with
hidden=no (this prefix= namespace in your case). I guess I'll need to add a
check to have Dovecot fail if there aren't any.
Also it's not necessary to duplicate the location setting, since it defaults
to the global mail_location.
namespace {
hidden = yes
list = no
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix = #mbox/
separator = /
type = private
}
This namespace is unlikely to be useful to you, unless you actually have some
IMAP client configured to use #mbox/ prefix.
namespace {
hidden = yes
list = yes
location = mbox:~/mail:INBOX=/var/spool/mail/%u
prefix = ~/mail/
separator = /
type = private
}
This shouldn't have list=yes .. I'm not sure what even happens with it. I guess
if client lists all mailboxes from root you'll get a mailbox named ~, which
has a mail child, which has all of your mailboxes duplicated.
Re: [Dovecot] dbox vs. mdbox
On 28.3.2012, at 13.13, Hangas wrote: Timo Sirainen tss at iki.fi writes: 4. Are there real-world benchmarks showing measurable differences between maildir, sdbox mdbox? Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. .. My ideia is to install a fresh server and replicate the production maildir on it to build a test source disk that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly The main problem is that it's difficult to do any real world tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much.
Re: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage
On 28.3.2012, at 11.57, Tomislav Mihalicek wrote: Could someone explain what this strings mean in dovecot 2.1.3 debug log? Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 te...@example.net service=lib-storage Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 te...@example.net service=lib-storage Dovecot is asking a user's home directory via userdb lookup. Looks like your userdb isn't returning a home directory. There should be an error message about it?
Re: [Dovecot] LDAP Lookup not returning value in maxStorage
On 28.3.2012, at 0.06, Bruce, Andrew wrote: Mar 27 13:19:27 auth: Debug: ldap(username@site,192.168.1.5): no fields returned by the server .. Further investigation shows that there are a few other fields that we can't retrieve in Dovecot, Looks to me like you can't retrieve any fields from LDAP, possibly because the dn user doesn't have access to the information or some other reason.
Re: [Dovecot] zlib_save per namespace/mailbox?
On 27.3.2012, at 19.49, Ben Schumacher wrote:
On Thu, Sep 22, 2011 at 8:44 AM, Lutz Preßler lutz.press...@sernet.de wrote:
the zlib_save question reminds me of a wish:
I think it's not possible to set zlib_save parameter per namespace (or even
mailbox). Per namespace would be something for the wish list to get rid of
the cron job method to compress archival mailboxes.
And maybe an option to add a Z flag to compressed maildir message files
as recommended in the wiki regarding compress crob job.
+1 on this request. I have a slightly different use case -- I have
both an dbox and Maildir. Incoming email goes to Maildir, but I
archive off to dbox (using Thunderbird). After I archive my emails,
compression seems like a reasonable choice.
Any idea if this feature will be available at some point?
For v2.2 I'm hoping to have per-namespace mail settings, although I'm not sure
if it actually gets implemented - so many other things to do as well. Then
(among other things) you could do:
namespace foo {
plugin {
zlib_save = gz
}
}
Re: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more
On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. doveconf -n output? 2.0.x: with Prefix ~/Mail A0003 LIST ~/Mail/sent A0003 OK List completed.: Works with my v2.0.19 config: x list ~/Mail/sent * LIST (\NoInferiors \UnMarked) / ~/Mail/sent x OK List completed.
Re: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook
On 28.3.2012, at 1.24, stonegate wrote: Problem: When i receive a new email, it does not appear in my Outlook unless i have the IMAP Inbox Folder open (highlighted selection). Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize it unless i click on the inbox folder. Before that problem occured on my old system ( i think it was dovecot as well ) the inbox folder refreshed automatically and i instantly knew when i had new emails. It was kind of like with my exchange account. Does anyone have a clue what might be wrong ? Since i did not re-install outlook or something it cannot be a client thing since no settings changed on the client side. Maybe i have to do something with dovecots config file? I think it should keep the connection to the server open or something. I´d appreciate any kind of help. I'm not aware of any Dovecot change or any Dovecot setting that could change this behavior in Outlook. It's the client's choice how it monitors the mailboxes.
Re: [Dovecot] doveadm purge on clusterfs
On 27.3.2012, at 12.47, Jan-Frode Myklebust wrote: Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? Yeah, should work fine. Or you should make it run in background so user's login won't slow down because of the purging. I'm not sure if simply adding at the end works (or if the process dies after login is finished, or what happens if it runs over 30 secs which is when normally post-login script is killed by SIGALRM).
Re: [Dovecot] Many messages clustered around the same date.saved value
On 27.3.2012, at 4.16, Joseph Tam wrote: However, I noticed a strange thing: querying what would have been deleted doveadm -ftab fetch -A date.saved mailbox Trash savedbefore 7d showed many date.saved values are clustered around the same timestamp, even among different user's Trash mailbox. ... I can't explain why many different users would have messages with the same (or closeby) date.saved value. Which mailbox format? With Maildir the date.saved is taken from dovecot.index.cache file, and in some cases that might get dropped. If it does, then it fallbacks to using the file's ctime. mbox. Ah, with mbox there isn't any usable fallback for date.saved. If it's not in dovecot.index.cache, the current time is used. These wrong values shouldn't cause problems with expunge queries since they err on the side of safety. Right.
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: On 22.3.2012, at 23.25, Jeff Gustafson wrote: [root@n24 bu]# time dsync backup -u testu...@domain.com \ mdbox:/home/bu/testuser real1m9.519s user1m7.592s sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. ...Jeff
Re: [Dovecot] Dovecot 1.2.9 Crash, NFS
On 26.3.2012, at 18.25, Müller Lukas wrote: Thanks for the quick answer. I realised, that the error didn't occur since quite a while, opposed to what our client suggested. Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. Is it possible that those could have improved the situation? Unlikely. Mar 6 08:26:31 mail02 dovecot: IMAP(u...@example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(u...@example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(u...@example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(u...@example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 - 1598) .. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). I had a quick look a tour NFS (NetApp), but didn't find anything useful. In case the problem persists, I will check with the coworker responsible for NetApp. I guess you just had very bad luck. NetApp has a good NFS implementation and normally works fine with Dovecot.
Re: [Dovecot] SIS and restoring from backups
On 25.3.2012, at 18.12, Charles Marcus wrote: On 2012-03-24 9:16 AM, Timo Sirainen t...@iki.fi wrote: On 24.3.2012, at 14.54, Charles Marcus wrote: On 2012-03-24 8:08 AM, Timo Sirainent...@iki.fi wrote: You can do full backups from a filesystem snapshot, which works well enough (might leave some unused attachments lying around in some rare cases, but that can also happen if Dovecot crashes/dies). But the problem isn't with backups, but with restores, right? Ah, right. Then it gets tricky. Yeah, I seem to remember it was a comment like that that scared me about enabling it... Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) It's easy to restore a full backup. And it's easy to restore specific users if you have the full backup easily accessible (just run doveadm import with proper settings pointing to backup). What's difficult is if you just want to restore a specific user from the backup and can't easily do random access to all files. Then you'll first need to restore the user's dbox files and then somehow figure out which attachments to restore from the SIS directory. Am I correct that enabling SIS as it is currently implemented would break this backup tool? I'm not sure. Are you running rsnapshot on live filesystem or on a snapshot? On live filesystem there would be race conditions. I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: file has vanished: /var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2, rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] I'd guess that with rsnapshot + Maildir you can get duplicate Maildir files if the rsnapshot is accessing a large maildir at the same time as user is changing a message flag. Dovecot usually notices these duplicates and logs a warning about them.
Re: [Dovecot] dsync is SLOW compared to rsync
On 29.3.2012, at 2.51, Jeff Gustafson wrote: Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. Try also with INDEX=MEMORY, since the problem may be related to updating the indexes. Another way to test if the problem is dsync or Dovecot's generic mail saving code is to run: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all Or if it's the mail reading code: time doveadm fetch -u user@domain text all /dev/null
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u...@domain.com/Maildir all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test Or if it's the mail reading code: time doveadm fetch -u user@domain text all /dev/null This ran quicker than a full dsync. Only 40s for 3.1GB. rsync still beat it clocking in at 16s. I ran the fetch command twice figuring the files would get cached by the OS. ...Jeff
Re: [Dovecot] dsync is SLOW compared to rsync
On 29.3.2012, at 3.48, Jeff Gustafson wrote: On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u...@domain.com/Maildir all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user@domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields.
Re: [Dovecot] dsync is SLOW compared to rsync
On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: On 29.3.2012, at 3.48, Jeff Gustafson wrote: On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/u...@domain.com/Maildir all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user@domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u u...@domain.com maildir:/home/users/user%domain.com/Maildir/ all doveadm(u...@domain.com): Error: Can't find namespace for mailbox Trash doveadm(u...@domain.com): Error: Can't find namespace for mailbox test ...Jeff
Re: [Dovecot] Many messages clustered around the same date.saved value
Timo Sirainen t...@iki.fi wrote: Which mailbox format? With Maildir the date.saved is taken from dovecot.index.cache file, and in some cases that might get dropped. If it does, then it fallbacks to using the file's ctime. mbox. Ah, with mbox there isn't any usable fallback for date.saved. If it's not in dovecot.index.cache, the current time is used. I'm a little confused as to why it needed a fallback. In other words, why wasn't date.saved put into the index as soon as the IMAP operation copied it into Trash? If this data isn't set at that time, when does it get instantiated? When I actually ask for it? Joseph Tam jtam.h...@gmail.com
Re: [Dovecot] dsync is SLOW compared to rsync
On 29.3.2012, at 5.07, Jeff Gustafson wrote: That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u u...@domain.com maildir:/home/users/user%domain.com/Maildir/ all doveadm(u...@domain.com): Error: Can't find namespace for mailbox Trash doveadm(u...@domain.com): Error: Can't find namespace for mailbox test Oh, you don't have prefix= namespace? If you have e.g. prefix=INBOX. namespace then use: time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user@domain maildir:/home/users/user%domain.com/Maildir/ INBOX all
Re: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more
On Thu, 29 Mar 2012, Timo Sirainen wrote:
On 27.3.2012, at 20.28, Gerhard Wiesinger wrote:
After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in
Webmail applications like roundcube mail and squirrelmail.
doveconf -n output?
2.0.x: with Prefix ~/Mail
A0003 LIST ~/Mail/sent
A0003 OK List completed.:
Works with my v2.0.19 config:
x list ~/Mail/sent
* LIST (\NoInferiors \UnMarked) / ~/Mail/sent
x OK List completed.
# 2.0.19: /etc/dovecot/dovecot.conf
# OS: cutted for security reasons
listen = *
mail_full_filesystem_access = yes
mail_location = mbox:~:INBOX=/var/mail/%u
mbox_lazy_writes = no
mbox_write_locks = fcntl
passdb {
driver = pam
}
protocols = imap
service auth {
unix_listener /var/run/dovecot-auth-master {
group = users
mode = 0660
}
user = root
}
service imap {
executable = imap postlogin
}
service postlogin {
executable = script-login -d rawlog
}
ssl_cert = /etc/pki/dovecot/dovecot-extern.pem
ssl_key = /etc/pki/dovecot/private/dovecot-extern.pem
userdb {
driver = passwd
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
}
protocol lda {
auth_socket_path = /var/run/dovecot-auth-master
info_log_path = /path/to/log/deliver.log
log_path = /path/to/log/deliver.log
log_timestamp = %Y.%m.%d %H:%M:%S%t
mail_debug = yes
mail_location = mbox:~/Mail:INBOX=/var/mail/%u
postmaster_address = m...@mydomain.com
}
Thnx.
Ciao,
Gerhard
--
http://www.wiesinger.com/
