[Dovecot] dovecot lost mail! Cause?
Hi, After using dovecot for several years now, today happend something which makes me really feel unconfortable: An email received was just not delivered properly, or, is lost! The mail (from an external server) was sent to two local mailboxes, user1 and user2. user1 received the message but for user2, it *magically* disappeared. MTA is exim4 which definitely processed the messages and handed over to dovecot deliver: 2012-11-12 07:28:21 1TXnVG-00053I-GD SA: [...] id=8644593.887351352701 685934.JavaMail 2012-11-12 07:28:21 1TXnVG-00053I-GD = user1 us...@example.com R=dovecot T=dovecot_pipe 2012-11-12 07:28:21 1TXnVG-00053I-GD = user2 us...@example.com R=dovecot T=dovecot_pipe 2012-11-12 07:28:21 1TXnVG-00053I-GD Completed Also, the log of dovecot tells that the mail should have been stored: Nov 12 07:28:21 mail dovecot: deliver(user1): sieve: msgid=8644593.887351352701685934.JavaMail: stored mail into mailbox 'INBOX' Nov 12 07:28:21 mail dovecot: deliver(user2): sieve: msgid=8644593.887351352701685934.JavaMail.orbitz: stored mail into mailbox 'INBOX' user1 received the mail but user2 not. Since user2 is myself, I *know* that I did not accidently delete any mail or something like that. It was just never received! Disk space is 3GB left, so enough. So I grepped the whole Maildir of user2 for the message ID. There is only one match in the dovecot.index.cache and within that, the most important data of the mail (Message ID, Date, Sender, Receiver, Subject) appears. But apart from that, not a single file! Is there hope to ever find out why what was going wrong here? It feels me very unconfortable because from now on I can never be sure any more that all my mails are really received :( :( However, as I said, my mail system processed maybe millions of messages the past 8 years. Although, I can not be sure if that was the case ... :( And help greatly appreciated! Luke PS: Dovecot version 1.2.15 (Debian 6.0.6) with Maildir backend on local harddrive. No NFS, nothing which can go wrong ... PPS: Original log files, just named replaced for privacy.
Re: [Dovecot] dovecot lost mail! Cause?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 12 Nov 2012, Lukas Haase wrote: Nov 12 07:28:21 mail dovecot: deliver(user2): sieve: msgid=8644593.887351352701685934.JavaMail.orbitz: stored mail into mailbox 'INBOX' are there any other log lines of user2? - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBUKC/FmoxLS8a3A9mAQJLEAgAkghKGBYWFj94OMCo5mM26XV4c0nHKgob ec0ELqgOIGZf+DA7+Dztwq/MWgdkhB/ZbUSQ2rd4qQ7nf7gEO10L0WXUXWzMJ9sm upvn8JaOJDZ37Ne7AeoOib/m5fXyQUa0oiW7y8ShdeveTAOtn+Bu0OT2BuibOeT8 /EsTA+DfVlymTgHrMYU0LIjjvHh94Duj4at1k1X1So2kTaNbw48ljKYMd0qb2+pR 39D/ZtynOqnEKzj5f+JU+WmCcCAEAW9IL8U8pySvuZaXkPN+cfcLO82J9UIEmIAf IKymFt7JuNabGCsJ0FpMeuLAyXNOxJdKmGxgqmpyfilPY2ty8hstmg== =Uz9E -END PGP SIGNATURE-
[Dovecot] Help me with IMAP config
Hello! I have dovecot running. Imap works fine. When i connect with outlook an when i delete a message it gets crossed out but still stays in my mailbox. I want to disable this feature. I want it once its deleted then go to trash folder on the mail server and thats it. Is there an option for this? I'm running dovecot 1.2.15 Thanks!
Re: [Dovecot] dovecot lost mail! Cause?
Hi, On 11/12/2012 1:19 AM, Steffen Kaiser wrote: On Mon, 12 Nov 2012, Lukas Haase wrote: Nov 12 07:28:21 mail dovecot: deliver(user2): sieve: msgid=8644593.887351352701685934.JavaMail.orbitz: stored mail into mailbox 'INBOX' are there any other log lines of user2? Anfortunately not :-( Grepped everything, and I would have posted otherwise. My hope is that the dovecot.index.cache may provide some insights (I made a backup copy of it). Luke
Re: [Dovecot] Help me with IMAP config
On 11/12/2012 4:40 AM, Tibby wrote: Hello! I have dovecot running. Imap works fine. When i connect with outlook an when i delete a message it gets crossed out but still stays in my mailbox. I want to disable this feature. I want it once its deleted then go to trash folder on the mail server and thats it. Is there an option for this? I'm running dovecot 1.2.15 Thanks! This is the way IMAP works. Deleted items are marked delete and then removed when the client issues the 'expunge' command. It's best to change this in the email client. Look at your Outlook configuration for a way to 'move deleted items to trash' or 'expunge'. Bill
[Dovecot] Invalid Managesieve commands are counted twice
Hi,
the Managesieve server closes the connection if it receives an unknown
command before authentication:
IMPLEMENTATION Dovecot Pigeonhole
SIEVE fileinto reject envelope encoded-character vacation subaddress
comparator-i;ascii-numeric relational regex imap4flags copy include
variables body enotify environment mailbox date ihave
NOTIFY mailto
SASL PLAIN
STARTTLS
VERSION 1.0
OK Dovecot ready.
-- BOGUS
NO Error in MANAGESIEVE command received by server.
NO Error in MANAGESIEVE command received by server.
BYE Too many invalid MANAGESIEVE commands.
Connection closed by foreign host.
Note that only one bogus command has been sent by the client, however
the server sends two identical error messages.
This seems to be a bug in Pigeonhole 0.3.3. In version 0.2.6, the
connection was kept open after the error message.
This is actually important to us because we use the sieveshell utility
which is shipped with the Python managesieve package. The
managesieve.py module always sends a BOGUS command after the TLS
handshake. According to its comments, this is done to work around
problems with other server implementations:
# Some servers send capabilities after TLS handshake, some
# do not. We send a bogus command, and expect a NO. If you
# get something else instead, read the extra NO to clear
# the buffer.
typ, data = self._command('BOGUS')
(The full source is at http://pydoc.net/managesieve/0.4.2/managesieve)
As a result, sieveshell cannot be used with TLS and a current
Dovecot/Pigeonhole server.
Cheers,
Christoph
--
Christoph Bußenius
Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik
Technische Universität München
+49 89-289-18519 Raum 00.05.040 Boltzmannstr. 3 Garching
Re: [Dovecot] Invalid Managesieve commands are counted twice
Hi Stephan, On 12.11.2012 11:18, Stephan Bosch wrote: I fixed this a while back, but hasn't been released so far: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/ceef02768dee thanks, I am going to try out the current hg version. I guess I should have tried this first... Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 Raum 00.05.040 Boltzmannstr. 3 Garching
Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
On 08/11/2012 23:53, Ben Morrow wrote: At 3AM -0600 on 8/11/12 you (Stan Hoeppner) wrote: 1.0.7 is absolutely ancient and no longer officially supported. You need 1.2.x minimum, 2.x.x even better. And you say you just recently upgraded your Linux distro? What planet do you live on son? You're a few light years behind current stable software. [A light-year is a measure of distance, not of time.] LDA completely eliminates lock contention. As we have discussed before, using the LDA does not prevent lock contention, it just prevents the problems that arise when different software is using different locking strategies on the same mailbox (assuming nothing except LDA and imap is touching the mailbox directly). There are valid reasons for not using the LDA: the OP might be already using procmail, for instance, and have users with procmail recipies which sort into IMAP folders. These folders will need to be locked by procmail even if the default delivery to INBOX is changed (globally) to happen through dovecot-lda. While migrating to sieve (and mdbox, and LMTP) would, IMHO, be the best long-term solution, this isn't necessarily something that can be set up overnight. Ben - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2221 / Virus Database: 2441/5382 - Release Date: 11/08/12 Thanks for your replies. I switched to Dovecot LDA this morning, but the issue still persists, albeit logged slightly differently by Dovecot now instead of Postfix: save failed to INBOX: Timeout while waiting for lock The reason is because some pop3 clients are holding their connection for 5 or 6 minutes (don't ask me why - and the iPhone seems to be the major culprit). In dovecot.conf I changed: mbox_lock_timeout = 300 to mbox_lock_timeout = 600 Which seems to have helped. I am unclear if this value only applied to Dovecot LDA or if it would have worked previously before switching to Dovecot LDA?
[Dovecot] Commercial features in Dovecot future: Object storage, archive
Hi all, Dovecot Oy’s web pages at www.dovecot.fi have been updated. The products page lists two features that will be available for commercial licensing, extending the functionality of the basic open-source version of Dovecot. * Storing emails to (high-latency) object storage, initially supporting Amazon S3, Caringo CAStor and Scality. * Email archive storage. See http://www.dovecot.fi/products/index.html for details. I’ve been developing Dovecot for over 10 years now. For a long time it was my primary motivation in life to create the best IMAP server available :) I think I've pretty much accomplished that by now. The future is looking very bright for Dovecot: we will continue the open source development stronger than ever, but in addition, for the long term it needs some additional licensed components that bring the money to cover the cost for future Dovecot development and to be able to build up the support in a professional way. These new features will be added as plugins on top of Dovecot to extend the functionality. Note that I’m not just randomly choosing which features will be open and which will be licensed. Only some specific features will be licensed where my company is going to make money with partnerships and in other measurable ways.
Re: [Dovecot] Commercial features in Dovecot future: Object storage, archive
Il 12/11/2012 12:33, Timo Sirainen ha scritto: Hi all, Dovecot Oy’s web pages at www.dovecot.fi have been updated. The products page lists two features that will be available for commercial licensing, extending the functionality of the basic open-source version of Dovecot. * Storing emails to (high-latency) object storage, initially supporting Amazon S3, Caringo CAStor and Scality. * Email archive storage. See http://www.dovecot.fi/products/index.html for details. I’ve been developing Dovecot for over 10 years now. For a long time it was my primary motivation in life to create the best IMAP server available :) I think I've pretty much accomplished that by now. The future is looking very bright for Dovecot: we will continue the open source development stronger than ever, but in addition, for the long term it needs some additional licensed components that bring the money to cover the cost for future Dovecot development and to be able to build up the support in a professional way. These new features will be added as plugins on top of Dovecot to extend the functionality. Note that I’m not just randomly choosing which features will be open and which will be licensed. Only some specific features will be licensed where my company is going to make money with partnerships and in other measurable ways. I'm really interesting in storing email into object storage, since our IaaS provider is using Scality we can simple buy dovecot's plugin for scality. I will contact Dovecot Oy for more informations. -- Alessio Cecchi is: @ ILS - http://www.linux.it/~alessice/ on LinkedIn - http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux - http://www.cecchi.biz/ @ PLUG - ex-Presidente, adesso senatore a vita, http://www.prato.linux.it
Re: [Dovecot] v2.1 memory usage
On 12/11/2012 04:13, Daniel L. Miller wrote: The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. Openssl is a *massive* project and I'm unsure that gnutls is much smaller... We should assume that both are quite scary from a security point of view. Licensing is the main thing which divides them, gnutls is stated as GPL compatible (however, the nominal incompatibility of openssl seems difficult to understand?) OpenVPN integrated with PolarSSL and got Dutch government official approval for the combined package. I think elsewhere it's stated that openssl would not have been approved because something like the codebase was too large to inspect and sign off http://polarssl.org/news?item=0132 I haven't worked with PolarSSL, so no idea, but it's massively smaller codebase is likely attractive if you are the kind of person who actually *does* security audits on the software you run in secure situations. Openssl is just a complete swiss army knife of tools! Ed W
Re: [Dovecot] v2.1 memory usage
On 12.11.2012, at 6.13, Daniel L. Miller wrote: where is the problem with openssl? I don't know what the problem is - I just know that I've heard from a number of developers (including the Postfix Dovecot developers) that they don't like OpenSSL - but while GnuTLS looks interesting they aren't interested in working on the interface - though they're willing to accept patches. (My full apologies right now if Timo or Wietse are offended by my speaking out of turn). OpenSSL documentation is very bad. Its API has some annoying missing features. For example you can load certificates from a directory or a file but not from anything else like from a string in memory. I had to copypaste a few functions from OpenSSL code just to be able to do them. The tiny bit of Googling I've done tells me GnuTLS seems to be a more standards-compliant implementation, and MAY be safer than OpenSSL. However, as OpenSSL is the de-facto standard used by most Linux programs, acceptance of GnuTLS is quite limited. I've been intrigued by what I've read about it, and took a quick look at enabling support in Dovecot for GnuTLS directly - but while it didn't seem overly heavy at first glance the fact that Timo doesn't want to do it tells me I'm underestimating the complexity. I already once wrote GnuTLS support for Dovecot, but GnuTLS changed its APIs since then and it was probably originally already buggy. I think the only somewhat special APIs that Dovecot needs nowadays are related to reading cert/keys from memory instead of from files. If GnuTLS can do that, I don't think there's anything special in supporting it. Although it might be a bit complex to make it work properly asynchronously. istream-openssl was a bit annoying in that way (all the data read from the fd must be parsed and decoded all the way through to the SSL istream, regardless of any max buffer limits).
Re: [Dovecot] mbox vs. maildir storage block waste
On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote: Have you made systematic tests? I.e. compared times for all of these with those from the different dovecot backends. The choice of Dovecot backends made no substantial difference. I used maildir, sdbox, and mdbox. I also added SiS (with mdbox). Initial tests were on local multi-spindle RAID5 storage, but to handicap Dovecot, I pushed it over NFS (also Linux 3.2 on a local GigE segment). It wasn't slow enough to make dbmail competitive, even though you have to start turning off performance optimisation features in Dovecot to avoid NFS bugs. There wasn't a task that the dbmail setup performed faster than Dovecot, in either low or high load situations. Which backend did you use? Backend for dbmail? Two MySQL versions (5.0 and 5.5) - InnoDB is required for dbmail, by the way. Postgres 8.4 and 9.1 backends, using its default storage engine. I tried the tests with both a separate DB machine, as well as a cohosted one with the dbmail connector using local sockets instead of TCP/IP, but that didn't significantly alter the performance. I've found my first notes from the tests. It was the second round of tests with the latest MySQL 5.0 server given some tuning to more aggressively use system memory. You will note the puny size of the mail folder hive in this round. The mysqld process has consumed nearly an hour of CPU time during this process. dbmail is configured to use local sockets rather than network I/O. I'm using the PERL MailTools http://search.cpan.org/dist/MailTools/ to import about 10 folders' worth of email, totaling about 560MB in raw size, constituting about 23,000 emails. The script basically creates the folders, and does an APPEND for each email. It's bog simple. I DROP the database, recreated it, added the one user, verify DBMail accepts authentication for the newly created mailbox, and then do the import. The MySQL files live on a freshly formatted ext4 filesystem. The import takes Dovecot (MailDir or mdbox format), or Panda IMAP (mix) about six minutes to complete. DBMail 3 took 4h 23m. Casual inspection of the system showed modestly high CPU usage in mysqld and dbmail-imapd (as well as the import perl command on occasion), but the Load Average didn't get too close to 1.0, let alone 2.0, which concerns me that I might have hit some kind of busy wait pathology. To clarify the above: To streamline iterative testing, I made a script to deactivate the currently running SQL server, unmount, re-format, re-mount, and re-populate the skeletal DB directories and restart the DB engine. So between each test, no matter the imapd or DB back-end, the mailstore was presented with a freshly formatted volume on dedicated spindles. The filesystem was ext4, formatted with: lazy_itable_init=0,lazy_journal_init=0,dir_index=1,extents=1,uninit_bg=0,flex_bg=0,has_journal=0,inode_size=256,dir_index=1, Do you have detailed numbers? Not really, but after it was clear that I wasn't going to get comparable performance even within the same magnitude, I stopped testing it. I included the IMAP SEARCH performance comparison against fts_squat in my original mail to this list. In addition to huge performance deficiencies, it also has/had fatal operational bugs. I guess you’ve only tried dbmail? I did try Manitou, but the lack of a proper IMAP service for it made extensive like for like testing very difficult. Manitou is still in the very early days, alas. It also relies on the SQL DB's underlying authentication systems which is rather ... alarming. It performs quite a bit better than dbmail, but still it's not close to Dovecot. At the time I tested it, only custom-rolled clients could talk to it, i.e., no imap4/pop3 gateways to it. I think I was most alarmed to see that the widely assumed benefits of putting mail on a SQL DB, i.e., fast searching/sorting, didn't actually happen in reality. As others have mentioned, I also shudder to think of backup/restore issues, especially on a single user level. The mechanisms of backing up and restoring maildirs and even mdboxes, i.e., simple files, are not only well understood, the failure modes are generally fully recoverable. SQL-DB file blobs, especially with MySQL, remind me too much of the PST Hell that Exchange administrators face. But maybe that's just my ignorance talking. All something I wouldn’t want to do on my production systems ;) Neither would I. But as I said, I was desperate to get this close to Dovecot's performance. I had about 2-3 weeks to pre-qualify mail storage back-ends with an eye towards 4 or 5 digits of usercount, and maybe tens to hundreds of TBs' scale of mail storage. Running across such poor performance with such relatively small loads disqualified the DB-based mail products very very quickly, for ME, anyway. If you want to run your own tests, my suggestion is to start with Postgres, put
Re: [Dovecot] mbox vs. maildir storage block waste
On 13.11.2012, at 0.44, Robin wrote: On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote: Have you made systematic tests? I.e. compared times for all of these with those from the different dovecot backends. The choice of Dovecot backends made no substantial difference. I used maildir, sdbox, and mdbox. I also added SiS (with mdbox). Initial tests were on local multi-spindle RAID5 storage, With local disks the tests often measure only the local RAM/CPU speed, unless you're testing thousands of users. but to handicap Dovecot, I pushed it over NFS (also Linux 3.2 on a local GigE segment). It wasn't slow enough to make dbmail competitive, even though you have to start turning off performance optimisation features in Dovecot to avoid NFS bugs. NFS makes a better test case if you're measuring single user performance. Much of it is probably due to the index file access latency, although not all. In some cases Dovecot's prefetching mails can help (maildir, sdbox backends with local disks currently, nothing preventing it from working in other use cases though, even with Dovecot-SQL backend). I guess you’ve only tried dbmail? I did try Manitou, but the lack of a proper IMAP service for it made extensive like for like testing very difficult. Manitou is still in the very early days, alas. It also relies on the SQL DB's underlying authentication systems which is rather ... alarming. It performs quite a bit better than dbmail, but still it's not close to Dovecot. At the time I tested it, only custom-rolled clients could talk to it, i.e., no imap4/pop3 gateways to it. Manitou seems to advertise itself as being email client .. although then also seems to say SQL is faster than IMAP (which doesn't make much sense itself). I think I was most alarmed to see that the widely assumed benefits of putting mail on a SQL DB, i.e., fast searching/sorting, didn't actually happen in reality. SQL has nothing that makes any type of email access even potentially efficient. SQL indexes are mostly about binary trees, and there are about zero things in IMAP where I have thought of binary tree being even potentially useful. (Okay, potentially for expunging old mails when you have 1M mails in one folder. Not something you normally optimize for.) With most of Dovecot's optimized lookups, latency is the most important thing. SQL is bad for latency. With remote systems it's usually much faster to just download 1 MB blob and parse it than fetch a couple of 100 byte blocks. As others have mentioned, I also shudder to think of backup/restore issues, especially on a single user level. The mechanisms of backing up and restoring maildirs and even mdboxes, i.e., simple files, are not only well understood, the failure modes are generally fully recoverable. SQL-DB file blobs, especially with MySQL, remind me too much of the PST Hell that Exchange administrators face. But maybe that's just my ignorance talking. I'd think everyone would use the human-readable SQL dumps for database backups. At least with MySQL/PostgreSQL I wouldn't really trust anything else.
Re: [Dovecot] mbox vs. maildir storage block waste
Uh.. On 13.11.2012, at 1.02, Timo Sirainen wrote: On 13.11.2012, at 0.44, Robin wrote: On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote: Have you made systematic tests? I.e. compared times for all of these with those from the different dovecot backends. The choice of Dovecot backends made no substantial difference. I used maildir, sdbox, and mdbox. I also added SiS (with mdbox). Initial tests were on local multi-spindle RAID5 storage, With local disks the tests often measure only the local RAM/CPU speed, unless you're testing thousands of users. ..measuring disk I/O most importantly. but to handicap Dovecot, I pushed it over NFS (also Linux 3.2 on a local GigE segment). It wasn't slow enough to make dbmail competitive, even though you have to start turning off performance optimisation features in Dovecot to avoid NFS bugs. NFS makes a better test case if you're measuring single user performance. Much of it is probably due to the index file access latency, although not all. In some cases Dovecot's prefetching mails can help (maildir, sdbox backends with local disks currently, nothing preventing it from working in other use cases though, even with Dovecot-SQL backend). Prefetching is done only with mail_prefetch_count setting. Someone in blog.dovecot.org mentioned that it was bad for performance with local disk+maildir. Linux apparently doesn't do this with NFS. It would of course be possible to just have the prefetching create a new thread/process to download the mail locally and read it (similar to what the object storage plugin does).
Re: [Dovecot] Dovecot newbie (migrating from qmail)
Il 13/11/2012 06:09, Ajai Khattri ha scritto:
Ive been using qmail+vpopmail+courier-imap for many years but its time to
retire that server so I thought this might be an opportunity to see how I
could use Postfix and Dovecot to provide the same services. Im running
Dovecot 2.1.9 and Postfix 2.9.4.
I have spent a few days reading through some of the docs on the wiki
(there's doesn't seem to be any large overview of Dovecot concepts or books
so bear with me).
Im thinking something simple like passwd-file setup would suffice for me. I
want to use completely virtual users.
Id like to store mail under /home/vmail/$domain/$user. I think Ive figured
out how to do that. But how do I create the maildir?
Maildir/ will automatically created when the first email arrives or
during the first user login. It would be better if you create the folder
during the creation of the users.
I understand the need to have a mail directory but also a directory for
things like sieve - how to specify that?
Im thinking the structure would be something like:
/home/vmail/$domain/$user/mail -- mail stored here
Yes, better (by convention) if named Maildir/
/home/vmail/$domain/$user/ -- sieve and other sundry store here
Or should the sieve stuff also be under its own folder alongside the
maildir?
Inside Maildir/ named sieve/ or always without tarting with a dot
Is it possible to have a separate passwd file per domain? It is possible to
combine password and userdb files into one per domain? (I prefer to keep
all files related to each domain in its own folder).
Yes,
passdb {
driver = passwd-file
# Each domain has a separate passwd-file:
args = /etc/auth/%d/passwd
}
Yes: http://wiki2.dovecot.org/AuthDatabase/PasswdFile
Also would like to configure Postfix to use dovecot-sasl and I want to use
dovecot-lmtp for deliveries. Any good docs / example on those?
For SASL http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL
Ciao
--
Alessio Cecchi is:
@ ILS - http://www.linux.it/~alessice/
on LinkedIn - http://www.linkedin.com/in/alessice
Assistenza Sistemi GNU/Linux - http://www.cecchi.biz/
@ PLUG - ex-Presidente, adesso senatore a vita, http://www.prato.linux.it
