Re: Errors installing on Ubuntu 17.04

[Jeff Kletsky]

On 11/21/17 5:20 AM, Someone wrote:


On Tue, 21/11/17, Jeff Kletsky <dove...@allycomm.com> wrote:

On 11/20/17 1:58 PM, Someone wrote:

  > I've been

struggling with this on and off for a while now.  Upon


attempting to install Dovecot on Ubuntu

17.04 I get the following errors



[...]


"Hint: Some lines were ellipsized, use -l

to show in full."

I'm aware of that, yes.  Unfortunately I don't know how to tell any of
the myriad of package installers to "do the right thing" in such cases,

and, as stated, this is happening during installation.



  Without seeing the full messages, my guess is that you've got problems with 
your IPv6 configuration or connectivity.

Entirely possible, since I don't have IPv6 and have completely disabled
it as a result.  Even were I to re-enable it, I don't see it would help
if problems with IPv6 connectivity would also be a problem (because I
don't have it).
Without the log information, you've made it nearly impossible to help 
you. The problem looks to be occurring in the execution of dovecot, not 
the install process.


While you may not have IPv6 connectivity, with the little information 
you've provided:


Oct 05 14:08:56 ash dovecot[8885]: Error: socket() failed: Address family n…ocol
Oct 05 14:08:56 ash dovecot[8885]: Error: service(imap-login): listen(::, 1…ocol
Oct 05 14:08:56 ash dovecot[8885]: master: Error: socket() failed: Address …ocol
Oct 05 14:08:56 ash dovecot[8885]: Fatal: Failed to start listeners

it is possible that dovecot is expecting to be able to connect over IPv6 and is 
unable to do so.
"socket() failed: Address family" and "listen(::, " are strong hints of that.

If you have successfully disabled IPv6 (which, I believe, Ubuntu enables by 
default) and dovecot is trying to listen on an IPv6 address, it is not 
surprising that errors occur.

The output of 'ip addr' and 'sysctl -a | fgrep net.ipv6' would be helpful, as 
would you examining the output of 'doveconf' to see if dovecot is trying to 
listen on IPv6 addresses.



Jeff

Re: LMTP "Relative home directory paths not supported"

[Jeff Kletsky]

On 11/20/17 6:33 AM, Aki Tuomi wrote:



On 20.11.2017 16:20, Jeff Kletsky wrote:

On 11/20/17 6:15 AM, Aki Tuomi wrote:

On 20.11.2017 16:12, Jeff Kletsky wrote:

I'm in the process of moving from LDA to LMTP (Postfix upstream) prior
to a transition off `nix accounts to virtual accounts and am stumped
by LMTP reporting

  dovecot: lmtp(10019, j...@example.com): Error: Relative home
directory paths not supported: 0

[...]

mail_location = maildir:~/Maildir:LAYOUT=fs
# mail_location = maildir:%h/Maildir:LAYOUT=fs



Thanks!

Jeff

That seems like lmtp is not doing variable expansion for user settings.
What version of dovecot are you running?

Aki


$ sudo jexec mail dovecot --version
2.2.32 (dfbe293d4)

Jeff

I tried to repeat this issue locally, and was not able to. Can you
provide full output of

doveconf -n

and

doveadm user j...@example.com

with both %h and ~/

Aki


Found that I had two sets of passdb and userdb entries due to one from 
local.conf and and another from conf.d/10-auth.conf including 
auth-system.conf.ext -- unfortunately that wasn't the source of the 
problems.


Turning on auth_debug=yes in local.conf and enabling LOG_DEBUG level 
output shows that, in contrast to the "working" configuration, it looks 
like my passwd-file configuration is returning "home=0" -- consistent with


"Error: Relative home directory paths not supported: 0"



    auth: Debug: passwd-file(jeff,n.n.n.n,): lookup: 
user=jeff file=/usr/local/etc/dovecot/users
    auth: Debug: client passdb out: OK    1    user=jeff User    
&:/home/jeff:/usr/sbin/nologin


    dovecot: auth: Debug: passwd-file(jeff,n.n.n.n,): 
lookup: user=jeff file=/usr/local/etc/dovecot/users
    dovecot: auth: Debug: master userdb out: USER    1234567890 jeff    
uid=x    gid=x    home=0 auth_token=verylongauthtoken



These were the clues needed -- While 
https://wiki2.dovecot.org/AuthDatabase/PasswdFile talked about using a 
FreeBSD master.passwd file as a passdb, it neglected to state that the 
Linux format and FreeBSD format are different.


Linux   -- user:password:uid:gid:(gecos):home:(shell):extra_fields
FreeBSD -- name:password:uid:gid:class:change:expire:gecos:home_dir:shell
--^

Login will succeed, but the home variable will be populated with the 
FreeBSD "change" which is often 0


Wiki page updated with details on how to use FreeBSD /etc/master.passwd 
extracts with passwd-file


Jeff

Re: Errors installing on Ubuntu 17.04

[Jeff Kletsky]

On 11/20/17 1:58 PM, Someone wrote:


I've been struggling with this on and off for a while now.  Upon
attempting to install Dovecot on Ubuntu 17.04 I get the following errors

Selecting previously unselected package dovecot-imapd.
(Reading database ... 243684 files and directories currently installed.)
Preparing to unpack .../dovecot-imapd_1%3a2.2.27-2ubuntu2_amd64.deb ...
Unpacking dovecot-imapd (1:2.2.27-2ubuntu2) ...
Processing triggers for dovecot-core (1:2.2.27-2ubuntu2) ...
Processing triggers for ufw (0.35-4) ...
Setting up dovecot-imapd (1:2.2.27-2ubuntu2) ...
Creating config file /etc/dovecot/conf.d/20-imap.conf with new version
Processing triggers for dovecot-core (1:2.2.27-2ubuntu2) ...
Job for dovecot.service failed because the control process exited with error 
code.
See "systemctl status dovecot.service" and "journalctl -xe" for details.
invoke-rc.d: initscript dovecot, action "restart" failed.
   dovecot.service - Dovecot IMAP/POP3 email server
 Loaded: loaded (/lib/systemd/system/dovecot.service; enabled; vendor 
preset: enabled)
 Active: failed (Result: exit-code) since Thu 2017-10-05 14:08:56 BST; 17ms 
ago
 Docs: man:dovecot(1)
   http://wiki2.dovecot.org/
   Process: 8880 ExecStop=/usr/bin/doveadm stop (code=exited, status=0/SUCCESS)
   Process: 8885 ExecStart=/usr/sbin/dovecot (code=exited, status=89)
Main PID: 8729 (code=exited, status=0/SUCCESS)

Oct 05 14:08:56 ash systemd[1]: Starting Dovecot IMAP/POP3 email server...
Oct 05 14:08:56 ash dovecot[8885]: Error: socket() failed: Address family n…ocol
Oct 05 14:08:56 ash dovecot[8885]: Error: service(imap-login): listen(::, 1…ocol
Oct 05 14:08:56 ash dovecot[8885]: master: Error: socket() failed: Address …ocol
Oct 05 14:08:56 ash dovecot[8885]: Fatal: Failed to start listeners
Oct 05 14:08:56 ash systemd[1]: dovecot.service: Control process exited, co…s=89
Oct 05 14:08:56 ash systemd[1]: Failed to start Dovecot IMAP/POP3 email server.
Oct 05 14:08:56 ash systemd[1]: dovecot.service: Unit entered failed state.
Oct 05 14:08:56 ash systemd[1]: dovecot.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
dpkg: error processing package dovecot-core (--configure):
subprocess installed post-installation script returned error exit status 1
(and more)

Apologies for any wrapping due to long lines in the above output.

"Hint: Some lines were ellipsized, use -l to show in full."

Without seeing the full messages, my guess is that you've got problems 
with your IPv6
configuration or connectivity.  It's a long shot, but might save you a 
round-trip

on the mailing list.

Jeff

Re: LMTP "Relative home directory paths not supported"

[Jeff Kletsky]

On 11/20/17 6:15 AM, Aki Tuomi wrote:


On 20.11.2017 16:12, Jeff Kletsky wrote:

I'm in the process of moving from LDA to LMTP (Postfix upstream) prior
to a transition off `nix accounts to virtual accounts and am stumped
by LMTP reporting

     dovecot: lmtp(10019, j...@example.com): Error: Relative home
directory paths not supported: 0

LDA does /not /have any problems with PAM or passwd-file passdb/userdb
and the ubiquitous definition of

     mail_location = maildir:~/Maildir:LAYOUT=fs

I have yet to be able to get LMTP to deliver mail, nor have I found
anything on the Dovecot wiki to describe how to represent that the
mail_location is /Maildir

I'd prefer to localize the user-to-file-space mapping localized to
Dovecot (rather than split within Postfix and Dovecot)

I'd definitely appreciate any pointers to web pages or specific
suggestions as to how to resolve this.

Excerpts of Dovecot local.conf follow, full doveconf -n output on
request. The/users /file is in the same format as FreeBSD
/etc/master.passwd with name, password, UID, gid, home_dir, and shell
meaningfully populated. The configuration below works with LDA
(end-to-end). With LMTP, it identifies <j...@example.com> as a valid
destination, but fails on delivery with the "relative home directory
paths" error message.

passdb {
   driver = passwd-file
   args = /path/to/users
}
userdb {
   driver = passwd-file
   args = username_format=%n /path/to/users
}

# See http://wiki2.dovecot.org/MailLocation/Maildir
# https://wiki.dovecot.org/VirtualUsers/Home

mail_location = maildir:~/Maildir:LAYOUT=fs
# mail_location = maildir:%h/Maildir:LAYOUT=fs



Thanks!

Jeff

That seems like lmtp is not doing variable expansion for user settings.
What version of dovecot are you running?

Aki


$ sudo jexec mail dovecot --version
2.2.32 (dfbe293d4)

Jeff

LMTP "Relative home directory paths not supported"

[Jeff Kletsky]

I'm in the process of moving from LDA to LMTP (Postfix upstream) prior 
to a transition off `nix accounts to virtual accounts and am stumped by 
LMTP reporting


    dovecot: lmtp(10019, j...@example.com): Error: Relative home 
directory paths not supported: 0


LDA does /not /have any problems with PAM or passwd-file passdb/userdb 
and the ubiquitous definition of


    mail_location = maildir:~/Maildir:LAYOUT=fs

I have yet to be able to get LMTP to deliver mail, nor have I found 
anything on the Dovecot wiki to describe how to represent that the 
mail_location is /Maildir


I'd prefer to localize the user-to-file-space mapping localized to 
Dovecot (rather than split within Postfix and Dovecot)


I'd definitely appreciate any pointers to web pages or specific 
suggestions as to how to resolve this.


Excerpts of Dovecot local.conf follow, full doveconf -n output on 
request. The/users /file is in the same format as FreeBSD 
/etc/master.passwd with name, password, UID, gid, home_dir, and shell 
meaningfully populated. The configuration below works with LDA 
(end-to-end). With LMTP, it identifies  as a valid 
destination, but fails on delivery with the "relative home directory 
paths" error message.


passdb {
  driver = passwd-file
  args = /path/to/users
}
userdb {
  driver = passwd-file
  args = username_format=%n /path/to/users
}

# See http://wiki2.dovecot.org/MailLocation/Maildir
# https://wiki.dovecot.org/VirtualUsers/Home

mail_location = maildir:~/Maildir:LAYOUT=fs
# mail_location = maildir:%h/Maildir:LAYOUT=fs



Thanks!

Jeff

Re: letsencrypt

[Jeff Kletsky]

You can either drop the authentication token into /.wellknown on your
running server, or take down the server for a minute to run certbot
every couple months.

I'm not a fan of symlinks out of config directories and certainly not
across chroot / jail boundaries so I manually copy the certs into the
a subdirectory of the dovecot config directory.

Here's the segment from my local.conf file. The notes on permission
choices are mine and are stronger than many suggest.

---

# Preferred permissions: root:wheel 0444
ssl_cert = /fullchain.pem

# Preferred permissions: root:wheel 0400
ssl_key = /privkey.pem

---

FreeBSD uses a different directory structure than most Linux-based
systems, so the path to the dovecot config directory may be different
for you.

I didn't ever find any documentation of the 'var = 

Re: Replacement for antispam plugin

[Jeff Kletsky]

Glad I poked around on the list today!

Thanks to all for the suggestions about integration with dspam.

I'll definitely have to look into this, as I rely on moving messages
to a specific folder with various IMAP clients to retrain dspam false
positives and negatives.

A quick pair of questions:

* Does Dovecot support the IMAP "MOVE" command at this time?

* If so, what is the syntax for "COPY or MOVE" for the _causes variables?


I did see messages from 2011 discussing it, but nothing since.


While the script looks like it could be modified for use with dspam
(with the great suggestions from others on the list), it has the same
problem as "antispam" with bulk moves being serialized and tying of
the client until they complete.  I'll probably have to break down and
look into using FreeBSD's auditd to trigger the actions and then
de-queue the successfully processed messages.

Sieve doesn't look like it can handle asynchronous processing, but I'd
certainly be interested if I'm missing something there.  One less thing
to configure and maintain!



Jeff



On 2/12/17 5:52 AM, Aki Tuomi wrote:

On February 10, 2017 at 10:06 AM Aki Tuomi  wrote:


Hi!
Since antispam plugin is deprecated and we would really prefer people
not to use it, we wrote instructions on how to replace it with
IMAPSieve. Comments and suggestions are most welcome.

https://wiki.dovecot.org/HowTo/AntispamWithSieve

---
Aki Tuomi
Dovecot oy

Hi everyone,

thank you all for your feedback, questions and comments. We have upgraded the 
documentation based on this, including information how to exclude Trash folder 
in ham script.

Aki

Any way to simplify repeated use of same string-list?

[Jeff Kletsky]

I often use the same set of header names when doing an 'address' 
comparison.
I would prefer to be able to specify that once, then re-use it in 
several places.


As an example

set "address_mailman_fields" "\"To\", \"Cc\", \"From\", \"Errors-To\"";

if address :domain :matches ["${address_mailman_fields}"] 
"example.com" {

[...]
}

if address :domain :matches ["${address_mailman_fields}"] 
"dovecot.org" {

[...]
}

or the like would be quite helpful.

From what I can tell, even though the variable interpolation *looks* 
like a string-list,

it isn't parsed as such. For example, from sieve-test output:

sieve-test(jeff): Debug: Mailbox 
Opened mail UID=1 because: header "To", "Cc", "From", "Errors-To" (Cache 
file is unusable)
testing: line 27: info: DEBUG: address ["${address_mailman_fields}"] 
didn't match.


sieve-test(jeff): Debug: Mailbox 
Opened mail UID=1 because: header ["To", "Cc", "From", "Errors-To"] 
(Cache file is unusable)
testing: line 34: info: DEBUG: address "[${address_mailman_fields}]" 
didn't match.



Is there any trick available to specify a string-list once, then re-use 
it throughout the script?



Thanks,


Jeff

Generalized looping possible in pigeonhole sieve?

[Jeff Kletsky]

I'm struggling to achieve the kind of filtering with sieve that I was 
able to do with procmail.


TL;DR

I'd like a way to loop through a set of (address, destination) pairs in 
sieve so that I can maintain the (address, destination) pairs in one 
place and not have to explicitly write scores of nearly identical


[...]

if address :matches ["From", "Sender", "To", "Cc"] "address53" {
fileinto "destination53"
}

if address :matches ["From", "Sender", "To", "Cc"] "address54" {
fileinto "destination54"
}

[...]


Longer:

I've been using a table-based dispatch approach with procmail for years 
and it was working reasonably well.


The approach uses list of pairs of address and destination boxes and 
doing a lookup based on the message to be delivered. With procmail, I 
collected the list of addresses and passed them to external scripts to 
do the lookup.


If I just had a dozen boxes that I deal with, that would be 
straightforward to implement and maintain in sieve. As the list is an 
order of magnitude greater than that, maintaining the list of pairs in 
an external file and programmatically looping through it is a lot easier.


I've looked at vnd.dovecot.execute and it certainly can return a 
destination box name given the input information.


Unfortunately, the construct of (for example)

address :matches "To" "*"'

only returns the first address, not the list of addresses.

I can't see a good way to get all the addressees from sieve to pass to 
the external program. If I have to pass the message (or at least the 
interesting headers) to the external program and then parse the address 
list myself, then I'm nearly all the way to just using a full-on 
external program to parse the whole message.


While I'm not adverse to writing the whole thing in Python (or the like) 
and just using sieve as a thin shell, I'd like to make sure I'm not 
missing something in sieve.


One way to achieve this would be to be able to loop through the pairs 
and using variable substitution for each iteration. I haven't seen 
anything in Dovecot Pigeonhole sieve that allows this.


If you have any ideas on how to harness sieve for this, I'd appreciate it!


Jeff

Processing Maildir contents on message-by-message basis

[Jeff Kletsky]

I (finally) moved over to Maildir storage here and would like to 
implement some scripts to manage taking actions on emails manually 
identified as misclassified as spam/ham. After reading through the 
Dovecot 2 description of how it works to try to see how it interacts 
with other processes changing the files.


I'm concerned that I would be corrupting the message indexes if I just 
go hog-wild and run the scripts on the filesystem, rather than through 
Dovecot in some way.


The types of actions taken would likely be:
* Select a message from a given mailbox (the source)
* Potentially modify it drastically (remove spamassassin markup, for 
example)
* Pipe the modified message to a mail-delivery agent (still running 
procmail here) and/or to sa-learn
* Assuming successful completion of the pipe action(s), remove the 
source message from the mailbox


While I can use doveadm to do bulk move/delete actions, I don't see a 
clear way to iterate through a set of messages and perform actions on them.


First off, if I wrangle and mangle the message files directly, do I have 
to worry about the indexes, or do the indexes magically repair 
themselves in cases where the messages are either altered (including 
headers) or removed?


Have I missed a way to iterate over messages and process it using 
external tools using the dovecot tools?


Has this been discussed ad nauseum and I somehow missed it? (If so, for 
which I apologize profusely).


Thanks,

Jeff

Re: [Dovecot] BEWARE: This list is being harvested for leads

[Jeff Kletsky]

Even with good intent the message in question is clearly in violation 
of CAN-SPAM and Cal. Bus. Prof. Code Sec. 17529, of which the sender was 
informed of when my server was accessed. It was very clearly an 
electronic mail message the primary purpose of which is the commercial 
advertisement or promotion of a commercial product or service. The 
sender did not have my direct consent to deliver such messages, nor did 
they have a preexisting business relationship with me.


There was no notification that subscribing to this list would result in 
off-list commercial messages. There is no way to opt out of these 
messages. The message in question did not conform to the requirements of 
CAN-SPAM in regard to, at least:

* Identifying the message as an advertisement
* Including the valid physical postal address of the sender, recognized 
by the US Postal Service
* Providing an opt-out mechanism (at all in this case), compliant with 
the CAN-SPAM requirements


Yes, in this particular instance I found the content to be not 
distasteful; however is is very clearly spam as legally defined in the 
US and in California.

Re: [Dovecot] dsync on large mailbox fails

[Jeff Kletsky]

On 04/24/2012 10:08 AM, Timo Sirainen wrote:



25934 jeff  1 1030   135M   102M CPU22 128:47 100.00% doveadm

Looks like it could be in an infinite loop. To find out where:


That's a lot of getdirentries() calls in there, repeating all the time. So I 
don't think it's an infinite loop, just a lot of slowness repeating the same 
work over and over again. See if it helps if you add:

maildir_very_dirty_syncs = yes



I hope I understood correctly when I tried

$ dsync -Dv -o maildir_very_dirty_syncs=yes backup maildir:~/Maildir

into fresh directories (moving the previous Maildir aside).

I didn't see any significant change (and it did not complete in several 
hours) on the usual UFS directory, and have also tried ZFS to see if it 
might be a filesystem issue. That has not completed either after over 
nine  hours.


From what I can see, I am guessing that it is hanging in index 
creation. Below, mail is the source directory, and Maildir is the 
destination of the sync. This is about nine hours of run time.


[jeff@mail ~]$ ls -Al mail/.imap/cron_2010-07-10
total 9
-rw---  1 jeff  jeff  4910320 Feb 22  2011 dovecot.index
-rw---  1 jeff  jeff  3600384 Apr 25 01:18 dovecot.index.cache
-rw---  1 jeff  jeff   56 Feb 22  2011 dovecot.index.log
[jeff@mail ~]$ ls -Al Maildir/.cron_2010-07-10
total 26
drwx--  2 jeff  jeff   102291 Apr 25 01:17 cur
-rw---  1 jeff  jeff   36 Apr 25 01:18 dovecot-keywords
-rw---  1 jeff  jeff  4122971 Apr 25 01:17 dovecot-uidlist
-rw---  1 jeff  jeff   24 Apr 25 07:19 dovecot-uidlist.lock
-rw---  1 jeff  jeff  1637008 Apr 25 07:03 dovecot.index
-rw---  1 jeff  jeff  9442304 Apr 25 07:19 dovecot.index.cache
-rw---  1 jeff  jeff87984 Apr 25 07:19 dovecot.index.log
-rw---  1 jeff  jeff  1048600 Apr 25 07:03 dovecot.index.log.2
-rw---  1 jeff  jeff0 Apr 24 22:12 maildirfolder
drwx--  2 jeff  jeff2 Apr 24 22:12 new
drwx--  2 jeff  jeff2 Apr 25 01:17 tmp

[Dovecot] 2.1.x changes and dsync -- Wiki suggestion

[Jeff Kletsky]

There are a couple things that show up on the Upgrading/2.1 wiki page 
that would be useful to have on pages describing dsync (Tools/Dsync, 
Migration/MailFormat, Tools, perhaps others):


* dsync was merged into doveadm
* You can't dsync 2.0.x to/from 2.1.x

That second point is especially interesting to me as it seems that 
Apple MacOS X Server is still on 2.0.16, at least from what I see on 
opensource.apple.com. I was planning on migrating mail off the current 
server onto my friend's new MacMini, but I'll have to figure out how to 
downgrade the mail jail to a 2.0 version, it seems.


Thanks for all the great work,

Jeff

Re: [Dovecot] dsync on large mailbox fails

[Jeff Kletsky]

On 4/24/12 3:26 AM, Timo Sirainen wrote:

On 24.4.2012, at 6.38, Jeff Kletsky wrote:


25934 jeff  1 1030   135M   102M CPU22 128:47 100.00% doveadm

Looks like it could be in an infinite loop. To find out where:

1) strace for a few seconds:

strace -tt -p 25934 2log

2) get gdb backtrace a couple of times (2-5 should be enough, preferably 
slightly different ones):

gdb /usr/local/bin/doveadm 25934
bt full
copypaste the output to mail
cont
ctrl-c
bt full
copypaste the output to mail
cont
ctrl-c
..etc..
Rather than clutter everyone's inbox, even if we know they have a great 
IMAP server...


http://wildside.wagsky.com/dovecot/2012-04-24/54097/doveadm.gdb.log.bz2
http://wildside.wagsky.com/dovecot/2012-04-24/54097/doveadm.truss.log.bz2

Thanks for looking into this!

Jeff

[Dovecot] dsync on large mailbox fails

[Jeff Kletsky]

I've been trying to convert an existing set of mbox mail to maildir 
using dsync, but it seems to fail when it gets to the large boxes.


$ dsync -Dv mirror maildir:~/Maildir

Everything seems to work reasonably well until the first of the large 
boxes hits:


dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron.2010-07-10: 
Synchronization took 1210 seconds (102289 new msgs, 0 flag change 
attempts, 0 expunge attempts)
dsync(jeff): Warning: Transaction log file 
/home/jeff/Maildir/.cron.2010-07-10/dovecot.index.log was locked for 
1210 seconds


After this, the doveadm process goes from ~75% of the CPU core 
utilization to 100% and stays there, even overnight.


I've also tried

$ dsync -Dv backup maildir:~/Maildir

with similar results.

While I did see some threads on dsync speed, they seemed to be minutes 
for a few GB of data, not hours.


There is plenty of space on the drive, and, from what I understand, 
there is not a file-count limit on FreeBSD's ufs (at least not one that 
I'm anywhere near)


Any suggestions?

Thanks!

Jeff



[jeff@mail ~]$ du -h -d 0 ~/mail
3.9G/home/jeff/mail

[jeff@mail ~]$ ls -lh /var/mail/jeff
-rw---  1 jeff  jeff   554M Apr 23 11:27 /var/mail/jeff

[jeff@mail ~]$ df -h
Filesystem  SizeUsed   Avail Capacity  Mounted on
/dev/ada0s2g 38G 18G 16G53%/




(Note that the hostname and the location of the ssl certificate has been 
manually edited for this message in the following output)


From dmesg.boot:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-STABLE #0: Sat Apr 21 17:13:07 PDT 2012
r...@my.host.name:/usr/obj/usr/src/sys/GENERIC i386
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz 686-class CPU)
  Origin = GenuineIntel  Id = 0x106c2  Family = 6  Model = 1c  
Stepping = 2
  
Features=0xbfe9fbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE

  Features2=0x40e31dSSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
  AMD Features=0x2010NX,LM
  AMD Features2=0x1LAHF
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2077757440 (1981 MB)




[jeff@mail ~]$ doveconf -n
# 2.1.3: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.0-STABLE i386
auth_mechanisms = plain login
disable_plaintext_auth = no
first_valid_gid = 1000
first_valid_uid = 1000
listen = *
mail_location = mbox:~/mail/:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
}
passdb {
  args = session=yes dovecot
  driver = pam
}
protocols = imap
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  unix_listener auth-master {
mode = 0600
  }
  user = root
}
service imap-login {
  inet_listener imap {
port = 0
  }
  process_limit = 200
}
ssl = required
ssl_cert = /path/to/certificate.pem
ssl_key = /path/to/certificate.pem
userdb {
  args = blocking=yes
  driver = passwd
}
verbose_proctitle = yes
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
}

Re: [Dovecot] dsync on large mailbox fails

[Jeff Kletsky]

On 4/23/12 12:56 PM, Jeff Kletsky wrote:
I've been trying to convert an existing set of mbox mail to maildir 
using dsync, but it seems to fail when it gets to the large boxes.


$ dsync -Dv mirror maildir:~/Maildir

Everything seems to work reasonably well until the first of the large 
boxes hits:


dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron.2010-07-10: 
Synchronization took 1210 seconds (102289 new msgs, 0 flag change 
attempts, 0 expunge attempts)
dsync(jeff): Warning: Transaction log file 
/home/jeff/Maildir/.cron.2010-07-10/dovecot.index.log was locked for 
1210 seconds


After this, the doveadm process goes from ~75% of the CPU core 
utilization to 100% and stays there, even overnight.




I believe that some of problem may be that there were a handful of boxes 
named with . in them. These were not flagged in the conversion 
process in any way.


http://wiki2.dovecot.org/Migration/MailFormat indicates that you can't 
have a . in the name of a Maildir box


(I dutifully stopped reading that page at If you can successfully use 
dsync, you can skip the rest of this page.)


In my case, it may be a contributing factor that there were boxes that 
ended up stacking inside each other:

* cron
* cron.2010-07-10
* cron.2010-07-10.remaining

Still not blisteringly fast, but

dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron_2010-07-10: 
Synchronization took 686 seconds (102289 new msgs, 0 flag change 
attempts, 0 expunge attempts)
dsync(jeff): Warning: Transaction log file 
/home/jeff/Maildir/.cron_2010-07-10/dovecot.index.log was locked for 686 
seconds


is certainly better.

I'm still seeing 100% CPU utilization, with the tail of ls -Altr in 
~/Maildir showing


drwx--  5 jeff  jeff   512 Apr 23 18:19 .XDA
drwx--  5 jeff  jeff   512 Apr 23 18:19 .Spam
drwx--  5 jeff  jeff   512 Apr 23 18:42 .Health
drwx--  5 jeff  jeff   512 Apr 23 18:42 .cron_2010-07-10_remaining
drwx--  5 jeff  jeff   512 Apr 23 18:58 .cron_2010-07-10

It appears from looking in the .cron_2010-07-10 directory that indexing 
is the action underway


[jeff@mail ~/Maildir]$ ls -Altr .cron_2010-07-10
total 27
drwx--  2 jeff  jeff  512 Apr 23 18:15 new
-rw---  1 jeff  jeff0 Apr 23 18:15 maildirfolder
drwx--  2 jeff  jeff  5819392 Apr 23 18:42 tmp
drwx--  2 jeff  jeff  5237248 Apr 23 18:42 cur
-rw---  1 jeff  jeff  4122971 Apr 23 18:42 dovecot-uidlist
-rw---  1 jeff  jeff   36 Apr 23 18:42 dovecot-keywords
-rw---  1 jeff  jeff  1650764 Apr 23 18:42 dovecot.index.log.2
-rw---  1 jeff  jeff  1637008 Apr 23 18:42 dovecot.index
-rw---  1 jeff  jeff56124 Apr 23 19:04 dovecot.index.log
-rw---  1 jeff  jeff  9442304 Apr 23 19:04 dovecot.index.cache
-rw---  1 jeff  jeff   24 Apr 23 19:04 dovecot-uidlist.lock

(time passes)

It is now 20:35 and it has not advanced. I now no longer believe the 
problem is the mailbox naming, but the indexing.


  PID USERNAMETHR PRI NICE   SIZERES STATE   C   TIME   WCPU 
COMMAND
25934 jeff  1 1030   135M   102M CPU22 128:47 100.00% 
doveadm


[jeff@mail ~/Maildir]$ ls -Altr .cron_2010-07-10
total 28
drwx--  2 jeff  jeff  512 Apr 23 18:15 new
-rw---  1 jeff  jeff0 Apr 23 18:15 maildirfolder
drwx--  2 jeff  jeff  5819392 Apr 23 18:42 tmp
drwx--  2 jeff  jeff  5237248 Apr 23 18:42 cur
-rw---  1 jeff  jeff  4122971 Apr 23 18:42 dovecot-uidlist
-rw---  1 jeff  jeff   36 Apr 23 18:42 dovecot-keywords
-rw---  1 jeff  jeff  1650764 Apr 23 18:42 dovecot.index.log.2
-rw---  1 jeff  jeff  1637008 Apr 23 20:26 dovecot.index
-rw---  1 jeff  jeff   289120 Apr 23 20:36 dovecot.index.log
-rw---  1 jeff  jeff  9442304 Apr 23 20:36 dovecot.index.cache
-rw---  1 jeff  jeff   24 Apr 23 20:36 dovecot-uidlist.lock


Any suggestions as to how to proceed?

Jeff