Re: [Dovecot] Lucene and Zlib with 2.2.1
On Wednesday, May 15 at 04:29 PM, quoth Timo Sirainen:
On 25.4.2013, at 19.55, Kyle Wheeler kyle-dove...@memoryhole.net wrote:
I have an archive folder in my inbox, where I manually stick old mails into a
compressed mbox format. Since upgrading to Dovecot 2.2.1, I've started seeing messages
like the following in my log files:
imap(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable
indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable
indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable
indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable
imap(...): Error: indexer failed to index mailbox INBOX/Archive/2007/Sent.gz
What's your doveconf -n output? The errors about INBOX are
especially strange (maybe just the error message is wrong and it's
actually trying to index INBOX/Archive).
Here it is:
# 2.2.1: /service/dovecot-memoryhole.net/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.7
auth_default_realm = memoryhole.net
auth_mechanisms = plain login
base_dir = /var/run/dovecot/memoryhole
first_valid_gid = 64020
first_valid_uid = 64020
last_valid_gid = 64020
last_valid_uid = 64020
log_path = /dev/stderr
log_timestamp =
login_greeting = ...you two suckers! Stop shirkin' and get workin'!
mail_gid = 64020
mail_location = maildir:%h/Maildir
mail_max_userip_connections = 20
mail_plugins = fts fts_lucene zlib listescape
mail_uid = 64020
mailbox_list_index = yes
mbox_write_locks = fcntl
namespace {
hidden = no
inbox = yes
list = yes
location = maildir:~/Maildir
prefix =
separator = /
type = private
}
namespace {
hidden = yes
inbox = no
location = mbox:~/Maildir/Archive:LAYOUT=fs:INDEX=~/Maildir/ArchiveIndexes/
prefix = INBOX/Archive/
separator = /
type = private
}
passdb {
args = /var/lib/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
fts = lucene
fts_lucene = whitespace_chars=@.
zlib_save = gz
zlib_save_level = 6
}
protocols = imap
service auth {
user = vpopmail
}
service imap-login {
inet_listener imap {
address = imap.memoryhole.net
port = 143
}
inet_listener imaps {
address = imap.memoryhole.net
port = 993
}
user = dovecot
}
service imap {
executable = imap
}
service pop3-login {
user = dovecot
}
ssl_cert = /etc/ssl/certs/imap.memoryhole.net.pem
ssl_key = /etc/ssl/private/imap.memoryhole.net.key
userdb {
args = uid=64020 gid=64020 home=/var/lib/vpopmail/domains/%Ld/%Ln
allow_all_users=yes
driver = static
}
valid_chroot_dirs = /var/lib/vpopmail/domains
verbose_proctitle = yes
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep delay-newmail tb-lsub-flags
imap_idle_notify_interval = 29 mins
imap_logout_format = writebytes=%o, readbytes=%i, session=%{session}
mail_plugins = fts fts_lucene zlib imap_zlib listescape
}
protocol lda {
auth_socket_path = /var/run/dovecot/localhost/auth-master
hostname = memoryhole.net
mail_plugins = fts fts_lucene zlib listescape
postmaster_address = postmas...@memoryhole.net
}
~Kyle
--
The borrower is the slave of the lender.
-- Proverbs 22:7
pgp7cTsb7fvvY.pgp
Description: PGP signature
Re: [Dovecot] Dovecot vs MBox
On Tuesday, April 30 at 08:04 AM, quoth Gregory Sloop: Any ideas where to look next, what I might do to force dovecot to forget message ID's etc - that might force it to read the whole mailbox file again? Find the dovecot.index files for that mbox and delete them. They will be re-generated from the contents of the mbox. ~Kyle -- If man was meant to be nude, he would have been born that way. -- Oscar Wilde pgpRYjMHw3Yzv.pgp Description: PGP signature
Re: [Dovecot] mailbox_list_index_parse_header crash
On Tuesday, April 23 at 09:06 PM, quoth Timo Sirainen: On Tue, 2013-04-23 at 10:52 -0600, Kyle Wheeler wrote: I got all excited about IMAP NOTIFY support and tried enabling mailbox_list_index on my server. Unfortunately, I rather quickly ran into trouble of the segfault variety. This prevented me from doing basic things like seeing the contents of some of my folders. Disabling mailbox_list_index got me back up and running, but... darnit, that's annoying. The only message I got in my log file was this: imap(user@domain): Fatal: master: service(imap): child 6899 killed with signal 11 (core dumped) If I can help track this down, please let me know. Here's the backtrace: #0 hash_table_insert_node (table=0x10eff60, key=0x103, value=0x10fd8b0, check_existing=value optimized out) at hash.c:268 node = 0x69616d2d746e6573 prev = 0x10fbee0 hash = value optimized out __FUNCTION__ = hash_table_insert_node http://hg.dovecot.org/dovecot-2.2/rev/d3d380221043 should help? Solved the problem - thanks! ~Kyle -- Thanks to the power of denial, I'm immortal! -- Phillip J. Fry, Futurama pgpDes4ZjB9f0.pgp Description: PGP signature
Re: [Dovecot] Lucene and Zlib with 2.2.1
On Thursday, April 25 at 07:44 PM, quoth Robert Schetterer: Am 25.04.2013 18:55, schrieb Kyle Wheeler: As you can probably tell, I'm using the fts_lucene plugin, which used to work just fine (with Dovecot 2.1.x). Is this expected behavior? Are these errors truly ignorable? Why are these compressed mailboxes not selectable? sorry to ask , but did you compile with lucene ? Yes I did. I configured my compile like this: ./configure --without-sql --without-vpopmail --with-ldap \ --with-lucene --with-stemmer I've got CLucene 2.3.3.4 installed, and libstemmer version 0+svn527-1 (the version that's packaged as part of Debian). Running ldd on /usr/local/lib/dovecot/lib21_fs_lucene_plugin.so says it's linked against both libclucene and libstemmer: /usr/local/lib/dovecot/lib21_fts_lucene_plugin.so: linux-vdso.so.1 = (0x7fff3f3ff000) libclucene-core.so.1 = /usr/local/lib/libclucene-core.so.1 (0x7f9b0e5f5000) libstemmer.so.0d = /usr/lib/libstemmer.so.0d (0x7f9b0e3a4000) librt.so.1 = /lib/librt.so.1 (0x7f9b0e19b000) libstdc++.so.6 = /usr/lib/libstdc++.so.6 (0x7f9b0de87000) libm.so.6 = /lib/libm.so.6 (0x7f9b0dc05000) libc.so.6 = /lib/libc.so.6 (0x7f9b0d8a2000) libgcc_s.so.1 = /lib/libgcc_s.so.1 (0x7f9b0d68c000) libclucene-shared.so.1 = /usr/local/lib/libclucene-shared.so.1 (0x7f9b0d46c000) libpthread.so.0 = /lib/libpthread.so.0 (0x7f9b0d24f000) libz.so.1 = /usr/lib/libz.so.1 (0x7f9b0d038000) /lib64/ld-linux-x86-64.so.2 (0x7f9b0ebde000) ~Kyle -- It was we, the people; not we, the white male citizens; nor yet we, the male citizens; but we, the whole people, who formed the Union. [...] Men, their rights and nothing more; women, their rights and nothing less. -- Susan B. Anthony pgp6pEYcl3jtC.pgp Description: PGP signature
[Dovecot] Lucene and Zlib with 2.2.1
Hello, I have an archive folder in my inbox, where I manually stick old mails into a compressed mbox format. Since upgrading to Dovecot 2.2.1, I've started seeing messages like the following in my log files: imap(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable indexer-worker(...): Error: lucene: Failed to sync mailbox INBOX: Mailbox isn't selectable imap(...): Error: indexer failed to index mailbox INBOX/Archive/2007/Sent.gz These errors *appear* to be mostly benign; I still get results from searching the mailbox. But for whatever reason, the indexer appears to not be working for those folders. As you can probably tell, I'm using the fts_lucene plugin, which used to work just fine (with Dovecot 2.1.x). Is this expected behavior? Are these errors truly ignorable? Why are these compressed mailboxes not selectable? ~Kyle -- Where justice is denied, where poverty is enforced, where ignorance prevails, and where any one class is made to feel that society is an organized conspiracy to oppress, rob and degrade them, neither persons nor property will be safe. -- Frederick Douglas pgpjUsJjJskam.pgp Description: PGP signature
[Dovecot] mailbox_list_index_parse_header crash
Hi,
I got all excited about IMAP NOTIFY support and tried enabling
mailbox_list_index on my server. Unfortunately, I rather quickly ran
into trouble of the segfault variety. This prevented me from doing
basic things like seeing the contents of some of my folders. Disabling
mailbox_list_index got me back up and running, but... darnit, that's
annoying. The only message I got in my log file was this:
imap(user@domain): Fatal: master: service(imap): child 6899 killed with signal
11 (core dumped)
If I can help track this down, please let me know. Here's the
backtrace:
#0 hash_table_insert_node (table=0x10eff60, key=0x103, value=0x10fd8b0,
check_existing=value optimized out) at hash.c:268
node = 0x69616d2d746e6573
prev = 0x10fbee0
hash = value optimized out
__FUNCTION__ = hash_table_insert_node
#1 0x7fad738a11be in hash_table_insert (table=0x10eff60, key=0x1,
value=0x101) at hash.c:285
node = 0x0
#2 0x7fad73b81e68 in mailbox_list_index_parse_header (list=value optimized
out, view=0x11095f0,
force=value optimized out) at mailbox-list-index.c:196
data = 0x10f5f30
i = 4980
len = 1
size = 8192
id = 1
#3 mailbox_list_index_parse (list=value optimized out, view=0x11095f0,
force=value optimized out)
at mailbox-list-index.c:317
ilist = 0x10edc80
hdr = value optimized out
error = 0x10f5820 `▒\016\001
__FUNCTION__ = mailbox_list_index_parse
#4 0x7fad73b86b03 in mailbox_list_index_sync_begin (list=0x10ed890,
sync_ctx_r=0x7fff2ad61ee0)
at mailbox-list-index-sync.c:247
ilist = 0x10edc80
sync_ctx = value optimized out
index_sync_ctx = value optimized out
view = 0x11095f0
trans = value optimized out
hdr = value optimized out
#5 0x7fad73b86ffb in mailbox_list_index_sync (list=0x10eff60) at
mailbox-list-index-sync.c:382
sync_ctx = 0x1107300
ret = value optimized out
#6 0x7fad73b8233f in mailbox_list_index_refresh (list=0x10ed890) at
mailbox-list-index.c:376
ilist = 0x10edc80
view = 0x1107300
ret = value optimized out
#7 0x7fad73b85e19 in index_list_update_mailbox (box=0x10f40a0) at
mailbox-list-index-status.c:363
list_sync_ctx = 0x1107140
list_view = value optimized out
list_trans = value optimized out
changes = {status = {messages = 718675880, recent = 32767, unseen =
4294967295, uidvalidity = 0, uidnext = 68,
first_unseen_seq = 0, first_recent_uid = 1, last_cached_seq = 0,
highest_modseq = 3176264,
highest_pvt_modseq = 17854784, keywords = 0x10f40a0,
permanent_flags = 17778368, permanent_keywords = 0,
allow_new_keywords = 0, nonpermanent_modseqs = 0, have_guids = 0},
guid = p ▒*▒\177\000\000D\000\000\000\000\000\000, seq = 17750944,
rec_changed = false, msgs_changed = false,
hmodseq_changed = false}
__FUNCTION__ = index_list_update_mailbox
#8 0x7fad73b86089 in index_list_sync_deinit (ctx=0x1107140,
status_r=0x7fff2ad62070) at mailbox-list-index-status.c:470
box = 0x10f40a0
#9 0x7fad73b70f4a in mailbox_sync_deinit (_ctx=0x0, status_r=0x1) at
mail-storage.c:1655
ctx = 0x10eff60
box = 0x10f40a0
errormsg = value optimized out
error = value optimized out
ret = value optimized out
#10 0x7fad73b7100b in mailbox_sync (box=value optimized out, flags=value
optimized out) at mail-storage.c:1681
ctx = 0x0
status = {sync_delayed_expunges = 0}
#11 0x7fad73b9af72 in index_storage_get_status (box=0x10f40a0, items=17,
status_r=value optimized out)
at index-status.c:39
No locals.
#12 0x7fad72cabde4 in fts_mailbox_get_status (box=0x10f40a0, items=17,
status_r=0x7fff2ad621b0) at fts-storage.c:86
seq = value optimized out
#13 0x7fad73b864e6 in index_list_get_status (box=0x10f40a0, items=17,
status_r=0x7fff2ad621b0)
at mailbox-list-index-status.c:162
No locals.
#14 0x0041c8da in imap_status_get (cmd=0x10f1880, ns=0x10e8b00, mailbox=0x10cc2c8
Flight.RV, items=0x7fff2ad62240,
result_r=0x7fff2ad621b0) at imap-status.c:84
client = 0x10f0ce0
box = 0x10f40a0
errstr = value optimized out
ret = value optimized out
#15 0x0041219d in cmd_status (cmd=0x10f1880) at cmd-status.c:40
client = 0x10f0ce0
args = 0x10d7a38
list_args = 0x10d7be0
items = {status = 17, metadata = 0}
result = {status = {messages = 0, recent = 0, unseen = 0, uidvalidity =
0, uidnext = 0, first_unseen_seq = 0,
first_recent_uid = 0, last_cached_seq = 0, highest_modseq = 0,
highest_pvt_modseq = 0, keywords = 0x0,
permanent_flags = 0, permanent_keywords = 0, allow_new_keywords =
0, nonpermanent_modseqs = 0, have_guids = 1},
metadata = {guid = PY\r\001, '\000' repeats 11 times,
virtual_size =
Re: [Dovecot] mailbox_list_index_parse_header crash
Sorry, I should have said; I'm running 2.2.1
On Tuesday, April 23 at 10:52 AM, quoth Kyle Wheeler:
Hi,
I got all excited about IMAP NOTIFY support and tried enabling
mailbox_list_index on my server. Unfortunately, I rather quickly ran
into trouble of the segfault variety. This prevented me from doing
basic things like seeing the contents of some of my folders. Disabling
mailbox_list_index got me back up and running, but... darnit, that's
annoying. The only message I got in my log file was this:
imap(user@domain): Fatal: master: service(imap): child 6899 killed with signal
11 (core dumped)
If I can help track this down, please let me know. Here's the
backtrace:
#0 hash_table_insert_node (table=0x10eff60, key=0x103, value=0x10fd8b0,
check_existing=value optimized out) at hash.c:268
node = 0x69616d2d746e6573
prev = 0x10fbee0
hash = value optimized out
__FUNCTION__ = hash_table_insert_node
#1 0x7fad738a11be in hash_table_insert (table=0x10eff60, key=0x1,
value=0x101) at hash.c:285
node = 0x0
#2 0x7fad73b81e68 in mailbox_list_index_parse_header (list=value optimized
out, view=0x11095f0,
force=value optimized out) at mailbox-list-index.c:196
data = 0x10f5f30
i = 4980
len = 1
size = 8192
id = 1
#3 mailbox_list_index_parse (list=value optimized out, view=0x11095f0,
force=value optimized out)
at mailbox-list-index.c:317
ilist = 0x10edc80
hdr = value optimized out
error = 0x10f5820 `▒\016\001
__FUNCTION__ = mailbox_list_index_parse
#4 0x7fad73b86b03 in mailbox_list_index_sync_begin (list=0x10ed890,
sync_ctx_r=0x7fff2ad61ee0)
at mailbox-list-index-sync.c:247
ilist = 0x10edc80
sync_ctx = value optimized out
index_sync_ctx = value optimized out
view = 0x11095f0
trans = value optimized out
hdr = value optimized out
#5 0x7fad73b86ffb in mailbox_list_index_sync (list=0x10eff60) at
mailbox-list-index-sync.c:382
sync_ctx = 0x1107300
ret = value optimized out
#6 0x7fad73b8233f in mailbox_list_index_refresh (list=0x10ed890) at
mailbox-list-index.c:376
ilist = 0x10edc80
view = 0x1107300
ret = value optimized out
#7 0x7fad73b85e19 in index_list_update_mailbox (box=0x10f40a0) at
mailbox-list-index-status.c:363
list_sync_ctx = 0x1107140
list_view = value optimized out
list_trans = value optimized out
changes = {status = {messages = 718675880, recent = 32767, unseen =
4294967295, uidvalidity = 0, uidnext = 68,
first_unseen_seq = 0, first_recent_uid = 1, last_cached_seq = 0,
highest_modseq = 3176264,
highest_pvt_modseq = 17854784, keywords = 0x10f40a0, permanent_flags
= 17778368, permanent_keywords = 0,
allow_new_keywords = 0, nonpermanent_modseqs = 0, have_guids = 0},
guid = p ▒*▒\177\000\000D\000\000\000\000\000\000, seq = 17750944,
rec_changed = false, msgs_changed = false,
hmodseq_changed = false}
__FUNCTION__ = index_list_update_mailbox
#8 0x7fad73b86089 in index_list_sync_deinit (ctx=0x1107140,
status_r=0x7fff2ad62070) at mailbox-list-index-status.c:470
box = 0x10f40a0
#9 0x7fad73b70f4a in mailbox_sync_deinit (_ctx=0x0, status_r=0x1) at
mail-storage.c:1655
ctx = 0x10eff60
box = 0x10f40a0
errormsg = value optimized out
error = value optimized out
ret = value optimized out
#10 0x7fad73b7100b in mailbox_sync (box=value optimized out, flags=value
optimized out) at mail-storage.c:1681
ctx = 0x0
status = {sync_delayed_expunges = 0}
#11 0x7fad73b9af72 in index_storage_get_status (box=0x10f40a0, items=17,
status_r=value optimized out)
at index-status.c:39
No locals.
#12 0x7fad72cabde4 in fts_mailbox_get_status (box=0x10f40a0, items=17,
status_r=0x7fff2ad621b0) at fts-storage.c:86
seq = value optimized out
#13 0x7fad73b864e6 in index_list_get_status (box=0x10f40a0, items=17,
status_r=0x7fff2ad621b0)
at mailbox-list-index-status.c:162
No locals.
#14 0x0041c8da in imap_status_get (cmd=0x10f1880, ns=0x10e8b00, mailbox=0x10cc2c8
Flight.RV, items=0x7fff2ad62240,
result_r=0x7fff2ad621b0) at imap-status.c:84
client = 0x10f0ce0
box = 0x10f40a0
errstr = value optimized out
ret = value optimized out
#15 0x0041219d in cmd_status (cmd=0x10f1880) at cmd-status.c:40
client = 0x10f0ce0
args = 0x10d7a38
list_args = 0x10d7be0
items = {status = 17, metadata = 0}
result = {status = {messages = 0, recent = 0, unseen = 0, uidvalidity =
0, uidnext = 0, first_unseen_seq = 0,
first_recent_uid = 0, last_cached_seq = 0, highest_modseq = 0,
highest_pvt_modseq = 0, keywords = 0x0,
permanent_flags = 0, permanent_keywords = 0, allow_new_keywords = 0,
nonpermanent_modseqs = 0, have_guids = 1},
metadata = {guid = PY\r\001
Re: [Dovecot] mailbox_list_index_parse_header crash
On Tuesday, April 23 at 09:06 PM, quoth Timo Sirainen: On Tue, 2013-04-23 at 10:52 -0600, Kyle Wheeler wrote: I got all excited about IMAP NOTIFY support and tried enabling mailbox_list_index on my server. Unfortunately, I rather quickly ran into trouble of the segfault variety. This prevented me from doing basic things like seeing the contents of some of my folders. Disabling mailbox_list_index got me back up and running, but... darnit, that's annoying. The only message I got in my log file was this: imap(user@domain): Fatal: master: service(imap): child 6899 killed with signal 11 (core dumped) If I can help track this down, please let me know. Here's the backtrace: #0 hash_table_insert_node (table=0x10eff60, key=0x103, value=0x10fd8b0, check_existing=value optimized out) at hash.c:268 node = 0x69616d2d746e6573 prev = 0x10fbee0 hash = value optimized out __FUNCTION__ = hash_table_insert_node http://hg.dovecot.org/dovecot-2.2/rev/d3d380221043 should help? Excellent - running tests now... answers soon. ~Kyle -- Human beings are the only creatures that allow their children to come back home. -- Bill Cosby pgpt31sqQeiQL.pgp Description: PGP signature
Re: [Dovecot] namespaces and noselect
On Tuesday, January 4 at 12:14 PM, quoth Timo Sirainen: On Thu, 2010-12-30 at 08:33 -0700, Kyle Wheeler wrote: I'm using Dovecot 2.0.6. Here's the output of dovecot -n: .. 2 LIST INBOX * LIST (\HasChildren) / INBOX * LIST (\Noselect \HasChildren) / INBOX 2 OK List completed. I remember fixing something related to this. Also I couldn't reproduce this with your config and latest hg version. Maybe it's already fixed in v2.0.8. I just upgraded to v2.0.8 and was able to reproduce the problem. :( I don't see anything in the hg changelog summaries since 2.0.8 was released that suggest something related to this... hrm. If it helps understand the problem, here's a slightly different example of the problem: 2 LIST % * LIST (\HasChildren) . INBOX * LIST (\HasNoChildren) . Trash * LIST (\HasNoChildren) . Sent * LIST (\HasNoChildren) . addressbook * LIST (\HasNoChildren) . remote_pinerc * LIST (\Noselect \HasChildren) . INBOX 2 OK List completed. The fact that it repeated INBOX *after* giving all the mailboxes in the first namespace suggested to me that it was the second namespace that is (for some reason) generating this unselectable INBOX. Is there anything else I can provide you with to help reproduce it? ~Kyle -- To brand a book as unsuitable is an important step toward making it required reading. -- Marvin Kaye signature.asc Description: Digital signature
Re: [Dovecot] namespaces and noselect
On Tuesday, January 4 at 10:08 AM, quoth Kyle Wheeler: I just upgraded to v2.0.8 and was able to reproduce the problem. :( I don't see anything in the hg changelog summaries since 2.0.8 was released that suggest something related to this... hrm. If it helps understand the problem, here's a slightly different example of the problem: Actually, this bug seems to be quite strange. Here's another IMAP conversation (different account than before) that shows it happening in one case and not happening in another. 2 LIST % * LIST (\HasChildren) / INBOX * LIST (\HasNoChildren) / Sent Messages * LIST (\HasNoChildren) / Notes * LIST (\HasNoChildren) / Apple Mail To Do * LIST (\HasNoChildren) / Drafts * LIST (\HasNoChildren) / Deleted Messages 2 OK List completed. 3 LIST INBOX * LIST (\HasChildren) / INBOX * LIST (\Noselect \HasChildren) / INBOX 3 OK List completed. What on earth could be going on? ~Kyle -- Those who profess to favor freedom, and yet depreciate agitation, are men who want rain without thunder and lightning. -- Frederick Douglass signature.asc Description: Digital signature
Re: [Dovecot] namespaces and noselect
On Thursday, December 30 at 12:33 PM, quoth Timo Sirainen:
On Sun, 2010-12-26 at 14:40 -0600, Kyle Wheeler wrote:
I am trying to use two namespaces to create an archival directory
that is stored as mboxes (the rest of my tree is all stored as
maildirs). However, when I add the second namespace, suddenly
Dovecot starts emitting, in response to the LIST command, a second
version of the INBOX that is marked as \NoSelect. Here are my
namespac definitions... am I doing this wrong?
That's definitely a bug if there are two INBOXes listed. But you
didn't say which Dovecot version does this.
Ahh, well, I wasn't sure if I was doing it right.
I'm using Dovecot 2.0.6. Here's the output of dovecot -n:
# 2.0.6: /service/dovecot-memoryhole.net//dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian squeeze/sid
auth_default_realm = memoryhole.net
auth_mechanisms = plain login
base_dir = /var/run/dovecot/memoryhole
first_valid_gid = 64020
first_valid_uid = 64020
last_valid_gid = 64020
last_valid_uid = 64020
log_path = /dev/stderr
log_timestamp =
login_greeting = ...you two suckers! Stop shirkin' and get workin'!
mail_gid = 64020
mail_location = maildir:%h/Maildir
mail_plugins = fts fts_squat zlib listescape
mail_uid = 64020
mbox_write_locks = fcntl
namespace {
hidden = yes
list = no
location = mbox:~/Maildir/Archive:LAYOUT=fs:INDEX=~/Maildir/ArchiveIndexes/
prefix = INBOX/Archive/
separator = /
type = private
}
namespace {
hidden = no
inbox = yes
list = yes
location = maildir:~/Maildir
prefix =
separator = /
type = private
}
passdb {
args = /var/lib/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_squat = partial=4 full=4
zlib_save = gz
zlib_save_level = 6
}
protocols = imap
service auth {
user = vpopmail
}
service imap-login {
inet_listener imap {
address = imap.memoryhole.net
port = 143
}
inet_listener imaps {
address = imap.memoryhole.net
port = 993
}
service_count = 0
user = dovecot
}
service imap {
executable = /usr/local/bin/relay-ctrl-allow-wrapper.sh
/usr/local/libexec/dovecot/imap
service_count = 0
}
service pop3-login {
user = dovecot
}
ssl_cert = /etc/ssl/certs/imap.memoryhole.net.pem
ssl_key = /etc/ssl/private/imap.memoryhole.net.key
userdb {
args = uid=64020 gid=64020 home=/var/lib/vpopmail/domains/%Ld/%Ln
allow_all_users=yes
driver = static
}
valid_chroot_dirs = /var/lib/vpopmail/domains
verbose_proctitle = yes
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep delay-newmail
imap_logout_format = writebytes=%o, readbytes=%i
mail_plugins = fts fts_squat zlib imap_zlib listescape
}
protocol lda {
auth_socket_path = /var/run/dovecot/localhost/auth-master
hostname = memoryhole.net
mail_plugins = fts fts_squat zlib listescape
postmaster_address = postmas...@memoryhole.net
}
Here is an example IMAP conversation:
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
AUTH=PLAIN AUTH=LOGIN] ...you two suckers! Stop shirkin' and get workin'!
1 LOGIN kyle
1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS COMPRESS=DEFLATE] Logged in
2 LIST INBOX
* LIST (\HasChildren) / INBOX
* LIST (\Noselect \HasChildren) / INBOX
2 OK List completed.
3 LOGOUT
* BYE Logging out
3 OK Logout completed.
Just out of curiosity... why is IDLE listed twice in the second
CAPABILITY response?
~Kyle
--
A woman has the last word in any argument. Anything a man says after
that is the beginning of a new argument.
-- Unknown
pgp8ddJac6yZj.pgp
Description: PGP signature
[Dovecot] namespaces and noselect
Hello,
I am trying to use two namespaces to create an archival directory
that is stored as mboxes (the rest of my tree is all stored as
maildirs). However, when I add the second namespace, suddenly Dovecot
starts emitting, in response to the LIST command, a second version of
the INBOX that is marked as \NoSelect. Here are my namespac
definitions... am I doing this wrong?
namespace {
list = yes
hidden = no
inbox = yes
location = maildir:~/Maildir
prefix =
separator = /
type = private
}
namespace {
hidden = yes # do not list as a separate namespace
list = no #
inbox = no
location = mbox:~/Maildir/Archive:LAYOUT=fs:INDEX=~/Maildir/ArchiveIndexes/
prefix = INBOX/Archive/
separator = /
type = private
}
~Kyle
--
If you are going through hell, keep going.
-- Winston Churchill
signature.asc
Description: Digital signature
Re: [Dovecot] dovecot genesis v2.0.X
On Monday, October 18 at 04:12 PM, quoth Jim Pazarena: some over whelming need to update, and I would really like to know what this is Mail folders containing both messages and sub-folders is what I/my clients desire. I used Maildir++ layout with mboxes on dovecot 1.2.x. It wasn't documented in the wiki, but it supported the LAYOUT=Maildir++ option in mail_location just as well as dovecot 2.x does. Give it a try! ~Kyle -- The rule is, jam tomorrow and jam yesterday---but never jam today. -- Lewis Carroll pgp8R1xEUEkpS.pgp Description: PGP signature
[Dovecot] zlib plugin weirdness
Hello, I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Error: Next message unexpectedly corrupted in mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz at 1907 Error: Unexpectedly lost From-line from mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz at 1907 Error: zlib.read(/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz): unexpected EOF at 1765954 Error: i_stream_get_size() failed with mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz: No such file or directory A similar thing happens when I bzip it. I was worried that it wasn't even using the zlib plugin, but since it makes reference to zlib.read, I assume that it is. Does anyone know what might be going wrong? ~Kyle -- The only way to oppose a bad idea is to replace it with a good idea. -- Jack Kemp signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 05:06 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote: I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 No, that causes a rather ugly panic: Error: zlib.read(/var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.gz): unexpected EOF at 1764046 Panic: file istream-zlib.c: line 204 (i_stream_zlib_read): assertion failed: (zstream-eof_offset == high_offset) Error: Raw backtrace: /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 [0xb76769e1] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0 [0xb7676a5f] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_error+0) [0xb7676d18] - /tmp/dovecottesting/lib/dovecot/lib20_zlib_plugin.so [0xb74aaf7a] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(i_stream_read+0x7c) [0xb767d50c] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb7730cc1] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(istream_raw_mbox_get_header_offset+0x78) [0xb77316a8] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb773bb99] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0 [0xb773e154] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_sync+0x4b) [0xb773f35b] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mbox_storage_sync_init+0x6d) [0xb773f4cd] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x39) [0xb76d0949] - /tmp/dovecottesting/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x44) [0xb76d11f4] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](cmd_select_full+0x1d2) [0x8053962] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](cmd_select+0x19) [0x80543d9] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x805673c] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x80567d9] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](client_handle_input+0x2d) [0x805694d] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](client_input+0x5f) [0x805733f] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xf5) [0xb7683275] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) [0xb7682250] - /tmp/dovecottesting/lib/dovecot/libdovecot.so.0(master_service_run+0x2a) [0xb766f72a] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT](main+0x2b5) [0x805ffb5] - /lib/i686/cmov/libc.so.6(__libc_start_main+0xe5) [0xb74e3455] - dovecot/imap [k...@memoryhole.net 64.253.106.173 SELECT] [0x804eac1] master: Error: service(imap): child 15613 killed with signal 6 (core dumped) ~Kyle -- Why, this Satan's drink is so delicious that it would be a pity to let the infidels have exclusive use of it. We shall fool Satan by baptizing it, and making it a truly Christian beverage. -- Pope Clement VII, referring to coffee signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 05:46 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 11:21 -0500, Kyle Wheeler wrote: On Friday, October 1 at 05:06 PM, quoth Timo Sirainen: On Fri, 2010-10-01 at 10:21 -0500, Kyle Wheeler wrote: I'm trying to get the zlib plugin working on my 2.0 server. I started with an mbox that Dovecot can read just fine. Then I gzipped it, and now Dovecot complains that it's corrupted: Does this fix it? http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 No, that causes a rather ugly panic: What about the above + http://hg.dovecot.org/dovecot-2.0/rev/e7768ec9d3de That does it for gz-compressed files; it works! However, for bz2-compressed files, I get: Error: Next message unexpectedly corrupted in mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: Unexpectedly lost From-line from mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: i_stream_get_size() failed with mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: No such file or directory ~Kyle -- Time is an illusion. Lunchtime doubly so. -- Douglas Adams signature.asc Description: Digital signature
Re: [Dovecot] zlib plugin weirdness
On Friday, October 1 at 11:51 AM, quoth Kyle Wheeler: That does it for gz-compressed files; it works! However, for bz2-compressed files, I get: Error: Next message unexpectedly corrupted in mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: Unexpectedly lost From-line from mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2 at 1907 Error: i_stream_get_size() failed with mbox file /var/lib/vpopmail/domains/memoryhole.net/kyle/Maildir/Archive/Intellego.bz2: No such file or directory I made the change suggested by http://hg.dovecot.org/dovecot-2.0/rev/ab24859c3527 to istream-bzlib.c, and things now work perfectly for bz2-compressed files as well. ~Kyle -- He who joyfully marches in rank and file has already earned my contempt. He has been given a large brain by mistake, since for him the spinal cord would suffice. -- Albert Einstein signature.asc Description: Digital signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Wednesday, September 1 at 06:08 PM, quoth Timo Sirainen: On Wed, 2010-08-25 at 15:34 -0500, Kyle Wheeler wrote: Attached. It looks like the rest of the commands in the same packet after mutt told Dovecot that it was DONE with IDLE were ignored. Hmh. I can't seem to find anything wrong with the code. I also tried mutt (1.5.20-7ubuntu1) with imap_idle enabled and didn't manage to break it. I did fix a couple of bugs though, but I don't think those fix this hang: http://hg.dovecot.org/dovecot-2.0/rev/4d9768fd1a55 http://hg.dovecot.org/dovecot-2.0/rev/c7b351d415d9 Yeah, you're right, they don't fix this hang. Were you at least able to duplicate the problem by playing back a similar series of commands? Is there anything else I can do to help isolate the problem? ~Kyle -- It is precisely because it is fashionable for Americans to know no science, even though they may be well educated otherwise, that they so easily fall prey to nonsense. -- Isaac Asimov pgppEOjhXTLin.pgp Description: PGP signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Wednesday, August 25 at 03:34 PM, quoth Kyle Wheeler: On Tuesday, August 24 at 12:17 AM, quoth Timo Sirainen: On 23.8.2010, at 23.37, Kyle Wheeler wrote: In that trace Dovecot has sent reply to everything mutt has asked for. Although it looks like mutt has skipped logging some of the commands it has sent (LIST commands, e.g. a0002 or a0042). Perhaps it sent something else that Dovecot didn't reply to yet? .. Get a log with Dovecot's rawlog: http://wiki2.dovecot.org/Debugging/Rawlog I'm attaching the output of the rawlog. Could you do once more with -bt options? Attached. It looks like the rest of the commands in the same packet after mutt told Dovecot that it was DONE with IDLE were ignored. Has this been dropped? ~Kyle -- Every American expects and deserves clean air, and then we act on that belief, then we will set an example for the rest of the world to follow. -- George H. W. Bush pgp70km3JptZa.pgp Description: PGP signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Tuesday, August 24 at 12:17 AM, quoth Timo Sirainen: On 23.8.2010, at 23.37, Kyle Wheeler wrote: In that trace Dovecot has sent reply to everything mutt has asked for. Although it looks like mutt has skipped logging some of the commands it has sent (LIST commands, e.g. a0002 or a0042). Perhaps it sent something else that Dovecot didn't reply to yet? .. Get a log with Dovecot's rawlog: http://wiki2.dovecot.org/Debugging/Rawlog I'm attaching the output of the rawlog. Could you do once more with -bt options? Attached. It looks like the rest of the commands in the same packet after mutt told Dovecot that it was DONE with IDLE were ignored. ~Kyle -- When did 'There but for the grace of God, go I' get replaced with 'It sucks to be you'? And what can you and I do about it? -- Billy Payne dovecot.rawlog.tar.bz2 Description: application/tar-bz2 signature.asc Description: Digital signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Friday, August 20 at 10:25 PM, quoth Timo Sirainen: On 20.8.2010, at 22.20, Kyle Wheeler wrote: On Friday, August 20 at 07:58 PM, quoth Timo Sirainen: Did you try this with IMAP client or via talking IMAP directly? I did it with mutt as my IMAP client. I'm attaching a log of the IMAP conversation. The log ends at the point that I had to kill mutt to regain control. It looks like mutt is stuck waiting for Dovecot to reply a0049 OK Idle completed. In that trace Dovecot has sent reply to everything mutt has asked for. Although it looks like mutt has skipped logging some of the commands it has sent (LIST commands, e.g. a0002 or a0042). Perhaps it sent something else that Dovecot didn't reply to yet? .. Get a log with Dovecot's rawlog: http://wiki2.dovecot.org/Debugging/Rawlog I'm attaching the output of the rawlog. ~Kyle -- No people can be great who have ceased to be virtuous. -- Samuel Johnson, on the behavior of the British colonists in America dovecot.rawlog.tar.bz2 Description: application/tar-bz2 signature.asc Description: Digital signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Friday, August 20 at 08:13 PM, quoth Timo Sirainen: On Thu, 2010-08-19 at 13:12 -0500, Kyle Wheeler wrote: On Thursday, August 19 at 07:05 PM, quoth Timo Sirainen: I have log_path set to /dev/stderr, and no syslog_facility setting, so... what could be going wrong? I've never tried logging to /dev/stderr with v2.0. I guess it should be possible to fix it.. :) Fixed: http://hg.dovecot.org/dovecot-2.0/rev/d7a3abc9e0e0 Thanks! Works like a charm. ~Kyle -- When the people fear the government, there is tyranny; when the government fears the people, there is liberty. -- Thomas Jefferson signature.asc Description: Digital signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Friday, August 20 at 07:58 PM, quoth Timo Sirainen: Did you try this with IMAP client or via talking IMAP directly? I did it with mutt as my IMAP client. I'm attaching a log of the IMAP conversation. The log ends at the point that I had to kill mutt to regain control. It looks like mutt is stuck waiting for Dovecot to reply a0049 OK Idle completed. Is it really relevant that it's being done across namespaces? I just tried it within the same namespace: no, it's not relevant. If it seems to be, try also within a namespace but with maildir_copy_with_hardlinks=no? I just tried it with maildir_copy_with_hardlinks=no; I got the exact same behavior. ~Kyle -- To invent, you need a good imagination and a pile of junk. -- Thomas Jefferson 5 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN AUTH=LOGIN] There was suppose to be an earth-shattering KA-BOOM!!! 5 a AUTHENTICATE LOGIN 5 + VXNlcm5hbWU6 5 5 + UGFzc3dvcmQ6 5 5 a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in 5 a0001 CAPABILITY 5 * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS 5 a0001 OK Capability completed. 5 * LIST (\Noselect) . 5 a0002 OK List completed. 5 a0003 STATUS INBOX.Drafts (MESSAGES) 5 * STATUS INBOX.Drafts (MESSAGES 0) 5 a0003 OK Status completed. 5 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk JunkRecorded $Junk $Forwarded NotJunk Old $MDNSent) 5 * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk JunkRecorded $Junk $Forwarded NotJunk Old $MDNSent \*)] Flags permitted. 5 * 40 EXISTS 5 * 0 RECENT 5 * OK [UIDVALIDITY 1173499967] UIDs valid 5 * OK [UIDNEXT 24152] Predicted next UID 5 * OK [HIGHESTMODSEQ 2994] Highest 5 a0004 OK [READ-WRITE] Select completed. 5 a0005 UID FETCH 1:24151 (UID FLAGS) 5 * 1 FETCH (UID 14477 FLAGS (\Seen)) 5 * 2 FETCH (UID 18291 FLAGS (\Seen)) 5 * 3 FETCH (UID 18332 FLAGS (\Seen)) 5 * 4 FETCH (UID 18410 FLAGS (\Answered \Seen)) 5 * 5 FETCH (UID 18490 FLAGS (\Seen)) 5 * 6 FETCH (UID 18676 FLAGS (\Answered \Seen)) 5 * 7 FETCH (UID 18688 FLAGS (\Answered \Seen)) 5 * 8 FETCH (UID 19810 FLAGS (\Seen)) 5 * 9 FETCH (UID 19950 FLAGS (\Seen)) 5 * 10 FETCH (UID 19971 FLAGS (\Seen)) 5 * 11 FETCH (UID 19973 FLAGS (\Seen)) 5 * 12 FETCH (UID 20207 FLAGS (\Seen)) 5 * 13 FETCH (UID 20362 FLAGS (\Seen)) 5 * 14 FETCH (UID 20895 FLAGS (\Seen)) 5 * 15 FETCH (UID 22321 FLAGS (\Seen $Forwarded)) 5 * 16 FETCH (UID 23330 FLAGS (\Seen)) 5 * 17 FETCH (UID 23652 FLAGS (\Answered \Seen)) 5 * 18 FETCH (UID 23669 FLAGS (\Seen)) 5 * 19 FETCH (UID 23672 FLAGS (\Seen)) 5 * 20 FETCH (UID 23673 FLAGS (\Answered \Seen)) 5 * 21 FETCH (UID 23675 FLAGS (\Seen)) 5 * 22 FETCH (UID 23708 FLAGS (\Seen)) 5 * 23 FETCH (UID 23709 FLAGS (\Seen)) 5 * 24 FETCH (UID 23712 FLAGS (\Seen)) 5 * 25 FETCH (UID 23714 FLAGS (\Seen)) 5 * 26 FETCH (UID 23715 FLAGS (\Seen)) 5 * 27 FETCH (UID 23722 FLAGS (\Seen)) 5 * 28 FETCH (UID 23746 FLAGS (\Seen)) 5 * 29 FETCH (UID 23749 FLAGS (\Seen)) 5 * 30 FETCH (UID 23762 FLAGS (\Seen)) 5 * 31 FETCH (UID 23764 FLAGS (\Seen)) 5 * 32 FETCH (UID 23818 FLAGS (\Seen)) 5 * 33 FETCH (UID 23827 FLAGS (\Seen)) 5 * 34 FETCH (UID 23946 FLAGS (\Seen)) 5 * 35 FETCH (UID 23979 FLAGS (\Seen)) 5 * 36 FETCH (UID 24063 FLAGS (\Seen)) 5 * 37 FETCH (UID 24114 FLAGS (\Seen)) 5 * 38 FETCH (UID 24134 FLAGS (\Answered \Seen)) 5 * 39 FETCH (UID 24149 FLAGS (\Seen)) 5 * 40 FETCH (UID 24150 FLAGS (\Seen)) 5 a0006 STATUS INBOX.Family (UIDNEXT UIDVALIDITY UNSEEN RECENT) 5 a0005 OK Fetch completed. 5 * STATUS INBOX.Family (RECENT 0 UIDNEXT 6183 UIDVALIDITY 1173549357 UNSEEN 0) 5 a0006 OK Status completed. 5 * STATUS INBOX.Friends.WeBeSmart (RECENT 0 UIDNEXT 1519 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0007 OK Status completed. 5 * STATUS INBOX.Subscribed.Debian.Security (RECENT 0 UIDNEXT 927 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0008 OK Status completed. 5 * STATUS INBOX.Subscribed.Apple Security (RECENT 0 UIDNEXT 221 UIDVALIDITY 1173549362 UNSEEN 0) 5 a0009 OK Status completed. 5 * STATUS INBOX.Logs.Mail (RECENT 0 UIDNEXT 9025 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0010 OK Status completed. 5 * STATUS INBOX.Logs.Salinan (RECENT 0 UIDNEXT 5 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0011 OK Status completed. 5 * STATUS INBOX.Logs.Me (RECENT 0 UIDNEXT 1 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0012 OK Status completed. 5 * STATUS INBOX.Logs.WOPR (RECENT 0 UIDNEXT 3208 UIDVALIDITY 1173549210 UNSEEN 0) 5 a0013 OK Status completed. 5 * STATUS
[Dovecot] 2.0 migration weirdnesses: logs and hang
Hello,
I'm testing out an upgrade to dovecot 2.0 from 1.2.11, and I've
stumbled across two weirdnesses that I need help with.
First, I'm only getting one log message:
master: Info: Dovecot v2.0.0 starting up
After that, while I can connect, log in, read mail, etc., no further
log messages are created.
I have log_path set to /dev/stderr, and no syslog_facility setting,
so... what could be going wrong?
Second, when I create a new mailbox and copy messages across
namespaces, dovecot hangs. The mailbox is created and the messages are
copied successfully, but dovecot simply stops responding. I can't even
logout!
I'm attaching my config file, if it helps.
~Kyle
--
The effect of liberty to individuals is, that they may do what they
please; we ought to see what it will please them to do, before we risk
congratulations.
-- Edmund Burke
# 2.0.0: ./dovecot.conf-testing-migrate
# OS: Linux 2.6.26-2-686 i686 Debian 5.0.5
auth_default_realm = memoryhole.net
auth_mechanisms = plain login
base_dir = /var/run/dovecot/memoryhole2
first_valid_gid = 64020
first_valid_uid = 64020
last_valid_gid = 64020
last_valid_uid = 64020
log_path = /dev/stderr
info_log_path = /dev/stderr
log_timestamp =
login_greeting = There was suppose to be an earth-shattering KA-BOOM!!!
mail_gid = 64020
mail_location = maildir:%h/Maildir
mail_uid = 64020
mbox_write_locks = fcntl
namespace {
hidden = no
inbox = yes
location = maildir:~/Maildir
prefix =
separator = .
type = private
}
namespace {
hidden = yes
inbox = no
location =
mbox:~/Maildir/Archive:LAYOUT=maildir++:INDEX=~/Maildir/ArchiveIndexes/
prefix = INBOX.Archive.
separator = .
type = private
}
passdb {
args = /var/lib/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_squat = partial=4 full=4
}
protocols = imap
service auth {
unix_listener auth-master {
group = vchkpw
mode = 0600
user = vpopmail
}
user = vpopmail
}
service imap-login {
inet_listener imap {
address = imap.memoryhole.net
port = 143
}
inet_listener imaps {
address = imap.memoryhole.net
port = 993
}
user = dovecot
}
service imap {
executable = /tmp/dovecottesting/libexec/dovecot/imap
}
service pop3-login {
user = dovecot
}
ssl_cert = /etc/ssl/certs/imap.memoryhole.net.pem
ssl_key = /etc/ssl/private/imap.memoryhole.net.key
userdb {
args = uid=64020 gid=64020 home=/var/lib/vpopmail/domains/%Ld/%Ln
allow_all_users=yes
driver = static
}
valid_chroot_dirs = /var/lib/vpopmail/domains
verbose_proctitle = yes
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep delay-newmail
imap_logout_format = writebytes=%o, readbytes=%i
mail_plugins = fts fts_squat zlib
}
protocol lda {
auth_socket_path = /var/run/dovecot/memoryhole2/auth-master
hostname = memoryhole.net
mail_plugins = fts fts_squat
postmaster_address = postmas...@memoryhole.net
}
signature.asc
Description: Digital signature
Re: [Dovecot] 2.0 migration weirdnesses: logs and hang
On Thursday, August 19 at 07:05 PM, quoth Timo Sirainen:
I have log_path set to /dev/stderr, and no syslog_facility setting,
so... what could be going wrong?
I've never tried logging to /dev/stderr with v2.0. I guess it should be
possible to fix it..
:)
Second, when I create a new mailbox and copy messages across
namespaces, dovecot hangs. The mailbox is created and the messages
are copied successfully, but dovecot simply stops responding. I
can't even logout!
So you can easily reproduce this? Could you get gdb backtrace from the
hanging process?
gdb -p pid of hanging imap process
bt full
Sure thing. By the way, the proctitle is
dovecot/imap [k...@memoryhole.net 64.253.106.173 CLOSE UID COPY]
Here's the backtrace:
#0 0xb7fab424 in __kernel_vsyscall ()
No symbol table info available.
#1 0xb7defff8 in epoll_wait () from /lib/i686/cmov/libc.so.6
No symbol table info available.
#2 0xb7ec51f9 in io_loop_handler_run (ioloop=0x806d360) at ioloop-epoll.c:179
ctx = (struct ioloop_handler_context *) 0x806d480
event = value optimized out
list = value optimized out
io = value optimized out
tv = {tv_sec = 1794, tv_usec = 888651}
t_id = value optimized out
msecs = 1794889
ret = 0
i = value optimized out
j = value optimized out
call = value optimized out
#3 0xb7ec4250 in io_loop_run (ioloop=0x806d360) at ioloop.c:350
No locals.
#4 0xb7eb172a in master_service_run (service=0x806d2b0, callback=0x8060040
client_connected) at master-service.c:496
No locals.
#5 0x0805ff95 in main (argc=Cannot access memory at address 0x0
) at main.c:358
service_flags = value optimized out
storage_service_flags = value optimized out
postlogin_socket_path = 0x0
username = 0x0
c = value optimized out
set_roots = {0x8062c80, 0x0}
I can recompile without optimization, if that would help (I just used
the default -O2).
~Kyle
--
The future is here. It's just not widely distributed yet.
-- William Gibson
signature.asc
Description: Digital signature
Re: [Dovecot] Urgent problem with deleting emails and maildirsize
On Thursday, July 2 at 05:28 PM, quoth Jose Luis Marin Perez: When calculating the quota through maildirsize did not consider the emails with flag T. What's the point of having a quota if users can circumvent it by simply labeling their messages as deleted? In all truth, if it weren't for the expunge command, the \Deleted label would be no different from any other label. And from that perspective, what you're asking for is essentially equivalent to a \DoesntCount tag, so that users can circumvent the quota by giving their messages a tag that prevents them from counting towards their quota limit. The messages *aren't* deleted; they can still be read, copied, forwarded. What is to prevent a user from storing several gigabytes worth of deleted messages on your server? From that perspective, this is a security problem: any user can attack your server by creating a denial-of-service condition. They can simply store a huge number of deleted messages, occupying so much disk space that no other users can receive mail. Normally, a quota mechanism is supposed to protect you from this type of attack, but you're explicitly asking for a way for users to easily avoid the quota restrictions. Dovecot is working perfectly, the problem is that as Courier had this feature, That's not a feature, that's a security bug. users will not purge mails So what's the point of having a quota? ~Kyle -- Those who profess to favor freedom, and yet depreciate agitation, are men who want rain without thunder and lightning. -- Frederick Douglass pgpfmI4n7SN52.pgp Description: PGP signature
Re: [Dovecot] Users with large (4GB) inboxes crippling dovecot
On Friday, May 29 at 09:46 AM, quoth Curtis Maloney: This is certainly one advantage dbox and maildir have -- not being limited to the FS file size limit per folder. That's not *entirely* accurate. Certainly no single message can exceed the 2GB limit even with maildir, and the other issue that begins to come up is the impact/effect of large numbers of files. Depending on the filesystem (I'm assuming ext2?), there's probably a hard limit on the number of files per directory, and almost certainly there's a big performance penalty for that many files. To get good performance with Maildir and really large folders, you need a filesystem that can handle large numbersof files. Ext3 has directory hashing, ReiserFS is good... I believe XFS and several others have tackled the problem as well (I don't know about FFS). That said... eGADS - a real life FC4 in the wild?!?! According to fedoraproject.org: For 20030101-20050607 there are a potential 863 CVE named vulnerabilities that could have affected FC4 packages. 759 (88%) of those are fixed because FC4 includes an upstream version that includes a fix, 10 (1%) are still outstanding, and 94 (11%) are fixed with a backported patch. That would make me a little nervous that's just the issues over the course of two years, ending in 2005 (FOUR years ago). ~Kyle -- I would therefore like to posit that computingâs central challenge, how not to make a mess of it, has not yet been met. -- Edsger Dijkstra signature.asc Description: Digital signature
Re: [Dovecot] [bug] dovecot 1.1.15: segfault after message move
On Monday, May 25 at 10:27 AM, quoth Pascal Volk: On 05/23/2009 06:03 AM Kyle Wheeler wrote: Interesting. I recently upgraded, and I get the same thing - but I use Maildir. Just a question: How big is the Maildir in MB and messages? It seems to happen on all sizes. It's happened on my INBOX (30 messages, 164K) and on larger mailboxes (540 messages, 508K). ~Kyle -- If an elderly respected expert in a given field tells you that something can be done he is almost certainly right. If an elderly respected expert in a given field tells you that something is impossible, he is almost certainly wrong. -- Robert A. Heinlein signature.asc Description: Digital signature
Re: [Dovecot] [bug] dovecot 1.1.15: segfault after message move
On Friday, May 22 at 01:49 PM, quoth Juergen Daubert: On Wed, May 20, 2009 at 01:47:42PM +0200, Juergen Daubert wrote: found the following in my error log: May 20 13:27:48 ser dovecot: imap-login: Login: user=juergen, method=PLAIN, rip=192.168.0.17, lip=192.168.0.90, TLS May 20 13:28:10 ser dovecot: Panic: IMAP(juergen): file imap-sync.c: line 439 (cmd_sync_delayed): assertion failed: (client-mailbox != NULL) May 20 13:28:10 ser dovecot: IMAP(juergen): Raw backtrace: imap [0x80cc01e] - imap [0x80cc08a] - imap [0x80cba78] - imap [0x806642f] - imap [0x80602c1] May 20 13:28:10 ser dovecot: child 23536 (imap) killed with signal 6 (core dumps disabled) it's almost always reproducible using the Heirloom mailx [1] mail client, with mutt I get a 'connection closed' message but no segfault: - login to the dovecot server via imap/imaps - move a message from INBOX to a another large mbox-file - quit Seems to be a new issue introduced with 1.1.15 because I don't see that with 1.1.14 or older versions. Interesting. I recently upgraded, and I get the same thing - but I use Maildir. Here's my error: 2009-05-22 17:42:35.797527500 imap-memoryhole.net: dovecot: Panic: IMAP(k...@memoryhole.net): file imap-sync.c: line 439 (cmd_sync_delayed): assertion failed: (client-mailbox != NULL) 2009-05-22 17:42:35.797844500 imap-memoryhole.net: dovecot: Error: IMAP(k...@memoryhole.net): Raw backtrace: imap [0x80d3e80] - imap [0x80d3eda] - imap [0x80d378a] - imap(cmd_sync_delayed+0x292) [0x8066d62] - imap [0x80609a7] - imap(client_continue_pending_input+0x86) [0x8060626] - imap [0x805c097] - imap(io_loop_handler_run+0x110) [0x80dc0d0] - imap(io_loop_run+0x28) [0x80daf18] - imap(main+0x4b1) [0x8068671] - /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7e9bea8] - imap [0x805a271] 2009-05-22 17:42:35.817635500 imap-memoryhole.net: dovecot: Error: child 28270 (imap) killed with signal 6 (core dumped) I've done some more tests on that issue and found that I can fix it if I revert commit http://hg.dovecot.org/dovecot-1.1/rev/78ab57f321c8. Cool! ~Kyle -- The whole art of government consists in the art of being honest. -- Thomas Jefferson: Rights of British America, 1774 pgpcfojnqqkXM.pgp Description: PGP signature
[Dovecot] Mailbox is locked?
Hello, I recently upgraded from Dovecot 1.1.8 to 1.1.14, and I've started to get errors I never got before. Specifically, I'm getting Mailbox is locked, will abort in xx seconds errors. My mail client (mutt) will just sit there in the background (presumably in IDLE) and randomly will show this error (I THINK it has something to do with delivering new messages to the folder; I use the dovecot LDA). Nothing else about my configuration has changed, just the version of dovecot. Does anyone know what could be causing these? I assume it has something to do with the index files, but... Nothing interesting shows up in the log files. How do I get rid of these error messages? Or even begin tracking them down? For what it's worth, mail is stored on the server in Maildirs, and the lock_method is fcntl, the mbox_write_locks is fcntl (I don't set mbox_read_locks). ~Kyle -- For to accuse, requires less eloquence, such is man's nature, than to excuse; and condemnation, than absolution more resembles justice. -- Hobbes, Leviathan pgpzcdBUg3kUG.pgp Description: PGP signature
Re: [Dovecot] Mailbox is locked?
On Thursday, April 23 at 01:18 AM, quoth Timo Sirainen: Do you actually notice something being broken/hanging or is it just that it gives those messages? Nope - Dovecot's behavior is otherwise exemplary. Do the xx seconds ever reach below 119? I haven't been keeping track, but I generally only see them once at a time, with relatively high timeout values. I added those notifications only recently and I guess it's possible that sometimes it gives them way too early (and actually I remember seeing one myself once). Hmm. Yeah, looks like the code doesn't really even try to prevent that. I'll add some check to make it hide them until at least 15 seconds has passed. Cool! ~Kyle -- If Jack Valenti had been around at the time of Gutenberg he would have organized the monks to come and burn down the printing press. -- ITAA president Harris Miller pgppBRehxrNzg.pgp Description: PGP signature
Re: [Dovecot] Calling dovecot deliver from procmail
On Wednesday, February 25 at 09:00 AM, quoth Harry Lachanas: I came up with this trick That is Since my previous mail server had a bunch of complex procmail recipies and I am not sure that I can turn them into sieve scripts I am calling dovecot deliver to drop mail in it's final $DEFAULT destination. :0 | /some/path/dovecot/deliver in order to update the index, control files etc upon delivery time. That's perfectly reasonable; I do it on my server. I even use that deliver program to deliver to non-default mailboxes as well. ~Kyle -- Genius may have its limitations but stupidity is not thus handicapped. -- Elbert Hubbard pgpdes0tksiuz.pgp Description: PGP signature
Re: [Dovecot] qmail and LDA: passdb didn't return userdb entries
On Tuesday, December 2 at 03:35 PM, quoth Alessio Cecchi:
i'm testing Dovecot LDA to works with qmail and vpopmail. I have
added into a .qmail for a users this line:
[EMAIL PROTECTED]: /home/vpopmail/domains/test.com/0/test# cat .qmail
| /var/qmail/bin/preline -f /usr/libexec/dovecot/deliver -d [EMAIL PROTECTED]
The easiest way to do this is to follow the instructions on the wiki
(http://wiki.dovecot.org/LDA). Dovecot's `deliver` needs to know two
things: the $HOME and the FROM_ENVELOPE. Why does it need the latter?
I haven't a clue. But this should work for you:
| /var/qmail/bin/preline -f /usr/libexec/dovecot/deliver -f ${SENDER:-}
Using the -d flag is a bad idea unless your dovecot is set up with a
passdb and userdb.
The problem for you with instituting a static userdb is that you've
enabled user hashing in vpopmail. On my system, I can use this:
userdb static {
args = uid=XXX gid=XXX home=/var/lib/vpopmail/domains/%Ld/%Ln
allow_all_users=yes
}
But that won't work for you because the location of the home directory
isn't so simple on your system. I don't know how to tell Dovecot how
to hash user directories the way that vpopmail does.
I have read in the wiki that deliver needed to know some much
information about users, like HOMEDIR, but this information are not
returned by vpopmail authentication?
It also needs the -f flag (for unknown reasons) AND you need to avoid
the -d flag (the -d flag tells deliver to ignore the $HOME environment
variable and to attempt to look up the home directory in the userdb
which, as you've noticed, can be tricky).
~Kyle
--
Moral indignation is jealousy with a halo.
-- H. G. Wells
pgpva7EsGdhzB.pgp
Description: PGP signature
Re: [Dovecot] v1.1.6 released
On Wednesday, November 19 at 10:56 AM, quoth Adam McDougall: Just wanted to mention that 1.1.6 seems fine so far in our testing, and I think the lack of reported problems on the mailing list is probably a very good sign! For whatever reason, we ran into the userdb didn't return a home directory problem with 1.1.6, and quickly downgraded back to 1.1.5. http://thread.gmane.org/gmane.mail.imap.dovecot/34008/focus=34009 It's rather silly too, since the userdb *does* return a home directory (which is why I'm skeptical of the fix mentioned there that forces a default home directory of /tmp). Here's hoping 1.1.7 (whenever it comes out) is a smoother upgrade. ~Kyle -- History will be kind to me, for I intend to write it. -- Winston Churchill pgpCoEIFBXAaL.pgp Description: PGP signature
Re: [Dovecot] Mailbox Hashing
On Friday, November 14 at 05:30 AM, quoth Charles Marcus: On 11/13/2008, Kyle Wheeler ([EMAIL PROTECTED]) wrote: (ReiserFS is often viewed as a purely experimental filesystem, and not reliable for production systems) Please stop spreading FUD. shrug I'm not saying that's *true*, I'm just saying I've heard that a lot... It's entirely possible that ReiserFS is just as reliable as any other filesystem *now* but someone pushed it into the mainline Linux kernel before it was ready, thereby biting the early adopters with bugs that hadn't been worked out yet and creating the impression that it isn't very stable. ~Kyle -- Unthinking respect for authority is the greatest enemy of truth. -- Albert Einstein pgpsppJdKgVt8.pgp Description: PGP signature
Re: [Dovecot] Mailbox Hashing
On Friday, November 14 at 11:51 AM, quoth Charles Marcus: shrug I'm not saying that's *true*, I'm just saying I've heard that a lot... Thats called spreading FUD. No, it's not. FUD would be a strategic attempt to influence public opinion by disseminating negative (and vague) information. I am not trying to influence public opinion, I'm reporting existing public opinion. The consensus opinion of the sysadmins I trust most highly is that ReiserFS is still relatively experimental and has not yet earned their trust---several of them have been bitten by ReiserFS bugs on their development machines (read: data loss due to unrecoverable filesystem corruption). That said, their problems were several years ago. Unfortunately, in the world of filesystem reliability, trust comes slowly once lost (check out how recently ReiserFS has been fixing quota-related problems, including ACL deadlocks). In any case, I have no strategic purpose here. I have no interest or stake in any filesystem taking over the world. If ReiserFS is extremely stable and extremely reliable, then that's awesome, but it does have a bit of a reputation problem. Denying that it has a negative reputation, or claiming that anyone who describes its reputation is spreading FUD, is not only pointless but also counter productive. If you want to say well, that may be what you've heard, but I've used ReiserFS on several large, heavily-used, mission-critical systems for several years and have not had any problems, then that would be a useful and important statement. You'd even be helping ReiserFS's reputation. But by having such a knee-jerk reaction to the fact that it's got a negative reputation, you're making the filesystem seem like it's used largely by proselytizers and zealots---which is not a good way to build ReiserFS's reputation. I've heard plenty of horror stories about ext2/ext3, xfs, etc ALL losing data... Of course - any filesystem can loose data in bad situations (such as power loss, bad disks, etc.). Ext3 is certainly not perfect for all situations. For example, it's a bad idea on flash media because it keeps its journal file in a fixed spot on the drive, which can wear out that part of a flash drive quickly. The real question is: what are the reputations of those filesystems, and why? Ext2/3 have been around for a very long time, and are extremely well-tested by virtue of their popularity, and as such tend to be more trusted for mission-critical systems (unless there's a reason they shouldn't be used). The fact is, I've been using reiserfs on numerous boxes for many years with ZERO problems. Excellent! What kind of systems are we talking about? How heavily loaded? Did you use it with LVM? Did you ever have to use the recovery tools? How well did they work? ~Kyle -- The next best thing to solving a problem is finding some humor in it. -- Frank A. Clark pgpEn3WGrrFWb.pgp Description: PGP signature
Re: [Dovecot] Mailbox Hashing
On Thursday, November 13 at 05:20 PM, quoth Justin Krejci: Is there any method for hashing the inbox automatically after say 5,000 messages are stored? Example $Maildir/in/0/message0 $Maildir/in/0/message1 $Maildir/in/0/message2 Not in Maildir. The Maildir format does not allow that, so... It may be possible to do with something like dbox, since that's a Dovecot-specific format. In general, though, that kind of hashing is usually a workaround for a lousy filesystem (such as ext2), rather than something you'd really *want* to do. The one exception might be if you want to split someone's inbox over several filesystems, but even that could be accomplished using something like UnionFS. Of course, we're getting outside the realm of production-tested options here, and it would probably introduce all kinds of potential problems with locking and such. I am not currently using Dovecot but am interested to know if this is available or does running with 20,000+ messages in a single inbox not affect the performance much? It all depends on the filesystem and what operations you're doing. Dovecot does a *lot* of caching to avoid hitting the filesystem whenever it can. However, randomly accessing messages in your mailbox *will* cause a filesystem access, and the speed of that depends on having a halfway decent filesystem. I have looked into other file system tuning techniques such as enabling ext3 dir_index or using ReiserFS (maybe not ReiserFS anymore). There will likely be 15,000 to 20,000 accounts spread out on one or more servers using a 6-drive RAID10 setup. Most accounts are not expected to have high message quantities but there will be lots of concurrent connections via pop and imap (and webmail imap). You should be fine. I'd probably encourage something more stable like ext3 with dir_index (ReiserFS is often viewed as a purely experimental filesystem, and not reliable for production systems). The ext3 documentation suggests that 100k-1M+ files in a single directory should not pose a significant performance problem when using dir_index. I haven't tried it with directories that are *that* big, but I regularly use mailboxes with over 5k messages without problems. ~Kyle -- A woman is like a tea bag. It's only when she's in hot water that you realize how strong she is. -- either Eleanor Roosevelt or Carl Sandberg pgpaOMtmeGqlY.pgp Description: PGP signature
Re: [Dovecot] mbox to Maildir conversion
I have been thinking about converting also. Will the standard auto detect routines work with both types during the conversion, or will I need to deal with namespaces? The standard auto-detect routines will work well; I recommend also using the convert plugin. ~Kyle -- Victory goes to the player who makes the next-to-last mistake. -- Chessmaster Savielly Gricorievitch Tatrtak pgpWLLTUdgpyv.pgp Description: PGP signature
Re: [Dovecot] mbox to Maildir conversion
Thanks Kyle, the Procmailrc script I am using is as follows: ... I believe that this matches your correct example. That is just where Procmail places them. Huh, well, indeed it does match my example. The only other reason I can think of for Dovecot to not see new mail that gets delivered properly is if the permissions on that mail are wrong. (Mail that it cannot read essentially does not exist.) Make sure you're delivering mail as the right user. ~Kyle -- Every gun that is made, every warship launched, every rocket fired signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed. -- Dwight D. Eisenhower pgpxA8rLs6tgs.pgp Description: PGP signature
Re: [Dovecot] mbox to Maildir conversion
On Wednesday, October 22 at 11:35 PM, quoth Albert E. Whale: I am currently testing a single user, and have successfully converted the mail messages from mbox to Maildir format, and now I am setting up the procmail tool to place the messages into the correct folder. I have been following the http://wiki.dovecot.org/Migration/MailFormat formula, but now have a question. New Messages are now being placed into the ~user/Maildir/new folder. Procmail understands Maildir natively; you don't have to tell it to put messages into the new folder, you should just tell it to put things into the Maildir folder and end the line with a /, like so: # correct! :0 ~user/Maildir/ # incorrect :0 ~user/Maildir/new/ # also incorrect :0 ~user/Maildir/new If you specify the new directory, you're either telling procmail to treat that directory like an MH folder, which is wrong, or you're telling procmail that the new directory is a Maildir, in which case it will create another new directory within that directory (along with a tmp and cur directories). ~Kyle -- As an adolescent I aspired to lasting fame, I craved factual certainty, and I thirsted for a meaningful vision of human life. So I became a scientist. This is like becoming an archbishop so you can meet girls. -- M. Cartmill pgpSBFWoHw1df.pgp Description: PGP signature
Re: [Dovecot] Support of client specific flags?
On Wednesday, September 10 at 07:50 PM, quoth Steinar Bang: Gnus use a lot of custom flags to represent the messages state wrt. Gnus. How well are these flags supported in dovecot? How fast are access to them? (Ie. will dovecot have to open the message and parse the headers) The flag information is stored in the index, and for Maildir backends the first 26 flags are stored in the filename as well. In other words, access to flags is pretty dang fast, and for Maildir storage, never requires opening the message or parsing headers. When you use mbox, though, the keywords are stored in the message's headers (X-IMAPbase), so parsing the headers is sometimes necessary. HOWEVER, Dovecot adaptively chooses what message information to keep in the index and cache. If a user is using Gnus, those keywords will end up in the mailbox cache, and generally won't need to be read from the message file. Large numbers of keywords (esp. more than 26), though, hasn't been tested a whole lot, but should be reasonably fast (largely because of the index and cache files). Timo talked about this a few months ago and if I remember right he indicated that there were some inefficiencies in dealing with really large numbers of keywords, but since they're so rarely used in any significant volume, improving that component was a low priority. ~Kyle -- The optimist thinks this is the best of all possible worlds. The pessimist fears it is true. -- J. Robert Oppenheimer pgp5RuGQgKL3f.pgp Description: PGP signature
Re: [Dovecot] Question about subfolders
On Wednesday, September 10 at 02:58 PM, quoth Oliver Fromme:
I've got an old dovecot installation (0.99.10.5)
Yikes! Talk about ancient! :)
Now I need to decide whether to update it to a newer version of
dovecot or to switch to a different software.
My understanding is that Dovecot has changed so much since then that
it's almost like switching to an entirely new piece of software. But,
IMHO, it's the best IMAP server out there.
One feature that I really need is to be able to place messages and
subfolders in a folder at the same time. This is with mbox format
files (I can't use maildir). The old version of dovecot doesn't seem
to support that, you can have either subfolders or messages in a
folder, but not both.
Does the current version 1.1.3 support it?
Absolutely!
The reason that it wasn't supported before is that Dovecot used (and
still does use by default) a layout technique it calls FS for mbox
mail hierarchies. In essence a mailbox on disk could either be a file
(an mbox capable of storing messages, but incapable of storing
subfolders) or it could be a folder (capable of storing subfolders,
but incapable of storing messages). Like I said, that's still the
default for mbox mail storage. HOWEVER, you can tell it to use
Maildir++-style layout.
For example, imagine you have the following IMAP mailboxes:
INBOX
INBOX.Archive.Family
INBOX.Family
Trash
Using the (old) FS layout, they would be arranged something like this:
INBOX - /var/mail/username
INBOX.Archive.Family - ~/Mail/INBOX/Archive/Family
INBOX.Family - ~/Mail/INBOX/Family
Trash - ~/Mail/Trash
Using the maildir++ layout, they would be arranged like this:
INBOX - /var/mail/username
INBOX.Archive.Family - ~/Mail/.INBOX.Archive.Family
INBOX.Family - ~/Mail/.INBOX.Family
Trash - ~/Mail/.Trash
The config line for maildir++ layout as I just described would be:
mail_location = mbox:%h/mail:LAYOUT=maildir++:INBOX=/var/mail/%u
Now, that makes the index files hard to place (they'd look like
mailboxes), so you probably want to put them someplace else
explicitly, like so (all on one line):
mail_location =
mbox:%h/mail:LAYOUT=maildir++:INBOX=/var/mail/%u:INDEX=%h/indexes
(Pick your own locations, of course.)
The issue for you, I think, is going to be converting your current FS
layout into maildir++ layout. Thankfully, that's trivial to do with
Dovecot's convert plugin. Just use something like so (in combination
with the previous mail_location line):
protocol imap {
mail_plugins = convert
}
plugin {
convert_mail = mbox:%h/mail:LAYOUT=fs
}
Check out http://wiki.dovecot.org/Plugins/Convert for details.
~Kyle
--
No one will ever win the battle of the sexes; there's too much
fraternizing with the enemy.
-- Henry Kissinger
pgpOViHTGoB63.pgp
Description: PGP signature
Re: [Dovecot] filename format question
On Thursday, September 4 at 06:51 PM, quoth Giorgenes Gelatti: Is it possible to configure the filename format in dovecot? For example, to change from unique,W=size:2,FLAGS to unique,size.hostname:2,FLAGS_unique2? There's no config option for it, but it's theoretically possible to do just about anything. Generally, though, the format of those filenames is defined by the Maildir standard. Changing the filename as you suggest would likely break other mail programs that might access those Maildir folders. I suppose the question is: why would you want to change that? ~Kyle -- It is better to dwell in the wilderness than with a contentious and an angry woman. -- Proverbs 21:19 pgp9LUZYYAhRX.pgp Description: PGP signature
Re: [Dovecot] Webmail app ... again.
On Thursday, August 14 at 07:01 AM, quoth Eric Toczek: While it's not free, a really nice webmail that does a lot of smart things (persistent imap connections, ldap connection pooling, and one of the best interfaces I've seen) is Nitido's PIM http://www.nitido.com/products/index.shtml?web_pim . It's used by a few of the larger US/Canadian ISPs for their webmail, as well as some big hosted email resellers. A bright group of guys too. Do they have some better screenshots or a live demo anywhere? ~Kyle -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- Brian Kernighan signature.asc Description: Digital signature
Re: [Dovecot] v1.1.2 released
On Monday, August 4 at 01:54 PM, quoth Timo Sirainen:
On Thursday, July 24 at 03:05 AM, quoth Timo Sirainen:
+ Added a new maildirlock utility for write-locking Dovecot
Maildir.
Every time I attempt to use this (as root), it exits with a return code
of 143 and my shell prints out Terminated.
I'm guessing that what happens is a result of this:
/* locked - send a */
if (kill(parent_pid, SIGTERM) 0)
i_fatal(kill(parent, SIGTERM) failed: %m);
Any idea on how I can fix it?
Not really.. The parent PID should be the parent maildirlock process,
which also should have caught the TERM signal. You could see what happens
if you change all SIGTERMs to SIGHUPs for example. What shell do you
use? Try another one?
I use bash. I tried using CSH and it does the same thing.
I tried recompiling it after replacing all instances of SIGTERM with
SIGHUP, and instead of printing out Terminated, it prints out
Hangup (i.e. the same problem).
The issue, I think, is that maildirlock has a race condition. There is
no guarantee that the parent has set up its signal handlers before the
child gets around to killing it.
To demonstrate it, I modified the source to force the child to give
the parent some time, like this:
if (pid != 0) {
io_loop_run(ioloop);
if (!success)
return 1;
printf(%s, dec2str(pid));
return 0;
} else {
sleep(1);
}
And that fixed the problem. Now, telling the child to sleep is
obviously an unacceptable solution.
I think signals are probably a bad way of handling this inter-process
communication. How about using a pipe? I've attached a patch that
seems to work.
~Kyle
--
Time is an illusion. Lunchtime doubly so.
-- Douglas Adams
--- maildirlock.c 2008-08-04 14:03:11.0 -0500
+++ maildirlock-new.c 2008-08-04 14:02:55.0 -0500
@@ -46,6 +46,7 @@ int main(int argc, const char *argv[])
struct dotlock *dotlock;
unsigned int timeout;
pid_t pid, parent_pid;
+ int trigger[2];
if (argc != 3) {
fprintf(stderr, Usage: maildirlock path timeout\n
@@ -53,6 +54,10 @@ int main(int argc, const char *argv[])
return 1;
}
parent_pid = getpid();
+ if (pipe(trigger) != 0) {
+ fprintf(stderr, pipe() failed: %m);
+ return 1;
+ }
pid = fork();
if (pid == (pid_t)-1) {
@@ -69,10 +74,11 @@ int main(int argc, const char *argv[])
lib_signals_set_handler(SIGCHLD, TRUE, sig_die, NULL);
if (pid != 0) {
- /* master - wait for the child process to finish locking */
- io_loop_run(ioloop);
- if (!success)
+ char junk;
+ if (read(trigger[0], junk, 1) != 1) {
+ fprintf(stderr, read() failed: %m);
return 1;
+ }
printf(%s, dec2str(pid));
return 0;
}
@@ -85,9 +91,9 @@ int main(int argc, const char *argv[])
if (maildir_lock(argv[1], timeout, dotlock) = 0)
return 1;
- /* locked - send a */
- if (kill(parent_pid, SIGTERM) 0)
- i_fatal(kill(parent, SIGTERM) failed: %m);
+ /* locked - send a byte */
+ write(trigger[1], , 1);
+
io_loop_run(ioloop);
file_dotlock_delete(dotlock);
pgpmui2ePV1oN.pgp
Description: PGP signature
Re: [Dovecot] RFE: Disallow DELETE of non-empty MBOX
On Monday, August 4 at 04:15 PM, quoth Timo Sirainen: On Aug 4, 2008, at 3:42 PM, Kenneth Porter wrote: It occurs to me that another possibility is to make only the Trash folder maildir, but I think Tbird only allows setting the folders can contain both messages and folders option account-wide, not per-folder. You could switch to Maildir++ layout for mboxes to get dual-use mailboxes: mail_location = mbox:~/mail:LAYOUT=maildir++ If you do that, though, you have to put your INDEXES somewhere else. Otherwise you get warnings like this: dovecot: Error: IMAP([EMAIL PROTECTED]): open() failed with file /domains/example.net/user/mail/. Trash.DeleteMe/dovecot.index.log: Not a directory It's not a directory, obviously, because .Trash.DeleteMe is the mbox file. ~Kyle -- As far as the laws of mathematics refer to reality, they are not certain, and as far as they are certain, they do not refer to reality. -- Albert Einstein pgpkoa4CU1ECL.pgp Description: PGP signature
Re: [Dovecot] Namespaces
On Tuesday, July 22 at 10:33 AM, quoth Jason Frisvold: I'm working to convert from a bincIMAP setup to a Dovecot setup. I've tried messing around with namespaces to make the conversion transparent, but I'm getting nowhere. Are namespaces necessary? Or can I proceed by recursively renaming the folders on the server? A transparent solution would be wonderful, but I can't get it to work for me.. I converted from Binc to Dovecot a while back, and I didn't use namespaces at all. Since I had been using the IMAPdir schema in Binc, I had to rename all the folders to match the Maildir++ schema, but that was a pretty simple script to write. If you're using the Maildir++ schema in Binc, then you don't have to do any renaming at all! As part of the transition process, what I did first, while I was testing out Dovecot, was to create a Maildir++ full of symlinks that pointed back to the folders in the Binc IMAPdir. That way I could test it without affecting the live server. Once I had that working, all I had to do was change my script to move the directories rather than symlinking them. The trick for me wasn't so much getting the folders into the right location as it was migrating all the other data that I didn't think about, such as the subscription lists and most importantly: the UID data. If you don't migrate the UIDs correctly, your users will probably end up re-downloading EVERY message, because their mail clients have no way of knowing that the messages haven't actually changed. ~Kyle -- No nation is permitted to live in ignorance with impunity. -- Thomas Jefferson pgpnoFJW7uOgb.pgp Description: PGP signature
Re: [Dovecot] 1.1 namespaces and message corruption
On Tuesday, July 1 at 11:15 PM, quoth Timo Sirainen: zlib plugin is somehow broken. If it's loaded then non-hardlink copying from a maildir results in zero byte output. I'll look at it later - it's not obvious why it's broken. Aha! Excellent catch - thanks Timo! ~Kyle -- Truth springs from argument amongst friends. -- David Hume pgpq9Kpcn0wSD.pgp Description: PGP signature
[Dovecot] 1.1 namespaces and message corruption
Hello,
I've been trying out the new dovecot's namespace support, and I'm
having some issues moving messages between namespaces. Specifically,
when I move some messages from a Maildir-based namespace to an
mbox-based namespace, the message gets corrupted: its contents are
removed, and an entirely empty message is added to the destination
mailbox. This is 100% repeatable (some messages just *cannot* be
copied intact).
I have two namespaces, set up like so:
namespace private { # Default
separator = .
hidden = no
inbox = yes
prefix =
location = maildir:%h/Maildir
}
namespace private {
separator = .
hidden = no
prefix = INBOX.ArchiveTest.
location = mbox:%h/Maildir/Archive
}
All I do is tell my client (mutt) to copy a message from a folder in
the maildir namespace into a folder in the mbox namespace.
Does anyone know what might be going wrong or how I can even debug
this? No error messages of any kind are showing up in the dovecot log
file, and the mail_debug setting doesn't seem to give me any useful
information.
Message corruption of this sort worries me...
~Kyle
--
To sin by silence when they should protest makes cowards of men.
-- Abraham Lincoln
pgpBqj905fwCi.pgp
Description: PGP signature
Re: [Dovecot] 1.1 and the zlib plugin
On Wednesday, June 25 at 01:24 PM, quoth Ed W: Kyle Wheeler wrote: Conveniently, there's the zlib plugin. From what I could tell from the documentation, the compressed mbox names must end in .gz, Not a solution, but I believe also now that you don't need to change the ending? I think the plugin looks at the first few bytes to determine the type of the file? I tried that... as you can see in my original email, it doesn't seem to work. I *have* been able to get the zlib plugin to work with compressed messages in Maildirs (no gz extension), but I can't seem to get it to work with compressed mboxes. :( ~Kyle -- This is my simple religion. There is no need for temples; no need for complicated philosophy. Our own brain, our own heart is our temple; the philosophy is kindness. -- Dalai Lama pgp2AmY0FjF72.pgp Description: PGP signature
Re: [Dovecot] 1.1 and the zlib plugin
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wednesday, June 25 at 12:57 PM, quoth [EMAIL PROTECTED]:
OT, maybe, but this...
namespace private {
separator = .
hidden = no
inbox = yes
prefox =
^^^
...is such a nice typo I I couldn't resist :-) But I doubt it has
to do with your original problem.
HEH! Yeah... sorry - that was my fault typing it into the email; it's
prefix in the actual config file.
~Kyle
- --
The real problem is not whether machines think, but whether men do.
-- B. F. Skinner
-BEGIN PGP SIGNATURE-
Comment: Thank you for using encryption!
iEYEARECAAYFAkhiRzcACgkQBkIOoMqOI16GKACg8IjCoSW+vilan2D0iwWGtpKV
9PgAn0pDtVkqQUD6jJL4a+xiuD2dlXWm
=DMRd
-END PGP SIGNATURE-
[Dovecot] 1.1 and the zlib plugin
Hello,
I've been playing around with using namespaces to change the way
messages are stored. My idea was to use mbox for archival stuff, like
this:
namespace private {
separator = .
hidden = no
inbox = yes
prefox =
location = maildir:%h/Maildir
}
namespace private {
separator = .
hidden = no
prefix = INBOX.Archive.
location = mbox:%h/Maildir/Archive
}
This works *GREAT*, and I'm really pleased with it. However, it
occurred to me that for deeply frozen archives, it might be nice to
gzip them. Conveniently, there's the zlib plugin. From what I could
tell from the documentation, the compressed mbox names must end in
.gz, so I thought I could simply go into ~/Maildir/Archive and gzip
everything. But when I did that, Dovecot started behaving strangely.
First of all, the result of LIST shows the files like this:
* LIST (\NoInferiors \Marked) . INBOX.Archive.Friends.gz
Which is obviously wrong. When I try to SELECT it, I get this:
? SELECT INBOX.Archive.Friends.gz
? NO Mailbox doesn't exist: Friends/gz
Which seems to indicate that Dovecot is treating the dot in the
extension as the separator (understandably), and failing. However, if
I leave off the extension, I get this:
? SELECT INBOX.Archive.Friends
? NO Mailbox doesn't exist: Friends
Which seems to indicate that Dovecot isn't translating Friends into
Friends.gz for some reason.
Then I remembered that in the ChangeLog, back in May, the zlib plugin
was changed to look for the zlib header rather than the Z flag. I
assume that was for using it with Maildirs, but I figured it was worth
a shot. So I renamed the Friends.gz file to remove the extension. But
that just gives me an error message:
? SELECT INBOX.Archive.Friends
? NO Mailbox isn't a valid mbox file
It seems like loading the zlib plugin either doesn't have any effect,
or I'm not doing it right. I have it in the config file along with
other plugins that work just fine:
mail_plugins = fts fts_squat zlib
And when I check that with dovecot -p, it shows up:
mail_plugins: fts fts_squat zlib
There's no entry in the plugin {} section of the config file for the
zlib plugin... but I assume that none is necessary.
Am I doing something wrong?
~Kyle
--
No people can be great who have ceased to be virtuous.
-- Samuel Johnson, on the behavior of the British colonists in America
pgpV7WGyKUHsB.pgp
Description: PGP signature
Re: [Dovecot] 1.1 and the zlib plugin
On Tuesday, June 24 at 01:59 PM, quoth Kyle Wheeler:
I've been playing around with using namespaces to change the way
messages are stored. My idea was to use mbox for archival stuff,
like this:
namespace private {
separator = .
hidden = no
inbox = yes
prefox =
location = maildir:%h/Maildir
}
namespace private {
separator = .
hidden = no
prefix = INBOX.Archive.
location = mbox:%h/Maildir/Archive
}
Something seems to have gone wrong with my setup here. As I've been
testing moving large numbers of email into the archive folders (2000+
messages), some of them never make it to the new namespace. Instead of
showing up like they're supposed to, they appear as entirely empty
messages. Inspecting the raw mbox file, the message has become just an
mbox header and an empty line. Not good! Does anyone have any idea how
I can help debug this?
~Kyle
--
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.
-- Brian Kernighan
pgpgM5pUr30Th.pgp
Description: PGP signature
[Dovecot] Testing using namespaces
Hey there, I'm interested in testing out using different storage formats for different folder hierarchies. I *think* this can be achieved using namespaces, but I'm not sure. For example, is it possible to, say, make all directories under INBOX.Archive be stored as mboxes, while everything else is stored as Maildirs? ~Kyle -- May your glass be ever full. May the roof over your head be always strong. May you be in heaven half an hour before the devil knows you're dead. -- Old Irish Toast signature.asc Description: Digital signature
Re: [Dovecot] Antispam plugin custom behavior?
On Wednesday, June 11 at 06:34 PM, quoth Andre Rodier: As a temporary solution, and if your linux box as iNotify support, I suggest you use incron. incron is an inotify cron system. It works like the regular cron but is driven by filesystem events instead of time events. Interesting idea... thanks! ~Kyle -- It's amazing how much mature wisdom resembles being too tired. -- Robert A. Heinlein pgpKXBqrFAqmn.pgp Description: PGP signature
Re: [Dovecot] Antispam plugin custom behavior?
On Wednesday, June 11 at 11:51 PM, quoth Johannes Berg: On Wed, 2008-06-11 at 23:43 +0200, Johannes Berg wrote: Kyle, Obviously, I qualify as a spammer because I wrote the antispam plugin. Or something like that. Heh, sorry about that - I have my server set to reject non-list-related messages to my kyle-dovecot address... which was a good idea until I started posting to the list again. ~Kyle -- When you pray, do not be like the hypocrites, who love to stand and pray in the synagogues and on street corners so that others may see them. Amen, I say to you, they have received their reward. -- Matthew 6:5 pgpuGyXDJ49Uz.pgp Description: PGP signature
Re: [Dovecot] Need a quick, safe method to empty /home/user/Maildir/{.Junk, .Trash}
On Thursday, June 12 at 11:10 AM, quoth Bill Cole: What is the safest way to empty all messages within, but not delete, the following folders from the server command line: /home/user/Maildir/.Junk /home/user/Maildir/.Trash I don't want the Thunderbird-2.0.14 client to report corrupt indexing, or worse not show the folder at next use. The way to do that in a shell (rather than via Dovecot) would be to remove all of the current message files and all of Dovecot's indexing and metadata caching: cd /home/user/Maildir/.Junk rm dovecot-index* cur/* cd /home/user/Maildir/.Trash rm dovecot-index* cur/* (restructure that as you like, I've written it that way to make the operations clear) If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them: find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete ... and then get rid of the index files as indicated above, of course. ~Kyle -- Men, as an organization, are getting more women than any other group working anywhere in the world. Wherever women are, we have men looking into it. -- Jerry Seinfeld pgpNC8uAxiTJC.pgp Description: PGP signature
Re: [Dovecot] Need a quick, safe method to empty /home/user/Maildir/{.Junk, .Trash}
On Thursday, June 12 at 01:02 PM, quoth Jeff Kowalczyk: On Thu, 12 Jun 2008 10:19:48 -0500, Kyle Wheeler wrote: If you have a *TON* of messages in those folders (i.e. several thousand), your shell may complain that there are too many arguments to the rm command. If that happens, these may be better commands to delete them: find /home/user/Maildir/.Junk/cur -type f -delete find /home/user/Maildir/.Trash/cur -type f -delete Thanks, I'm going to do that. And per dmeissler, I'm going to consider xargs. shrug xargs is good for other things, but if all you're doing is deleting files, find can do it itself without involving other applications. But whatever makes you comfortable. Any thoughts on a variant using find -name that could safely iterate over /home/*/Maildir for all users? Otherwise I would script it in python. find /home/ -path '*/Maildir/.Junk/cur/*' -type f -delete ~Kyle -- Nothing makes a woman more beautiful than the belief she is beautiful. -- Sophia Loren pgp8W7G26oYVZ.pgp Description: PGP signature
[Dovecot] Antispam plugin custom behavior?
Hello, I currently have a setup on my system with what I call magic folders to enable spam filter training. Here's how it works: 1. If you have a false-negative, put the spam into the Spam.Report folder 2. If you have a false-positive (which has all kinds of ugly spamassassin protective markup in it), put the message into the Spam.NotSpam folder Currently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup, retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted. I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes. I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder). Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way? ~Kyle -- The optimist thinks this is the best of all possible worlds. The pessimist fears it is true. -- J. Robert Oppenheimer pgpc1b5QsgPd9.pgp Description: PGP signature
Re: [Dovecot] Antispam plugin custom behavior?
On Wednesday, June 11 at 05:01 PM, quoth Hugo Monteiro: Have you tried the plugin using the mailtrain backend? The antispam plugin? No, I haven't... mostly because it looks like no matter which backend I use, I'd have to alter the user-visible interface to my training system (which I don't really want to do), and it still doesn't handle the altered message problem. Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like [EMAIL PROTECTED] or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way. Hmmm, load is something I hadn't thought about... (the system I'm working with at the moment has plenty of capacity to spare). That's a good point. However, one of the goals here is to make it so that if a user identifies a message that has been mistakenly tagged as spam (and sanitized by SpamAssassin, e.g. via the report_safe setting), they can get the message corrected (and back to its original form) immediately. As it is, they have to put it into the NotSpam folder and wait a couple minutes for the message to reappear in the INBOX (because the cron job only runs every so often). ~Kyle -- No, I don't know that Atheists should be considered as citizens, nor should they be considered patriots. This is one nation under God. -- George H. W. Bush, August 27, 1987 pgpFYF6mF6DkC.pgp Description: PGP signature
Re: [Dovecot] Antispam plugin custom behavior?
On Wednesday, June 11 at 05:33 PM, quoth Hugo Monteiro: Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior. Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved. ~Kyle No different behaviour for the end user. Your user could continue to dragdrop messages in/out of the designated Spam folder. You misunderstood my original email. Users have two folders: one for reporting spam that wasn't identified as spam (the Spam.Report folder), one for rescuing ham that was misidentified as spam (the Spam.NotSpam folder). Thus, there is no implicit behavior - putting a message in one of those folders explicitly tells the system learn this message or unlearn this message. The only different, which is not visible to the end user, is that the retrain of false positives is activated by pulling the messages out of the Spam folder, rather than having to specifically put it in a Ham folder. How is that not visible to the end-user? That said, they can continue to use the Ham folder as a placebo. ;) Heh. I see. But the user expects messages in the Ham folder to disappear and be automatically redelivered to their inbox. Out of curiosity, why would you need to alter the message when moving it around? Because my system uses SpamAssassin's report_safe feature (see http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html#item_report_safe). When a message is identified as spam, instead of modifying the original message, the user receives a report message with the original message attached as a text/plain attachment. This makes reading the original rather difficult (on purpose). Thus, one of the benefits of moving the message to the NotSpam folder is that the message is restored to its original form and redelivered. ~Kyle -- Unthinking respect for authority is the greatest enemy of truth. -- Albert Einstein pgpfbIeMTNV7z.pgp Description: PGP signature
[Dovecot] Can the LDA deliver email marked as read?
Hello, One of my users wants to be able to deliver email (e.g. via a procmail script) and mark it *read* when it's delivered. We use Maildir as our mail storage format. I have a bit of a hack script put together that will do it by directly manipulating the Maildir (it delivers the email using safecat, and then renames the file into the cur directory with the necessary flags appended to the file name), but this is a hack at best, and has the problem that if Dovecot is monitoring the file, it could potentially fail to work---there is a race between delivery and renaming. I've been slowly moving our site in the direction of using the Dovecot LDA to deliver mail. Is there a way (via a plugin, maybe?) to get the LDA to mark mail as read when it is delivered? ~Kyle -- He who dares not offend cannot be honest. -- Thomas Paine pgpPEsUJIrrWo.pgp Description: PGP signature
[Dovecot] deliver fails - passdb doesn't support lookups?
Hello,
I'm trying to get `deliver` (the LDA) to function properly. I can get
it to work without doing a user lookup, but for my own sense of
sanity, I want to know what I'm doing wrong getting user lookups
working. (I'm using dovecot 1.0.10).
Here's what I have in the auth default section:
auth default {
mechanisms = plain login
passdb ldap {
args= /var/lib/dovecot/dovecot-ldap.conf
}
userdb static {
args = uid=3728 gid=3728 home=/domains/%Ld/%Ln
}
user = vpopmail
socket listen {
master {
path = /var/run/dovecot/auth-master
mode = 0600
user = vpopmail
group = vchkpw
}
client {
path = /var/run/dovecot/auth-client
mode = 0660
}
}
}
And yet, when I run deliver, I get no warnings, no errors, and most
especially, no email delivered. Here's how I call deliver:
cat testemail | setuidgid vpopmail \
/usr/local/libexec/dovecot/deliver \
-d [EMAIL PROTECTED]
I tried running deliver within strace, and here's what I found. It
opens up the authentication socket and writes:
[EMAIL PROTECTED]
And it gets back:
VERSION\t1\t0\nSPID\t10917\nFAIL\t1\n
If I understand the authentication protocol correctly, that means an
internal error occurred.
The dovecot log file reports this:
Error: auth(default): static([EMAIL PROTECTED]): passdb doesn't
support lookups, can't verify user's existence
passdb? I thought it was the userdb that was important! What's going
on here?
~Kyle
--
Men, as an organization, are getting more women than any other group
working anywhere in the world. Wherever women are, we have men looking
into it.
-- Jerry Seinfeld
pgpEnizZR6vZf.pgp
Description: PGP signature
Re: [Dovecot] deliver fails - passdb doesn't support lookups?
On Wednesday, March 12 at 11:40 AM, quoth Andrew Roberts: On Wed, 12 Mar 2008, Kyle Wheeler wrote: I'm trying to get `deliver` (the LDA) to function properly. I can get it to work without doing a user lookup, but for my own sense of sanity, I want to know what I'm doing wrong getting user lookups working. (I'm using dovecot 1.0.10). Here's what I have in the auth default section: args = uid=3728 gid=3728 home=/domains/%Ld/%Ln passdb? I thought it was the userdb that was important! What's going on here? With a static userdb, dovecot still wants to verify that the user exists with a passdb lookup. If your MTA is configured to verify that the user exists before passing mails to the LDA, you safely tell userdb to disable this check by addding allow_all_users=yes to your passdb args. Ahhh! That makes sense, thank you! Okay, so allow_all_users=yes would make it work... but what if I do want it to check users. What might I have wrong in my ldap settings that would prevent it from looking up users? And what kind of a performance penalty do I pay for making that lookup? ~Kyle -- The best way to know God is to love many things. -- Vincent Van Gogh pgpbKMpbMwGGA.pgp Description: PGP signature
Re: [Dovecot] Deleting messages from MailDir
On Thursday, February 14 at 09:50 AM, quoth Bill Cole: I'm curious: do you have examples of mail software that doesn't use the timestamp? (I could see some run-once script not doing it, but I'd be surprised if widely-used software didn't.) The procmailrc man page says that MSGPREFIX defaults to 'msg.' but it is not clear to me whether MSGPREFIX is used for Maildir delivery or just for MH delivery. I've played it safe and use MSGPREFIX=`date +%s` but maybe that's not needed... MSGPREFIX is only used if procmail is delivering to a directory that hasn't been indicated as either an MH folder or a Maildir by a suffix. In other words, if you tell it to deliver to foo/, it'll use the official maildir methodology. If you tell it to deliver to foo/., it'll use the official MH methodology. If you tell it to deliver to foo, and foo happens to be a directory, then it'll use MSGPREFIX. If you tell it to deliver to foo and foo is a file, then it'll use an mbox methodology. ~Kyle -- They think I'm crazy, and maybe I am. But maybe I'm a genius. -- Mel Gibson pgpYO4bNgwGNB.pgp Description: PGP signature
Re: [Dovecot] [OT] Webmail Recommendation
On Friday, January 11 at 01:21 PM, quoth Hannes Erven: Before switching to dovecot, courier-imap handled the backend and I used Squirrelmail as the front-end. We used to use BincIMAP and Squirrelmail. imapproxy had a huge (positive) impact on performance, especially when browsing through folders with many messages. Startup (building the maildir tree with message counts) still took its time, and searching in Squirrelmail also was a pain. We had the same experience. Thanks to Dovecot, startup and search (in from/to, subject) now is really fast and I turned off imapproxy completely as it did not further improve the webmail's performance. I guess in environments where authentication is expensive (slow) imapproxy sure is worth a look at. We did the same, for the same reasons. Dovecot offered an order of magnitude increase (subjective) in responsiveness, and I couldn't tell the difference between using imapproxy and not using imapproxy---we removed it just to reduce the complexity of the system. ~Kyle -- There in America we are descended in blood and in spirit from revolutionist and rebel men and women who dare to dissent from accepted doctrine. As their heirs, may we never confuse honest dissent with disloyal subversion. -- Dwight D. Eisenhower pgpCyUcMKuWW5.pgp Description: PGP signature
Re: [Dovecot] migrating from mbox to maildir
On Saturday, November 24 at 10:13 PM, quoth mouss: If for some reason you are completely stuck with 0.99.x, then yes, changing to Maildir format is pretty easy, and there are plenty of conversion scripts out there in the world. Just ask Google. Keep in mind if the real problem is some bug in Centos that prevents Dovecot from using locks, then the Maildir conversion may not help much, because Maildir (in Dovecot) uses locks as well. why lock? To quote the Dovecot wiki (http://wiki.dovecot.org/MailboxFormat/Maildir): Although maildir was designed to be lockless, Dovecot locks the maildir while doing modifications to it or while looking for new messages in it. This is required because otherwise Dovecot might temporarily see mails incorrectly deleted, which would cause trouble. Basically the problem is that if one process modifies the maildir (eg. a rename() to change a message's flag), another process in the middle of listing files at the same time could skip a file. The skipping happens because readdir() system call doesn't guarantee that all the files are returned if the directory is modified between the calls to it. This problem exists with all the commonly used filesystems. ~Kyle -- The most important thing a father can do for his children is to love their mother. -- Fr. Theodpre Hesburgh pgpG2BgD58zmL.pgp Description: PGP signature
Re: [Dovecot] migrating from mbox to maildir
On Friday, November 23 at 01:53 PM, quoth Erick Perez: Hi all, I have a locking problem in a Centos 4.4 linux machine with the following config: dovecot-0.99.11-8.EL4 The problem is quite likely one that was fixed in more recent versions of dovecot. Versions 0.99.x are extremely old (to the point of nearly being an entirely different product from Dovecot 1.0). then someone told me that I need to change dovecot to maildir format. Is there a reference to change format? Or some comments about this? Dovecot 0.99.x's Maildir support may have been better than its mbox support, but that's a bad reason to convert all your mail storage. If you update your Dovecot to a non-antique version, chances are you'll fix your problem with much less effort. If for some reason you are completely stuck with 0.99.x, then yes, changing to Maildir format is pretty easy, and there are plenty of conversion scripts out there in the world. Just ask Google. Keep in mind if the real problem is some bug in Centos that prevents Dovecot from using locks, then the Maildir conversion may not help much, because Maildir (in Dovecot) uses locks as well. ~Kyle -- Man has the right to act in conscience and in freedom so as personally to make moral decisions. He must not be forced to act contrary to his conscience. Nor must he be prevented from acting according to his conscience, especially in religious matters. -- Catholic Catechism 1782 signature.asc Description: Digital signature
Re: [Dovecot] SSL/TLS with Outlook client
On Wednesday, November 14 at 11:51 AM, quoth Ed W: Is TLS always performed BEFORE auth with generally available POP/IMAP clients? Yes, because that's generally the entire point of using encryption. After all, what's more important: encrypting your username/password before transmitting it over an open wire, or encrypting your email messages before transmitting them over an open wire? (Hint: if you need your email encrypted, use PGP.) Technically, there's nothing in the IMAP spec that forbids doing it the other way around, however many IMAP servers (including Dovecot) typically reject unencrypted authentication attempts. Random idea but if there were some way to identify the client BEFORE presenting the certificate then it would be possible to present one of a number of certificates depending on the incoming client Of course, but unfortunately, there's very little. The only thing the server can reliably know is the client's IP address and source TCP port (and it's own IP address). Not much to go on. (don't fancy scraping SMTP server log files and matching back to IP addresses though...) HEH. SMTP-before-IMAP? What a bizarre concept. :) You'd just be transferring the problem: how does the SMTP server know what certificate to use? ~Kyle -- You can gain reconciliation from your enemies, but you can only gain peace from yourself. -- Rubin The Hurricane Carter pgpYb8OarAFL7.pgp Description: PGP signature
Re: [Dovecot] SSL/TLS with Outlook client
On Wednesday, November 14 at 09:35 PM, quoth Nikolay Shopik: And HELO in SMTP is entirely unreliable, unverifiable, and on many servers completely skippable. RFC says you SHOULD use FQDN for HELO nothing more. But still you can add SPF record for your HELO so nobody can foged your server HELO, thats it. To quote RFC 821: The HELO receiver MAY verify that the HELO parameter really corresponds to the IP address of the sender. However, the receiver MUST NOT refuse to accept a message, even if the sender's HELO command fails verification. If you prefer RFC 2821: An SMTP server MAY verify that the domain name parameter in the EHLO command actually corresponds to the IP address of the client. However, the server MUST NOT refuse to accept a message for this reason if the verification fails: the information about verification failure is for logging and tracing only. In practice, what that means is that HELO is useless for doing much of anything. Spammers or other criminals can forge your server's HELO to their hearts content and you are expressly forbidden from actually doing anything about it. SPF does not override the existing standards. And in any case, SPF HELO checks are a pointless exercise, since HELO is permitted to be anything at all without affecting the envelope of the message. A spammer can create his own domain, publish his own SPF settings that explicitly allow email from any source, and use that domain as his HELO string. ~Kyle -- I believe that every human has a finite number of heart-beats. I don't intend to waste any of mine running around doing exercises. -- Neil Armstrong pgpxoVg5qadLG.pgp Description: PGP signature
Re: [Dovecot] SSL/TLS with Outlook client
On Wednesday, November 14 at 09:15 PM, quoth Timo Sirainen: On Wed, 2007-11-14 at 12:29 -0600, Kyle Wheeler wrote: On Wednesday, November 14 at 11:51 AM, quoth Ed W: Is TLS always performed BEFORE auth with generally available POP/IMAP clients? .. Technically, there's nothing in the IMAP spec that forbids doing it the other way around, Actually there is :) STARTTLS is a valid command only in non-authenticated state. Ah! My mistake. That's what I get for not paying close enough attention. :) ~Kyle -- For every complex problem, there is a solution that is simple, neat, and wrong. -- H. L. Mencken pgpbeQYtKfXG3.pgp Description: PGP signature
Re: [Dovecot] SSL/TLS with Outlook client
On Wednesday, November 14 at 10:51 PM, quoth Marcus Rueckert: rejecting on wrong informations in HELO/EHLO saves me lots of spam. That's a half-baked idea at best, given that you're violating a MUST NOT in the SMTP specification. Plus, how do you judge wrong? Hotmail and MSN both fail to use their FQDNs in their HELO arguments---technically that's wrong. I suppose you reject all hotmail.com email? Check out: http://homepages.tesco.net/~J.deBoynePollard/FGA/smtp-avoid-helo.html ~Kyle -- Science is what we can tell a computer. Art is everything else. -- Knuth pgpkjtAo5yEZM.pgp Description: PGP signature
Re: [Dovecot] v1.0.6 released
On Friday, November 2 at 04:50 PM, quoth Timo Sirainen: On Thu, 2007-11-01 at 14:31 -0500, Kyle Wheeler wrote: On Sunday, October 28 at 03:16 AM, quoth Timo Sirainen: * SORT: If Date: header is missing or broken, fallback to using INTERNALDATE (as the SORT draft nowadays specifies). Since this is a subject I looked at before, I'm rather curious. Where in the SORT draft does it say to fall back to INTERNALDATE? Just search for INTERNALDATE :) DATE Sent date and time from the Date: header, adjusted by time zone. This differs from the SENTON criteria in SEARCH, which uses just the date and not the time, nor adjusts by time zone. If the sent date can not be determined (a Date: header is missing or can not be parsed), the INTERNALDATE for that message is used as the sent date. Very interesting... two different pages on the IETF's website have two different texts for version 19 of that draft. Odd. Anyway, at best, it would seem that the text has multiple potential interpretations, because the parenthetical explanation there of what it means for a sent date to be un-determinable is different from (and contradictory of) what it says in section 2.2 (which is what I quoted). Treating invalid time as 00:00:00 should probably be done at some point, but it's a pretty rare problem and also a SHOULD, not a MUST :) I would tend to disagree - the draft says more than simply that invalid time must be treated as 00:00:00, but rather: If there is no valid date or time, the date and time SHOULD be treated as 00:00:00 on the earliest possible date. Thus, even if you fall back to INTERNALDATE, the time should be zeroed out. But you're right, it is a SHOULD rather than a MUST. ~Kyle -- If God can work through me, he can work through anyone. -- St. Francis of Assisi pgpdjEj7TRgL7.pgp Description: PGP signature
Re: [Dovecot] Spliting Folders for Efficiency
On Saturday, October 13 at 09:25 AM, quoth Daniel W: Thanks for the insights. Is it also true that to read a single message in a 800MB mbox, you need to load 800MB of data into memory which is then searched for that message? Not at all. If you don't know what message you're looking for, then yes (kinda: you could just mmap the mbox file, which reduces your latency before beginning the search), but Maildir has an even worse problem: if you don't know what message you're looking for, you have to open and close every single message-file. And open()/close() typically has quite a bit more overhead than lseek(). More to the point, when searching for a file in an mbox, the OS has a very good idea of what you're going to be looking at next (linear search is predictable that way), so it can do a much better job of prefetching and I/O scheduling for a search through an mbox than it can for a Maildir search. Again, mbox wins. On the other hand, if you know exactly what message you're looking for, the necessary I/O is only slightly different. In an mbox, knowing which message you're looking for is best expressed as an offset within the file. Similarly, in a Maildir, knowing which message you're looking for is best expressed as a filename, or (better still, in some cases) an inode number. In an mbox, then, you have to open() the file and lseek() to the correct offset (which, in an exceedingly large mbox, may require log(sizeoffile) disk accesses to begin the first read). In a Maildir, you have to merely open() the file, however rather than dealing with the filesystem's method of storing a file, you have to deal with the filesystem's method of storing filenames. In fancy filesystems (e.g. ReiserFS or ext3 with dir_hashing turned on), this can be pretty fast ---on the order of log(numberofmessages), but in boring filesystems (e.g. ext2, ext3 without dir_hashing, vfat, etc.) this can take a lot of time. Between the two, on average, the I/O load is about the same for both actions, though the filesystem particulars are what really make one or the other a better fit for a given situation. The really irritating thing about Maildir is that the filenames can change, meaning that knowing which message you want (i.e. you have a filename) may still mean you have to scan through the list of available filenames and see which ones are similar to the name you wanted (see why having an inode number can be more useful?), which takes MUCH longer than lseek(). That would suggest that mbox is only scaleable to a realtively small inbox size. Not really. eg. Splitting by message size. If a message is much smaller than the block size, use a single file format and if larger, write out to it's own file. Every folder would have two mechanisms and Dovecot could just look at each message as it comes in to decide how to store it. Yes, but then you get to the question of: what does that buy you? And, better still: how do you find any given message? Filename+offset? You'd be compounding the worst details of both designs. Not only do you have to lseek() to find your small message, but you have to pay the filename lookup penalty as well---even if you know exactly where your message is. On the other hand, you've reduced the cost of both by relying on the other: your lseek overhead is lower because you are dealing with a smaller file than you'd ordinarily have to, and your filename lookup overhead is lower because you've got fewer files. So, whether this is a good idea probably, once again, depends very much on where the performance curves bend (e.g. if your filesystem gets much slower for more than 10,000 files in one directory, or if it gets much slower if your file is over 1G, or something like that). If your filesystem scales linearly, though, it's not a net gain. ~Kyle -- Come to me, son of Jor-El. Kneel before Zod. Snootchie-bootchies. -- Jay pgp8nBaNVnEph.pgp Description: PGP signature
Re: [Dovecot] v1.0.6 released
On Sunday, October 28 at 03:16 AM, quoth Timo Sirainen:
* SORT: If Date: header is missing or broken, fallback to using
INTERNALDATE (as the SORT draft nowadays specifies).
Since this is a subject I looked at before, I'm rather curious. Where
in the SORT draft does it say to fall back to INTERNALDATE? What I
read (from http://tools.ietf.org/id/draft-ietf-imapext-sort-19.txt,
section 2.2) is this:
As used in this document, the term sent date refers to the date
and time from the Date: header, adjusted by time zone to normalize
to UTC. For example, 31 Dec 2000 16:01:33 -0800 is equivalent
to the UTC date and time of 1 Jan 2001 00:01:33 +.
If the time zone is invalid, the date and time SHOULD be treated
as UTC. If the time is also invalid, the time SHOULD be treated
as 00:00:00. If there is no valid date or time, the date and time
SHOULD be treated as 00:00:00 on the earliest possible date.
The key part is that last sentence: without a Date header, there is no
valid date or time, so the date and time SHOULD be treated as 00:00:00
on the *earliest possible* date. The earliest possible date, however, is
not INTERNALDATE, but is instead the earliest timestamp in any of the
Received headers (or INTERNALDATE, if that's earlier, though that
would be pretty rare).
Even if we didn't feel like parsing Received headers,
mail_get_received_date() (i.e. the current 1.0.7 implementation) is
wrong because it isn't 00:00:00 on that date.
Thus, I think mail_sort_get_date() in imap-sort.c should look more
like this:
static time_t mail_sort_get_date(struct mail *mail)
{
time_t t, t2;
t = mail_get_date(mail, NULL);
if (t == 0 || t == (time_t)-1) {
/* missing or broken Date: header */
const char *const *headers = mail_get_headers(mail, Received);
if (headers != NULL headers[0] != NULL) {
while (headers[1] != NULL) headers++; // find the last one
/* first, find the semicolon */
const char * curs = headers[0];
while (curs[0] != ';' curs[0] != 0) {
if (curs[0] == '(')
while (curs[0] != ')' curs[0] != 0) curs++;
curs++;
}
if (curs[0] == ';') {
curs++;
if (curs[0] != 0) {
int tz;
message_date_parse((const unsigned char *)curs,
strlen(curs), t, tz);
}
}
}
t2 = mail_get_received_date(mail); // INTERNALDATE
if (t2 t) t = t2;
}
return t;
}
However, this doesn't follow the spec's requirement of being 00:00:00
on that date. I'm not sure if there's an easy fix, though (be nice if
we could just apply a bitmask; but we can't).
What do you think?
~Kyle
--
A tyrant must put on the appearance of uncommon devotion to religion.
Subjects are less apprehensive of illegal treatment from a ruler whom
they consider God-fearing and pious. On the other hand, they do less
easily move against him, believing that he has the gods on his side.
-- Aristotle
pgpmn76uXwT7i.pgp
Description: PGP signature
Re: [Dovecot] v1.0.6 released
On Thursday, November 1 at 02:31 PM, quoth Kyle Wheeler: On Sunday, October 28 at 03:16 AM, quoth Timo Sirainen: * SORT: If Date: header is missing or broken, fallback to using INTERNALDATE (as the SORT draft nowadays specifies). Since this is a subject I looked at before, I'm rather curious. Where in the SORT draft does it say to fall back to INTERNALDATE? What I read (from http://tools.ietf.org/id/draft-ietf-imapext-sort-19.txt, section 2.2) is this: A better, more stable link is here: http://tools.ietf.org/html/draft-ietf-imapext-sort As used in this document, the term sent date refers to the date and time from the Date: header, adjusted by time zone to normalize to UTC. For example, 31 Dec 2000 16:01:33 -0800 is equivalent to the UTC date and time of 1 Jan 2001 00:01:33 +. If the time zone is invalid, the date and time SHOULD be treated as UTC. If the time is also invalid, the time SHOULD be treated as 00:00:00. If there is no valid date or time, the date and time SHOULD be treated as 00:00:00 on the earliest possible date. ~Kyle -- The fact that we live at the bottom of a deep gravity well, on the surface of a gas-covered planet going around a nuclear fireball 90 million miles away, and think this to be normal, is obviously some indication of how skewed our perspective tends to be... -- Douglas Adams pgpIfEiAlG420.pgp Description: PGP signature
Re: [Dovecot] Spliting Folders for Efficiency
On Friday, October 12 at 11:06 AM, quoth Daniel Watts: What actually ARE the advantages of a 'one file per folder' format?? It depends on the environment. It's exceedingly efficient at storage: on a filesystem with 4k blocks, three 1k messages take up 1 block (4k), where in a one-file-per-message format they take up 3 blocks (12k). Some filesystems have mechanisms of coping with files that only occupy a partial block, but those mechanisms tend to be expensive, and are often only employed when strapped for space. The one-file-per-folder arrangement also helps when doing sequential reads (i.e. searches, or loading it into memory, or processing it with a filter, or whatever else): when the OS spools the file from disk, it loads it up a block at a time, which in a one-file-per-folder format is several messages, but in a one-file-per-message format is only ever a single message. I've often contemplated setting up a separate mbox-based namespace in my Dovecot setup (e.g. everything in the Archive folder is saved as an mbox), just for the space savings. ~Kyle -- Only the fool hopes to repeat an experience; the wise man knows that every experience is to be viewed as a blessing. -- Henry Miller pgpn7Yd1yyMdC.pgp Description: PGP signature
Re: [Dovecot] SORT(DATE) and missing Date headers
On Tuesday, September 25 at 06:26 PM, quoth Timo Sirainen:
That's not *quite* what I meant. ARRIVAL is when did this mail get
here, while DATE is supposed to be when was this mail sent. My
thought here is that when was this mail sent can be approximated
in the absence of a Date header by checking the earliest timestamp
in the Received headers.
So, something like:
const char *const *headers = mail_get_headers(mail, Received);
if (headers != NULL headers[0] != NULL) {
while (headers[1] != NULL) headers++;
// do your Received header parsing magic for headers[0]
}
Aha! That's perfect! (and so simple!)
If anyone in the future is interested in the code for this, here's
what I did that works for me. This goes in all three places that
mail_get_date() is used in the code:
t = mail_get_date(mail, NULL);
if (t == (time_t)-1 || t == 0) {
const char *const *headers = mail_get_headers(mail, Received);
if (headers != NULL headers[0] != NULL) {
while (headers[1] != NULL) headers++; // find the last one
/* find the semicolon */
const char * curs = headers[0];
while (curs[0] != ';' curs[0] != 0) {
if (curs[0] == '(')
while (curs[0] != ')' curs[0] != 0) curs++;
curs++;
if (curs[0] == ';') {
curs++;
if (curs[0] != 0) {
int tz;
message_date_parse((const unsigned char *)curs,
strlen(curs), t, tz);
}
}
}
}
Thanks, Timo!
~Kyle
--
Coffee is the common man's gold, and like gold, it brings to every
person the feeling of luxury and nobility.
-- Sheik Abd-al-Kadir
pgpkFPmJBJonl.pgp
Description: PGP signature
Re: [Dovecot] SORT(DATE) and missing Date headers
On Tuesday, October 2 at 09:49 AM, quoth Kyle Wheeler: Aha! That's perfect! (and so simple!) If anyone in the future is interested in the code for this, here's what I did that works for me. This goes in all three places that mail_get_date() is used in the code: I put up a patch for this (and the two other things I regularly patch dovecot to do) up on the web, in case other people find them useful: http://www.memoryhole.net/dovecot/ ~Kyle -- The average Ph.D thesis is nothing but the transference of bones from one graveyard to another. -- J. Frank Dobie, A Texan in England pgpsCfRx2IZy1.pgp Description: PGP signature
Re: [Dovecot] SORT(DATE) and missing Date headers
On Monday, September 24 at 04:19 PM, quoth Rich at Whidbey Telecom: We recently encountered this with a new VOIP voicemail system. However, using Thunderbird at least, the time the message file was written is used (probably using INTERNALDATE). This might only apply if you're using Maildir's, but it shows that the behavior you're seeing might be specific to Mutt. It's not mutt, actually. The client where this became an issue is RoundCube webmail. Mutt falls back to INTERNALDATE, but think about it: it requires sorting on the client's side, because you're combining SORT(DATE) and SORT(ARRIVAL), which completely negates the whole point of server-side sorting. Even then, though, INTERNALDATE is a bad approximation if I want to sort by time *sent* (compare, for example, a message that I bounced from my work address to my home mailbox). The best approximation would be to use the first-added Received header (which could be several days different from INTERNALDATE). Yes, I know *most* mail clients have a workaround for this problem. Is that really the best solution, though? To simply apply a workaround to all IMAP clients, rather than fix it in the IMAP server? ~Kyle -- Never think that war, no matter how necessary, no matter how justified, is not a crime. -- Ernest Hemingway pgp41fi5bO1yN.pgp Description: PGP signature
Re: [Dovecot] SORT(DATE) and missing Date headers
On Tuesday, September 25 at 12:39 PM, quoth Timo Sirainen: How hard would this be to hack into the current Dovecot source? Replace mail_get_date() calls in src/imap/imap-sort.c with something like: t = mail_get_date(..); if (t == (time_t)-1) t = mail_get_received_date(..); That's not *quite* what I meant. ARRIVAL is when did this mail get here, while DATE is supposed to be when was this mail sent. My thought here is that when was this mail sent can be approximated in the absence of a Date header by checking the earliest timestamp in the Received headers. mail_get_received_date() returns the *latest* timestamp in the Received headers (actually, in a Maildir backend, it just returns the fstat of the message file), so in a folder full of messages without Date headers, SORT(DATE) and SORT(ARRIVAL) would be identical, which is not what I'm aiming for. But, looking at it, I guess this becomes pretty difficult. To work around the fstat, I'd have to do something even more complicated than an mbox _read() to find the Received header with the oldest timestamp. Is that correct? ~Kyle -- The sacred rights of mankind are not to be rummaged for, among old parchments, or musty records. They are written, as with a sun beam in the whole volume of human nature, by the hand of the divinity itself; and can never be erased or obscured by mortal power. -- Alexander Hamilton, 1775 pgps2ln0YY4Q9.pgp Description: PGP signature
Re: [Dovecot] o/s tuning for imap
On Tuesday, September 4 at 08:26 PM, quoth Russell E. Meek: OS related tweaks, probably not. However you could utilize a imap proxy such as up-imapproxy which if using FreeBSD is in ports. Visit: http://www.imapproxy.org/ to learn more. This should relieve the load on Dovecot. We found that on our server, *not* using imapproxy improved our performance. We used to use imapproxy to great effect when we were using BincIMAP, but Dovecot is so darn fast (and caches its own authentication) that all imapproxy added was additional inter-process communication (translation: slower than just using Dovecot alone). ~Kyle -- Only a mediocre person is always at his best. -- Somerset Maugham pgp4jX7cgtYNt.pgp Description: PGP signature
Re: [Dovecot] o/s tuning for imap
On Tuesday, September 4 at 12:16 PM, quoth Ken A: I'm switching from a pop3 only dovecot install to a pop3/imap install and I'm wondering how many connections every 100 'normal' imap users might have/keep open? Mmmm, I usually estimate that most of the time users keep one connection open. Occasionally some clients (like Mail.app with the IDLE plugin) keep two or three open at all times. Then of course, there's occasional bursts where a client may open ten or more connections, though because IMAP is asynchronous that's technically unnecessary unless you're doing it to reduce latency (unlikely), but some clients do it anyway. So I'd leave a max of around 300-500 connections at minimum. I'm wondering if I need to tweak any o/s related things, like time_wait, etc. Any pointers would be greatly appreciated. The thing to keep in mind about IMAP versus POP is that more folks will be keeping things on your server. That means you're going to need more disk space and (particularly if you use a Maildir backend) more inodes, and you're going to want to reexamine your filesystem and mount options. For example, if you haven't already, turn off atime updating. It's entirely useless with Dovecot, and will just slow you down. ~Kyle -- The community which does not protect its humblest and most hated member in the free utterance of his opinions, no matter how false or hateful, is only a gang of slaves. If there is anything in the universe that can't stand discussion, let it crack. -- Wendell Phillips, 1863 pgpHYF2GIrw8t.pgp Description: PGP signature
Re: [Dovecot] removing IMAP keywords?
On Thursday, August 23 at 05:14 PM, quoth martin f krafft:
Also, does someone know where I can find specification on what
characters are allowed for keywords? RFC 3501 is strangely quiet on
this, or I am blind.
Check out section 9, Formal Syntax. Specifically, flag-keyword,
which is defined to be an atom, which is a sequence of ANY character
except the atom-specials. In other words, a flag-keyword is a string
of one or more characters, not including (, ), {, , control
characters, %, *, , \, and ].
~Kyle
--
Truth never damages a cause that is just.
-- Mahatma Gandhi
pgpPjPy4sL727.pgp
Description: PGP signature
Re: [Dovecot] UW-IMAP to Dovecot conversion - How to migrate the folders?
On Tuesday, August 21 at 02:06 PM, quoth Patrick - South Valley Internet: Thanks Kyle, but how do I convert the mbox-like IMAP folders into something Dovecot can read with the new config? One way (the most straightforward) is to use any of the available mbox-to-maildir converter scripts. Search google for the text mbox to maildir and you should get references to a bunch of them. You may be able to use Dovecot's convert plugin, but I've never played with it, so I don't know much about setting it up. ~Kyle -- The only fool bigger than the person who knows it all is the person who argues with him. -- Stanislaw Jerszy Lec pgpLPmUWS9CUF.pgp Description: PGP signature
Re: [Dovecot] UW-IMAP to Dovecot conversion - How to migrate the folders?
On Tuesday, August 21 at 02:15 PM, quoth Patrick - South Valley Internet: What's even more odd is that when I created a new folder within Outlook Express, I see it in /home/USERNAME/Maildir/subscriptions, but I don't see the folder anywhere...how does Dovecot see these IMAP folders? Is there some sort of tag that is contained within each Maildir email that tells what folder it goes in to? The new folders are created with names prefixed by a period. For example, imagine you have the following IMAP folders (subfolders separated by a slash (/)): INBOX Sent Trash Drafts INBOX/Business INBOX/Family These would typically be stored like so: /home/USERNAME/Maildir/ -- INBOX /home/USERNAME/Maildir/.Sent/ /home/USERNAME/Maildir/.Trash/ /home/USERNAME/Maildir/.Drafts/ /home/USERNAME/Maildir/.INBOX.Business/ /home/USERNAME/Maildir/.INBOX.Family/ Doing a simple `ls` in the Maildir won't show the other folders, because they begin with a period (and are considered hidden by ls). Do an `ls -a` and they'll show up. That's standard Maildir++ layout, which is what you get when your config says: mail_location = maildir:%h/Maildir ~Kyle -- It ain't the parts of the Bible that I can't understand that bother me, it is the parts that I do understand. -- Mark Twain pgpwfPNiVhBrO.pgp Description: PGP signature
Re: [Dovecot] Dovecot fails almost every 20 minutes exactly
On Sunday, August 19 at 02:13 PM, quoth Patrick - South Valley Internet: Now that we're in the production environment, we've noticed that every 20 minutes, Dovecot will stop running. Meaning what? Is the dovecot process still alive? Is the service unresponsive? Is it just not allowing logins? Is the shared NFS store the only thing shared between the two? Is there a load-balancer in front of the machines? I can't see anything out of the ordinary in the log files either. What are the last couple entries in the log files when it stops? The same things every time, or something different every time? ~Kyle -- If Mr. Einstein doesn't like the natural laws of the universe, let him go back to where he came from. -- Robert Benchley (1889-1945) pgplX5EPDAq36.pgp Description: PGP signature
Re: [Dovecot] UW-IMAP to Dovecot conversion - How to migrate the folders?
On Monday, August 20 at 04:07 PM, quoth Patrick - South Valley Internet: I have, but it didn't do anything that I could tell. I tried resyncing my IMAP but I didn't see the new folder. Does it matter that the UW-IMAP folders are in mbox-like format? It appears each folder is a single file, similar to mbox format. Yeah, it does matter. Dovecot doesn't allow you to mix-and-match different mail storage formats very easily. You may be able to set something like that up with the Namespaces support in Dovecot, but the easiest thing to do is to convert your mbox files into Maildir. ~Kyle -- Liberty means responsibility. That is why most men dread it. -- George Bernard Shaw pgpzxd5qajxuF.pgp Description: PGP signature
Re: [Dovecot] use of deliver from procmail advisable?
On Wednesday, August 15 at 06:08 PM, quoth martin f krafft:
This is exactly how I used to have it but then the need for
a vacation autoresponse to the From: address (as opposed to
Return-Path) arose and I had to switch to procmail:
http://dovecot.org/list/dovecot/2007-August/024766.html
Before that, I was using spamc with --pipe-to, but always had a bad
feeling about that, since the manpage says:
Note that there is a very slight chance mail will be lost here,
because if the fork-and-exec fails there’s no place to put the
mail message.
HEH. Using procmail has the same (or worse) problems, and has the very
real potential of lost mail if anything goes wrong (including
fork-and-exec failure). It's *possible* to configure procmail to
handle failure better, but it's cumbersome and irritating. You have to
follow every rule with something like this:
:0 e
{
EXITCODE=75
HOST
}
I mean, do you know and understand the importance of the ORGMAIL
variable to procmail? Most procmail users don't, and it's critical to
understanding procmail's behavior when faced with errors.
For example, if you run out of disk quota, procmail will happily fail
and exit with a successful return code, informing your MTA that the
message was delivered when in fact it was not. Or it will be
delivered, but to the ORGMAIL box (which most admins have never heard
of), and it will simply *appear* to be gone. But if it can't deliver
to ORGMAIL... according to the man page the mail will bounce back to
the sender, but do you know what that means? Will procmail exit with
a non-zero exit code? Or will procmail generate its own bounce and
feed it to sendmail?
and my message to SA-users on this was never answered[0].
0. http://marc.info/?l=spamassassin-usersm=115185095923772w=2
fork-and-exec can fail if:
1. There is insufficient memory to fork
2. There are insufficient PIDs to fork
3. The limit on the user's number of processes has been reached
(RLIMIT_NPROC)
4. The total number of bytes in the environment is too large
5. Search permission is denied on a component of the path prefix
6. The script's interpreter cannot be found (if you're exec'ing a
script)
7. The file to be executed, or the script interpreter, is not a
regular file
8. Execute permission is denied for the file to be executed
9. The file system is mounted 'noexec'
10. The executable is an invalid executable, or attempts to name
more than one interpreter
11. An I/O error occurred while reading the executable
12. The executable listed, or the script interpreter, is a
directory rather than a program
13. There is a loop in the script interpreter (e.g. the interpreter
specified in the script is itself a script whose interpreter is
the first script).
14. The limit on the user's number of open files has been reached.
15. The name of the executable is too long.
16. The limit on the operating system's number of open files has
been reached.
17. The file to be executed does not exist.
18. The executable is for a different architecture.
19. Insufficient kernel memory is available.
20. A component of the path to the executable, or to the script's
interpreter, is not a directory.
21. The filesystem is mounted 'nosuid', the user is not the
superuser, and the file has an SUID or SGID bit set.
22. The process is being traced, the user is not the superuser, and
the file has an SUID or SGID bit set.
23. The executable was open for writing by another process.
That's the SHORT list for why things might go wrong. Depending on the
binary that you ask spamc to run, waiting for it to finish may be
impossible (for example, if it does a fork/exec to handle things), so
no, the spamc client cannot be expected to simply wait and tell you
with certainty that the mail was delivered properly in all cases.
Now I am using procmail and at least now that failure will cause
postfix to defer a message.
It may not, unless you're significantly more in-tune with procmail
than your average sysadmin.
~Kyle
-- If I had only known, I would have been a locksmith.
-- Albert Einstein
pgpsUYrNVLotI.pgp
Description: PGP signature
Re: [Dovecot] use of deliver from procmail advisable?
On Tuesday, August 14 at 07:03 PM, quoth martin f krafft: It also understands the 'seive' filter language (an alternative to procmail). I don't consider it an alternative to procmail because you cannot pass mail to external programmes, like spamassassin or vacation. Sure, sieve has its own vacation module, but I find that to be rather limited. See this thread: http://dovecot.org/list/dovecot/2007-August/024686.html Well, the whole point of sieve, I believe, is to make it something that an admin would want to let arbitrary users modify on their own recognizance, and the ability to specify arbitrary programs to run would be just *asking* to be hacked. But I understand your desire there. At that point, I'd say the heck with it, just use procmail and don't worry about deliver. That's what I do, and my users are all impressed with dovecot's snappiness, irregardless of (or despite) its indexing. In a word: don't solve a problem that doesn't exist. :) ~Kyle -- Once a government is committed to the principle of silencing the voice of opposition, it has only one way to go, and that is down the path of increasingly repressive measures, until it becomes a source of terror to all its citizens and creates a country where everyone lives in fear. -- Harry Truman pgpaa5Z9vHfpR.pgp Description: PGP signature
Re: [Dovecot] use of deliver from procmail advisable?
On Tuesday, August 14 at 02:28 PM, quoth Charles Marcus: Well, the whole point of sieve, I believe, is to make it something that an admin would want to let arbitrary users modify on their own recognizance, and the ability to specify arbitrary programs to run would be just *asking* to be hacked. Wouldn't a decent, secure alternative to procmail be sieve+amavisd-new? Not for what the OP wants to do, which is use arbitrary vacation programs. ~Kyle -- Carpe per diem---seize the check. -- Robin Williams pgpN7MVmVAK37.pgp Description: PGP signature
Re: [Dovecot] use of deliver from procmail advisable?
On Tuesday, August 14 at 10:40 PM, quoth Timo Sirainen: On Tue, 2007-08-14 at 19:03 +0200, martin f krafft wrote: Does anyone have hard facts on how much the server process loses if it encounters a folder with an index inconsistency? With v1.0 deliver doesn't do much since it doesn't update cache file. With v1.1 it does and it's much more useful. But even then if your IMAP client is going to download the message body immediately anyway it probably doesn't matter much. dovecot.index inconsistency checking and updating is really cheap. It's done all the time anyway with maildir. In other words, on a maildir-based system, using deliver does NOT significantly improve performance? ~Kyle -- Democracy must be something more than two wolves and a sheep voting on what to have for dinner. -- James Bovard pgpE4w9YPRowC.pgp Description: PGP signature
Re: [Dovecot] Inbox re-initialized
On Saturday, August 11 at 02:26 AM, quoth Joshua, C.S. Chen: I just replaced my mail server's UW-imap with dovecot yesterday. For outlook express users, when they come in the office, the first time they launch outlook express, they found that the previous Inbox is re-transmitted. You need to migrate all the message UID's. Without them, dovecot generates new ones, which makes the client think that all the messages are new. (so some users got 3X or even 4X the past e-mails). I can't imagine why that would be... ~Kyle -- To invent, you need a good imagination and a pile of junk. -- Thomas Jefferson pgpOneb4q0S5F.pgp Description: PGP signature
Re: [Dovecot] auth_cache_...
On Thursday, July 26 at 03:15 PM, quoth Frank Elsner: auth_cache_size = 0, does it mean cache can grow ad infinitum or does it mean no cache at all ? Well... # Authentication cache size in kilobytes. 0 means it's disabled. That's pretty direct. 0 means no cache at all. I'm not seeing where there might be a contradiction or a confusion. ~Kyle -- The real problem is not whether machines think, but whether men do. -- B. F. Skinner signature.asc Description: Digital signature
Re: [Dovecot] avoiding getting maildir cur/ folder contents?
On Thursday, July 19 at 12:11 PM, quoth Johannes Berg: On Wed, 2007-07-18 at 15:37 -0600, Kyle Wheeler wrote: I don't suppose improving your IO is an option? I find turning on dir_hashing in ext3, and mounting my maildirs with noatime, both significantly improve my performance without requiring me to buy new hardware. I already have both of those options turned on, the problem is that it's running in Xen and I can't get off that right now. Hmmm, so how many mails in one directory are we talking about? If Xen is the only thing you can't get out of, have you considered using an alternate filesystem? For example, I hear ReiserFS does very well with Maildir (though I haven't tried it on anything big). ~Kyle -- Men think epilepsy divine, merely because they do not understand it. But if they called everything divine which they do not understand, why, there would be no end of divine things. -- Hippocrates pgp9dBQ614KIT.pgp Description: PGP signature
Re: [Dovecot] quota calculating
On Monday, June 25 at 10:10 AM, quoth Scott Zahn: Currently where I work we use qmail. Every time a message is delivered, qmail seems to walk through the user's entire maildir in order to calculate quota usage. Qmail doesn't support quota calculations. The quota calculation you describe must be being performed by some add-on to qmail (such as vpopmail's vdelivermail. If it's not doing a good job, you may want to look into using a different add-on. ~Kyle -- Disobedience, in the eyes of any one who has read history, is man's original virtue. It is through disobedience that progress has been made, through disobedience and through rebellion. -- Oscar Wilde pgpKTe55GTtyY.pgp Description: PGP signature
[Dovecot] unexpected error message (maildir_sync_index)
Hello, I recently saw these errors in my log, and one of my users complained that Dovecot disconnected him unexpectedly. Any idea what might have happened? dovecot: Error: IMAP([EMAIL PROTECTED]): file maildir-sync.c: line 1075 (maildir_sync_index): assertion failed: (uid prev_uid) dovecot: Error: IMAP([EMAIL PROTECTED]): Raw backtrace: /usr/local/libexec/dovecot/imap [0x80b6501] - /usr/local/libexec/dovecot/imap [0x80b641c] - /usr/local/libexec/dovecot/imap(maildir_sync_index+0x902) [0x8069042] - /usr/local/libexec/dovecot/imap [0x8069243] - /usr/local/libexec/dovecot/imap(maildir_storage_sync_init+0x48) [0x8069428] - /usr/local/libexec/dovecot/imap(imap_sync_init+0x40) [0x8062590] - /usr/local/libexec/dovecot/imap(cmd_sync+0x71) [0x80626d1] - /usr/local/libexec/dovecot/imap(cmd_noop+0x26) [0x8059cc6] - /usr/local/libexec/dovecot/imap [0x805b878] - /usr/local/libexec/dovecot/imap [0x805b907] - /usr/local/libexec/dovecot/imap(_client_input+0x6c) [0x805bfbc] - /usr/local/libexec/dovecot/imap(io_loop_handler_run+0x120) [0x80bc7e0] - /usr/local/libexec/dovecot/imap(io_loop_run+0x28) [0x80bb808] - /usr/local/libexec/dovecot/imap(main+0x4b0) [0x8063fd0] - /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) [0xb7e7eea8] - /usr/local/libexec/dovecot/imap [0x8056a11] dovecot: Error: child 8418 (imap) killed with signal 6 ~Kyle -- All we have to decide is what to do with the time that is given us. -- Gandalf the Grey pgpaQEEymB9pl.pgp Description: PGP signature
Re: [Dovecot] [IDEA] Re: wierd ssl-parameters.dat regeneration error
On Sunday, May 13 at 04:47 PM, quoth Timo Sirainen: On Mon, 2007-05-07 at 12:46 -0600, Kyle Wheeler wrote: When Dovecot regenerates its ssl-parameters.dat file, there is a race condition between the multiple instances of dovecot, because they all regenerate the file in the same compile-time-defined $statedir directory: /usr/local/var/lib/dovecot. This should fix it: http://dovecot.org/list/dovecot-cvs/2007-May/008756.html *Almost*; my several dovecot instances don't all use the same base_dir, because they got mad and started fighting over who got to put a dict-server and master.pid file in there, and who got to play with the login directory (I haven't figured out how to get them all to share an auth server yet). With this patch, some of my dovecot instances just won't regenerate the ssl-parameters.dat file (though they will fail more gracefully, which is nice). ~Kyle -- Liberty means responsibility. That is why most men dread it. -- George Bernard Shaw pgp8mROWuFQmA.pgp Description: PGP signature
Re: [Dovecot] [IDEA] Re: wierd ssl-parameters.dat regeneration error
On Sunday, May 13 at 09:57 AM, quoth Kyle Wheeler: On Sunday, May 13 at 06:27 PM, quoth Timo Sirainen: Only one of them needs to regenerate the file. The rest of them should just copy it to their login_dir. Hmm, okay. How do they know when the file is fully regenerated? Oh! I think I see; file_try_lock() blocks until the lock is obtained or fails, correct? ~Kyle -- Reliability means never having to say you're sorry. -- Dr. Daniel J. Bernstein pgphkLhn9ppKo.pgp Description: PGP signature
Re: [Dovecot] logging IMAP size?
On Saturday, May 12 at 03:28 AM, quoth Armijn Hemel: I work for a small hosting company. We host websites and people's mail on a shared hosting server. Mail is accessed through POP3 or IMAP. I need to be able to measure the IMAP size for accounting, since people pay for the bandwidth and I need to be able to justify it in my reports :-) It can already be done with POP3, where there is a size value: dovecot: May 07 09:11:54 Info: POP3([EMAIL PROTECTED]): Disconnected: Logged out top=0/0, retr=2/29469, del=2/2, size=29433 I think you're looking for this: http://www.dovecot.org/list/dovecot/2007-March/020478.html ~Kyle -- Testing can show the presence of errors, but not their absence. -- E. W. Dijkstra pgp6y6g113W73.pgp Description: PGP signature
