Re: dovecot Digest, Vol 154, Issue 14
Thanks, Hajo
This mostly works, but we can’t seem to send an arbitrary mailbox name with
UserDB, we have to initialize it in the config:
"namespace/inbox/mailbox=Junk namespace/inbox/mailbox/Junk/name"="Spam"
seems to initialize the folder somehow so that the subsequent
"namespace/inbox/mailbox/Junk/auto"="subscribe"
and
"namespace/inbox/mailbox/Junk/special_use"="\Junk"
have something to act upon.
(the first line doesn't work quite properly, however - it ends up
causing a folder named "" to show up)
we've noticed that if we set a very minimal config for the mailbox in
the main config (that wouldn't cause the mailbox to be autocreated),
that this also initializes the folder name allowing for subsequent auto
and subscribe directives to be processed for it:
namespace inbox {
inbox = yes
mailbox "Spam" {
auto = no
}
}
is there some userdb response that will simply initialize the folder
(without doing any name remapping) in the same way that the above config
lines do?
we have tried a few things based off Hajo's first line, but nothing we
guess at seems to do the trick. ie:
"namespace/inbox/mailbox"="Spam"
"namespace/inbox/mailbox=Spam"=“Spam"
>
> --
>
> Message: 2
> Date: Tue, 9 Feb 2016 23:33:53 +0100
> From: Hajo Locke <hajo.lo...@gmx.de>
> To: dovecot@dovecot.org
> Subject: Re: Per-user special folder?
> Message-ID: <56ba6951.8010...@gmx.de>
> Content-Type: text/plain; charset=windows-1252; format=flowed
>
> Hello,
>
> Am 09.02.2016 um 22:28 schrieb Richard Platel:
>> Hi
>>
>> It's possible to mark some folders as special use for IMAP in the config
>> like:
>>
>> namespace inbox {
>> mailbox Spam {
>> special_use = \Junk
>> }
>> }
>>
>>
>> Our webmail allows users to use an arbitrary folder for Spam, and we have
>> this settings and we'd like to return it in from our UserDB (which is a
>> custom dict proxy).
>>
>> For testing were able to set a namespace parameter like "separator" by
>> returning:
>> "namespace/inbox/separator" : "=",
>>
>> from UserDB, but can't figure out a way to set mailbox settings,
>>
>> "namespace/inbox/mailbox Spam/special_use" : "\Junk", for example doesn't
>> work.
>>
>> Is there a way to return this setting from a UserDB query?
>>
> we use this a lot with userdb to allow individual folders marked as special.
> your userbd-query should return something like this:
>
> namespace/inbox/mailbox=Junk namespace/inbox/mailbox/Junk/name=Spam
> namespace/inbox/mailbox/Junk/auto=subscribe
> namespace/inbox/mailbox/Junk/special_use=\Junk
> Spam is visible name in this case.
>
> Hajo
Per-user special folder?
Hi
It's possible to mark some folders as special use for IMAP in the config like:
namespace inbox {
mailbox Spam {
special_use = \Junk
}
}
Our webmail allows users to use an arbitrary folder for Spam, and we have this
settings and we'd like to return it in from our UserDB (which is a custom dict
proxy).
For testing were able to set a namespace parameter like "separator" by
returning:
"namespace/inbox/separator" : "=",
from UserDB, but can't figure out a way to set mailbox settings,
"namespace/inbox/mailbox Spam/special_use" : "\Junk", for example doesn't work.
Is there a way to return this setting from a UserDB query?
[Dovecot] maildir compressed message fix patch
When a compressed maildir message has a bad S= size in its filename it puts the
user in an unrecoverable state, since maildir's do_fix_size function just does
a stat() on the maildir file and saves the compressed size in the filename.
This (quick, rough, barely tested) patch addresses this issue, it's
inefficient, but we're already in a hopefully rare emergency situation.
--- maildir-mail.c 2014-02-11 22:23:37.0 +
+++ maildir-mail.c.new 2014-04-24 20:41:25.0 +
@@ -8,6 +8,7 @@
#include maildir-filename.h
#include maildir-uidlist.h
#include maildir-sync.h
+#include compression.h
#include stdio.h
#include stdlib.h
@@ -640,6 +641,10 @@
{
const char *fname, *newpath, *extra, *info, *dir;
struct stat st;
+ const struct stat * stp;
+ const struct compression_handler * handler;
+ struct istream * fstream;
+ struct istream * cstream;
fname = strrchr(path, '/');
i_assert(fname != NULL);
@@ -650,13 +655,29 @@
info = strchr(fname, MAILDIR_INFO_SEP);
if (info == NULL) info = ;
+ fstream = i_stream_create_file(path, 1024);
+ handler = compression_detect_handler(fstream);
+ if (handler != NULL handler-create_istream != NULL)
+ {
+cstream = handler-create_istream(fstream, TRUE);
+if (i_stream_stat(cstream, TRUE, stp) 0)
+{
+ return -1;
+}
+st = *stp; /* dumb copy */
+i_stream_unref(cstream);
+ }
+ else
+ {
if (stat(path, st) 0) {
if (errno == ENOENT)
return 0;
mail_storage_set_critical(mbox-storage-storage,
stat(%s) failed: %m, path);
return -1;
- }
+ }
+ }
+ i_stream_unref(fstream);
newpath = t_strdup_printf(%s/%s,S=%PRIuUOFF_T%s, dir,
t_strdup_until(fname, extra),
[Dovecot] zlib maildir reindex broken
I posted to the list about this a while ago but never got a response, I have a bit more information now. Dovecot 2.2.12 and other 2.2 versions are broken when using zlib and maildir. If messages are re-indexed, the INTERNALDATE of all messages is set to the time the re-index is done. The problem seems to be in src/plugins/zlib/zlib-plugin.c in the function zlib_mail_cache_open. During a reindex maildir_mail_get_received_date() does an i_stream_stat on the the i_stream_seekable stream created in zlib_mail_cache_open, but this istream does not know about the original maildir message file and always returns the current time for the file's stat times. This is also broken on initial index, but if mail is indexed when it's received (ours isn't) it coincidentally gets the right time.
[Dovecot] dovecot with maildir not using mtime on reindex
Hi.
It seems that dovecot is using the current time, not a maildir file's mtime for
INTERNALDATE when a message is re-indexed:
$ cd Index
$ rm -rf .INBOX
$ cd ../Maildir/cur
$ stat *
File: `1392914632.P54451Q0M08633.smtpin01,S=2215,W=2249:2,'
Size: 960 Blocks: 8 IO Block: 1048576 regular file
Device: 36h/54d Inode: 11132959Links: 1
Access: (0600/-rw---) Uid: (8/mail) Gid: (8/mail)
Access: 2012-01-01 00:00:00.0 +
Modify: 2012-01-01 00:00:00.0 +
Change: 2014-02-20 16:46:20.0 +
Birth: -
$ telnet imap01 143
Trying 10.5.45.1...
Connected to imap01.dev.firefly.tucows.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
AUTH=PLAIN] Dovecot ready.
A LOGIN rpla...@ff-dev.com
A OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND
URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED
I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH
LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA] Logged in
A SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags
permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1265835133] UIDs valid
* OK [UIDNEXT 4548] Predicted next UID
A OK [READ-WRITE] Select completed (0.035 secs).
A FETCH 1:* FULL
* 1 FETCH (FLAGS () INTERNALDATE 20-Feb-2014 16:59:51 + RFC822.SIZE 2249
ENVELOPE (Thu, 20 Feb 2014 11:43:50 -0500 Test message ((Richard Platel
NIL rplatel tucows.com)) ((Richard Platel NIL rplatel tucows.com))
((Richard Platel NIL rplatel tucows.com)) ((NIL NIL rplatel
ff-dev.com)) NIL NIL NIL bed6eb46-cc88-4bdb-ae8b-98c1cdead...@tucows.com)
BODY (text plain (charset us-ascii) NIL NIL 7bit 23 4))
A OK Fetch completed.
A LOGOUT
* BYE Logging out
A OK Logout completed.
Connection closed by foreign host.
$ date
Thu Feb 20 16:59:58 UTC 2014
Stracing the imap process, it seems dovecot does not stat the message file at
all. Performing the above with an old dovecot 1 server yields the expected
result, the INTERNALDATE of the message is the file's mtime.
$ dovecot -n -c /he/dovecot/conf/dovecot.conf
# 2.2.10.3: /he/dovecot/conf/dovecot.conf
# OS: Linux 3.4.46-dom0-2.0.0 x86_64 Debian 7.0
debug_log_path = syslog
disable_plaintext_auth = no
first_valid_uid = 8
info_log_path = syslog
lock_method = dotlock
log_timestamp =
mail_fsync = always
mail_gid = mail
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = zlib quota tc_mail_log notify tc_proc
mail_temp_dir = /var/run/dovecot_tmp
mail_uid = mail
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date ihave
mmap_disable = yes
namespace inbox {
inbox = yes
location =
prefix =
}
passdb {
args = host=localhost port=1143
username=%L{user}::%L{service}::%L{rip}::%L{session}
driver = imap
}
plugin {
antispam_backend = pipe
antispam_debug_target = syslog
antispam_pipe_program = /he/dovecot/utils/he_spamtrain.pl
antispam_pipe_program_args = --user=%u
antispam_pipe_program_notspam_arg = --falsepositive
antispam_pipe_program_spam_arg = --missed
antispam_pipe_tmpdir = /var/run/dovecot_as_tmp
antispam_signature_missing = move
antispam_spam = Spam;Inbox.Spam;INBOX.Spam;Junk;INBOX.Junk
antispam_trash_pattern_ignorecase = trash
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
flag_change append
mail_log_fields = uid box msgid flags hetag
memcached_servers = 10.5.47.223,10.5.47.222
zlib_save = gz
zlib_save_level = 6
}
protocols = imap pop3
service anvil {
unix_listener anvil-auth-penalty {
mode = 00
}
}
service imap-login {
inet_listener imap {
address = 0
}
inet_listener imaps {
port = 0
}
process_limit = 29
process_min_avail = 14
service_count = 0
}
service imap-postlogin {
executable = script-login -d /he/dovecot/utils/post_login.sh
}
service imap {
executable = imap imap-postlogin
process_limit = 1270
vsz_limit = 0
}
service pop3-login {
inet_listener pop3 {
address = 0
}
inet_listener pop3s {
port = 0
}
process_limit = 29
process_min_avail = 14
service_count = 0
}
service pop3-postlogin {
executable = script-login -d /he/dovecot/utils/post_login.sh
}
service pop3 {
executable = pop3 pop3-postlogin
process_limit = 206
vsz_limit = 512 M
}
ssl = no
userdb {
args = /he/dovecot/conf/dovecot-tc-dict-auth.conf
driver = dict
}
verbose_proctitle = yes
protocol imap {
mail_max_userip_connections = 30
mail_plugins = zlib quota tc_mail_log notify tc_proc imap_quota antispam
}
protocol pop3
Re: [Dovecot] dovecot with maildir not using mtime on reindex
Furthermore: it seems the behaviour is correct (mtime is used for internaldate)
if the message is not compressed.
On Feb 20, 2014, at 12:09 PM, Richard Platel rpla...@tucows.com wrote:
Hi.
It seems that dovecot is using the current time, not a maildir file's mtime
for INTERNALDATE when a message is re-indexed:
$ cd Index
$ rm -rf .INBOX
$ cd ../Maildir/cur
$ stat *
File: `1392914632.P54451Q0M08633.smtpin01,S=2215,W=2249:2,'
Size: 960 Blocks: 8 IO Block: 1048576 regular file
Device: 36h/54d Inode: 11132959Links: 1
Access: (0600/-rw---) Uid: (8/mail) Gid: (8/mail)
Access: 2012-01-01 00:00:00.0 +
Modify: 2012-01-01 00:00:00.0 +
Change: 2014-02-20 16:46:20.0 +
Birth: -
$ telnet imap01 143
Trying 10.5.45.1...
Connected to imap01.dev.firefly.tucows.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
AUTH=PLAIN] Dovecot ready.
A LOGIN rpla...@ff-dev.com
A OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA] Logged in
A SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags
permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1265835133] UIDs valid
* OK [UIDNEXT 4548] Predicted next UID
A OK [READ-WRITE] Select completed (0.035 secs).
A FETCH 1:* FULL
* 1 FETCH (FLAGS () INTERNALDATE 20-Feb-2014 16:59:51 + RFC822.SIZE
2249 ENVELOPE (Thu, 20 Feb 2014 11:43:50 -0500 Test message ((Richard
Platel NIL rplatel tucows.com)) ((Richard Platel NIL rplatel
tucows.com)) ((Richard Platel NIL rplatel tucows.com)) ((NIL NIL
rplatel ff-dev.com)) NIL NIL NIL
bed6eb46-cc88-4bdb-ae8b-98c1cdead...@tucows.com) BODY (text plain
(charset us-ascii) NIL NIL 7bit 23 4))
A OK Fetch completed.
A LOGOUT
* BYE Logging out
A OK Logout completed.
Connection closed by foreign host.
$ date
Thu Feb 20 16:59:58 UTC 2014
Stracing the imap process, it seems dovecot does not stat the message file at
all. Performing the above with an old dovecot 1 server yields the expected
result, the INTERNALDATE of the message is the file's mtime.
$ dovecot -n -c /he/dovecot/conf/dovecot.conf
# 2.2.10.3: /he/dovecot/conf/dovecot.conf
# OS: Linux 3.4.46-dom0-2.0.0 x86_64 Debian 7.0
debug_log_path = syslog
disable_plaintext_auth = no
first_valid_uid = 8
info_log_path = syslog
lock_method = dotlock
log_timestamp =
mail_fsync = always
mail_gid = mail
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = zlib quota tc_mail_log notify tc_proc
mail_temp_dir = /var/run/dovecot_tmp
mail_uid = mail
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
mmap_disable = yes
namespace inbox {
inbox = yes
location =
prefix =
}
passdb {
args = host=localhost port=1143
username=%L{user}::%L{service}::%L{rip}::%L{session}
driver = imap
}
plugin {
antispam_backend = pipe
antispam_debug_target = syslog
antispam_pipe_program = /he/dovecot/utils/he_spamtrain.pl
antispam_pipe_program_args = --user=%u
antispam_pipe_program_notspam_arg = --falsepositive
antispam_pipe_program_spam_arg = --missed
antispam_pipe_tmpdir = /var/run/dovecot_as_tmp
antispam_signature_missing = move
antispam_spam = Spam;Inbox.Spam;INBOX.Spam;Junk;INBOX.Junk
antispam_trash_pattern_ignorecase = trash
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
flag_change append
mail_log_fields = uid box msgid flags hetag
memcached_servers = 10.5.47.223,10.5.47.222
zlib_save = gz
zlib_save_level = 6
}
protocols = imap pop3
service anvil {
unix_listener anvil-auth-penalty {
mode = 00
}
}
service imap-login {
inet_listener imap {
address = 0
}
inet_listener imaps {
port = 0
}
process_limit = 29
process_min_avail = 14
service_count = 0
}
service imap-postlogin {
executable = script-login -d /he/dovecot/utils/post_login.sh
}
service imap {
executable = imap imap-postlogin
process_limit = 1270
vsz_limit = 0
}
service pop3-login {
inet_listener pop3 {
address = 0
}
inet_listener pop3s {
port = 0
}
process_limit = 29
process_min_avail = 14
service_count = 0
}
service pop3-postlogin {
executable = script-login -d /he/dovecot/utils/post_login.sh
}
service pop3 {
executable = pop3 pop3-postlogin
[Dovecot] EAGAIN in dict proxy
Hello,
We’re using a custom program to manage quotas talking to dovecot via the
dovecot dict proxy protocol over a unix socket:
plugin {
quota = dict:User quota::proxy:/var/run/auth_proxy_dovecot/quotasocket:quota
}
Dovecot gets slammed with quota requests periodically, seemingly because
Thunderbird thought it would be a good idea to hardcode having a quota check at
45 seconds past the minute, and thus every Thunderbird client makes several
GETQUOTAROOT requests simultaneously.
This causes dovecot to make many client connections to the quota proxy, and
many of them fail with EAGAIN. In the log we see:
Jan 15 16:52:46 imap25 dovecot: imap(rpla...@tucows.com): Error:
net_connect_unix(/var/run/auth_proxy_dovecot/quotasocket) failed: Resource
temporarily unavailable
And the client gets:
* QUOTAROOT Spam User quota[0d][0a]
* QUOTA User quota ()[0d][0a]
* BAD Internal quota calculation error[0d][0a]
19 OK Getquotaroot completed.[0d][0a]
Thunderbird transparently disconnects and reconnects at this point but
obviously this is not ideal.
Writing some toy programs, I found that even making a forked server with
several processes doing nothing but accept()ing on the listening socket, with a
high number for the listen queue, it’s easy to overwhelm it with simultaneous
clients who then get EAGAIN. If the clients do indeed immediately try again,
they are successful. (An INET listening socket does not seem to have this
problem, incidentally)
All of this is a long-winded way of saying that I believe in
lib-dict/dict-client.c:client_dict_connect() the call to net_connect_unix
should be a call to net_connect_unix_with_retries() with a small timeout.
It would also be useful for us if an INET socket could be used.
Re: [Dovecot] zlib config questions
You only need to add the zlib plugin to mail_plugins once.
As far as I know, there's no indication in the logs that mail is being
compressed, and the filename isn't modified to indicate that it's compressed,
but, of course, the files are zlib data on disk.
One gotcha if you're using maildir is, if you're planning to compress old mail,
make sure that the S= and W= sizes in the filenames are correct, otherwise
dovecot errors and closes the client session, and it can't repair the filename
on it's own.
On Dec 6, 2013, at 12:51 AM, dovecot-requ...@dovecot.org wrote:
Message: 6
Date: Thu, 5 Dec 2013 13:55:40 -0800
From: Terry Barnum te...@dop.com
To: Dovecot Mailing List dovecot@dovecot.org
Subject: [Dovecot] zlib config questions
Message-ID: ef7f46d0-de03-4b11-85ed-d669de78d...@dop.com
Content-Type: text/plain; charset=us-ascii
After nearly running out of space I swapped in larger disks and then saw the
recent threads about zlib compression. Unfortunately I'm still confused after
reading http://wiki2.dovecot.org/Plugins/Zlib.
In order to compress new email being stored do I only need to change
10-mail.conf to this:
mail_plugins = $mail_plugins zlib
and 90-plugin.conf to this:
plugin {
zlib_save_level = 6
zlib_save = gz
...
Or do I need to instead (or also?) add it to the list of plugins in
20-imap.conf?
A simple 'sudo doveadm reload' to enable?
Once enabled, is there an indication in the logs that compression is taking
place?
Do new mail files have a suffix like .Z to indicate they've been compressed?
Any gotchas to be aware of?
I'm running macports dovecot 2.2.5.
Thanks for any help.
-Terry
Terry Barnum
digital OutPost
http://www.dop.com
Re: [Dovecot] dovecot-antispam plugin problem with multiple messages
This seems to fix the issue
--- a/dovecot-antispam-plugin/src/antispam-storage-2.0.c
+++ b/dovecot-antispam-plugin/src/antispam-storage-2.0.c
@@ -91,15 +91,6 @@ antispam_copy(struct mail_save_context *ctx, struct mail *mai
int ret;
bool src_trash, dst_trash;
- if (!ctx-dest_mail) {
- /* always need mail */
- if (!ast-mail)
- ast-mail = mail_alloc(t, MAIL_FETCH_STREAM_HEADER |
- MAIL_FETCH_STREAM_BODY,
- NULL);
- ctx-dest_mail = ast-mail;
- }
-
i_assert(mail-box);
asbox-save_hack = FALSE;
@@ -145,7 +136,7 @@ antispam_copy(struct mail_save_context *ctx, struct mail *ma
else
ret = asbox-cfg-backend-handle_mail(
asbox-cfg, t, ast-backendctx,
- ctx-dest_mail,
+ mail,
move_to_class(asbox-movetype));
/*
On Nov 27, 2013, at 4:17 PM, Richard Platel rpla...@tucows.com wrote:
Hi
With dovecot 2.2.5, and dovecot-antispam built from a recent HEAD pull, when
copying multiple messages to or from a Spam folder, the plugin sends multiple
copies of the first message to the backend. I've tried this with the pipe
and spool2dir backends.
For example with the spool2dir backend, via IMAP doing
A COPY 1:3 Spam
yields 3 copies of message id 1 in the dir:
dev:imap-8.1 rplatel@imap01:/var/run/dovecot_as_tmp$ sudo -u mail md5sum *
28ad0a215eb7ecbd3a814a8a334d85bf
001385586164-rpla...@ff-dev.com-1s
28ad0a215eb7ecbd3a814a8a334d85bf
001385586164-rpla...@ff-dev.com-2s
28ad0a215eb7ecbd3a814a8a334d85bf
001385586164-rpla...@ff-dev.com-3s
I see the same behaviour with the pipe backend, the pipe program is invoked 3
times, but with the same message content.
[Dovecot] dovecot-antispam plugin problem with multiple messages
Hi With dovecot 2.2.5, and dovecot-antispam built from a recent HEAD pull, when copying multiple messages to or from a Spam folder, the plugin sends multiple copies of the first message to the backend. I've tried this with the pipe and spool2dir backends. For example with the spool2dir backend, via IMAP doing A COPY 1:3 Spam yields 3 copies of message id 1 in the dir: dev:imap-8.1 rplatel@imap01:/var/run/dovecot_as_tmp$ sudo -u mail md5sum * 28ad0a215eb7ecbd3a814a8a334d85bf 001385586164-rpla...@ff-dev.com-1s 28ad0a215eb7ecbd3a814a8a334d85bf 001385586164-rpla...@ff-dev.com-2s 28ad0a215eb7ecbd3a814a8a334d85bf 001385586164-rpla...@ff-dev.com-3s I see the same behaviour with the pipe backend, the pipe program is invoked 3 times, but with the same message content.
Re: [Dovecot] fts-solr indexer-worker connects to wrong solr host dovecot-2.2.4
Did some more digging.
The problem is that the fts-solr plugin has a global solr_conn pointer, that
persists between users. I think this patch fixes the problem:
--- a/dovecot/fts_solr_plugin/fts-solr-plugin.c
+++ b/dovecot/fts_solr_plugin/fts-solr-plugin.c
@@ -50,6 +50,13 @@ static void fts_solr_mail_user_create(struct mail_user
*user, const char *env)
{
struct fts_solr_user *fuser;
+ /** solr URL may be different per-user **/
+ if (solr_conn != NULL) {
+ solr_connection_deinit(solr_conn);
+ solr_conn = NULL;
+ }
+ /**/
+
fuser = p_new(user-pool, struct fts_solr_user, 1);
if (fts_solr_plugin_init_settings(user, fuser-set, env) 0) {
/* invalid settings, disabling */
On 2013-10-02, at 3:28 PM, Richard Platel rpla...@tucows.com wrote:
I've confirmed that this problem still exists in 2.2.5
It seems that indexer-worker only init's plugins at startup, so the fts_solr
plugin is holding the url= parameter from the first user.
The problem doesn't happen if the indexer-worker process is idle-killed
between users. A new process starts up with the new user's userdb settings.
I thought I could work around this problem by adjusting indexer-worker's
settings:
service indexer-worker {
service_count = 1
idle_kill = 1
}
but these changes don't seem to have any effect, the indexer-worker process
still hangs around idling after indexing a user, and isn't idle-killed for
upwards of a minute.
Any help?
On 2013-09-27, at 11:46 AM, Richard Platel rpla...@tucows.com wrote:
Hello.
We're setting up fts solr and want to have the solr server host be set
per-user via UserDB.
It looks like if a user connects and fts indexes mail, and then another user
connects and indexes mail, indexer-worker is connecting to the first user's
fts host:
User1, ham...@rp-auth-test.com connects, does a SEARCH for the first time,
indexer-worker gets UserDB settings and correctly indexes mail on ftsvs01:
[...]
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): lookup
shared/userdb/ham...@rp-auth-test.com
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): result:
{uid:8,fts:solr,quota_rule4:Spam:ignore,_session:talk15_590ec6d100042,quota_rule3:Trash:ignore,quota_rule2:*:messages=2684354,quota_rule:*:storage=5242880k,mail:maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/,fts_solr:debug
url=http://ftsvs01:8080/solr/,gid:8}
auth: Debug: userdb out: USER 1 ham...@rp-auth-test.com uid=8
fts=solrquota_rule4=Spam:ignore _session=talk15_590ec6d100042
quota_rule3=Trash:ignorequota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/gid=8
indexer-worker: Debug: auth input: ham...@rp-auth-test.com uid=8 fts=solr
quota_rule4=Spam:ignore _session=talk15_590ec6d100042
quota_rule3=Trash:ignore quota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/ gid=8
indexer-worker: Debug: Added userdb setting:
plugin/_session=talk15_590ec6d100042
indexer-worker: Debug: Added userdb setting: plugin/fts=solr
indexer-worker: Debug: Added userdb setting: plugin/fts_solr=debug
url=http://ftsvs01:8080/solr/
indexer-worker: Debug: Added userdb setting:
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ha
m...@rp-auth-test.com/
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule=*:storage=5242880k
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule2=*:messages=2684354
indexer-worker: Debug: Added userdb setting: plugin/quota_rule3=Trash:ignore
indexer-worker: Debug: Added userdb setting: plugin/quota_rule4=Spam:ignore
indexer-worker(ham...@rp-auth-test.com): Debug: Effective uid=8, gid=8, home=
indexer-worker(ham...@rp-auth-test.com): Debug: Namespace inbox:
type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions
=yes
location=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
indexer-worker(ham...@rp-auth-test.com): Debug: maildir++:
root=/mail/mailstore01/215/573/ham...@rp-auth-test.com,
index=/mail/index01/215/
573/ham...@rp-auth-test.com, indexpvt=, control=,
inbox=/mail/mailstore01/215/573/ham...@rp-auth-test.com, alt=
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache
field: pop3.order
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache
field: binary.parts
indexer-worker(ham...@rp-auth-test.com): Warning: Created dotlock file's
timestamp
Re: [Dovecot] fts-solr indexer-worker connects to wrong solr host dovecot-2.2.4
I've confirmed that this problem still exists in 2.2.5
It seems that indexer-worker only init's plugins at startup, so the fts_solr
plugin is holding the url= parameter from the first user.
The problem doesn't happen if the indexer-worker process is idle-killed between
users. A new process starts up with the new user's userdb settings.
I thought I could work around this problem by adjusting indexer-worker's
settings:
service indexer-worker {
service_count = 1
idle_kill = 1
}
but these changes don't seem to have any effect, the indexer-worker process
still hangs around idling after indexing a user, and isn't idle-killed for
upwards of a minute.
Any help?
On 2013-09-27, at 11:46 AM, Richard Platel rpla...@tucows.com wrote:
Hello.
We're setting up fts solr and want to have the solr server host be set
per-user via UserDB.
It looks like if a user connects and fts indexes mail, and then another user
connects and indexes mail, indexer-worker is connecting to the first user's
fts host:
User1, ham...@rp-auth-test.com connects, does a SEARCH for the first time,
indexer-worker gets UserDB settings and correctly indexes mail on ftsvs01:
[...]
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): lookup
shared/userdb/ham...@rp-auth-test.com
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): result:
{uid:8,fts:solr,quota_rule4:Spam:ignore,_session:talk15_590ec6d100042,quota_rule3:Trash:ignore,quota_rule2:*:messages=2684354,quota_rule:*:storage=5242880k,mail:maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/,fts_solr:debug
url=http://ftsvs01:8080/solr/,gid:8}
auth: Debug: userdb out: USER 1 ham...@rp-auth-test.com uid=8
fts=solrquota_rule4=Spam:ignore _session=talk15_590ec6d100042
quota_rule3=Trash:ignorequota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/gid=8
indexer-worker: Debug: auth input: ham...@rp-auth-test.com uid=8 fts=solr
quota_rule4=Spam:ignore _session=talk15_590ec6d100042
quota_rule3=Trash:ignore quota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/ gid=8
indexer-worker: Debug: Added userdb setting:
plugin/_session=talk15_590ec6d100042
indexer-worker: Debug: Added userdb setting: plugin/fts=solr
indexer-worker: Debug: Added userdb setting: plugin/fts_solr=debug
url=http://ftsvs01:8080/solr/
indexer-worker: Debug: Added userdb setting:
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ha
m...@rp-auth-test.com/
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule=*:storage=5242880k
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule2=*:messages=2684354
indexer-worker: Debug: Added userdb setting: plugin/quota_rule3=Trash:ignore
indexer-worker: Debug: Added userdb setting: plugin/quota_rule4=Spam:ignore
indexer-worker(ham...@rp-auth-test.com): Debug: Effective uid=8, gid=8, home=
indexer-worker(ham...@rp-auth-test.com): Debug: Namespace inbox:
type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions
=yes
location=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
indexer-worker(ham...@rp-auth-test.com): Debug: maildir++:
root=/mail/mailstore01/215/573/ham...@rp-auth-test.com,
index=/mail/index01/215/
573/ham...@rp-auth-test.com, indexpvt=, control=,
inbox=/mail/mailstore01/215/573/ham...@rp-auth-test.com, alt=
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache field:
pop3.order
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache field:
binary.parts
indexer-worker(ham...@rp-auth-test.com): Warning: Created dotlock file's
timestamp is different than current time (1380294685 vs 1380294612
): /mail/index01/215/573/ham...@rp-auth-test.com/.INBOX/dovecot.index.log
indexer-worker(ham...@rp-auth-test.com): Debug: http-client: request [POST
http://ftsvs01:8080/solr/update]: Submitted
[...]
User1 index finishes and imap searches against ftsvs01
[...]
imap(ham...@rp-auth-test.com): Debug: http-client: request [GET
http://ftsvs01:8080/solr/select?fl=uid,scorerows=2sort=uid+ascq=(hdr:%22moo%22+OR+body:%22moo%22)fq=%2Bbox:42faee1f735b1e52b321386e9ade+%2Buser:%22ham...@rp-auth-test.com%22]:
Submitted
[...]
User2 gr...@rp-auth-test.com connects and does a SEARCH, index worker gets
gets UserDB settings, including fts host ftsvs02, but connects to ftsvs01
(also note index-worker initially shows wrong user in loglines)
[...]
auth-worker(2195): Debug: dict(gr...@rp-auth
Re: [Dovecot] Index error copying compressed message
On 2013-09-22, at 12:35 AM, Timo Sirainen t...@iki.fi wrote: On 19.9.2013, at 23.59, Richard Platel rpla...@tucows.com wrote: Dovecot 2.2, with the zlib plugin, I think we're getting bad index entries on IMAP COPY. On copying a message to an empty folder, in the dovecot error log I see: Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: Cached message size smaller than expected (615 971) Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: Corrupted index cache file /mail/index01/434/860/gr...@rp-auth-test.com/.Bup/dovecot.index.cache: Broken physical size for mail UID 0 Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: read() failed: Invalid argument (uid=0) (Note this happens from the copy operation, not a subsequent access. Also note the UID is always 0). UID=0 means that it's trying to get the size for the mail that is still being saved (so not the copy source mail). You mean you can easily reproduce this simply by copying a mail to a newly created folder? I couldn't. Try if you can still reproduce it with a smaller config, especially removing non-zlib plugins. This was indeed a plugin configuration problem, thanks.
[Dovecot] fts-solr indexer-worker connects to wrong solr host dovecot-2.2.4
Hello.
We're setting up fts solr and want to have the solr server host be set per-user
via UserDB.
It looks like if a user connects and fts indexes mail, and then another user
connects and indexes mail, indexer-worker is connecting to the first user's fts
host:
User1, ham...@rp-auth-test.com connects, does a SEARCH for the first time,
indexer-worker gets UserDB settings and correctly indexes mail on ftsvs01:
[...]
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): lookup
shared/userdb/ham...@rp-auth-test.com
auth-worker(2195): Debug: dict(ham...@rp-auth-test.com): result:
{uid:8,fts:solr,quota_rule4:Spam:ignore,_session:talk15_590ec6d100042,quota_rule3:Trash:ignore,quota_rule2:*:messages=2684354,quota_rule:*:storage=5242880k,mail:maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/,fts_solr:debug
url=http://ftsvs01:8080/solr/,gid:8}
auth: Debug: userdb out: USER 1 ham...@rp-auth-test.com uid=8
fts=solrquota_rule4=Spam:ignore _session=talk15_590ec6d100042
quota_rule3=Trash:ignorequota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/gid=8
indexer-worker: Debug: auth input: ham...@rp-auth-test.com uid=8 fts=solr
quota_rule4=Spam:ignore _session=talk15_590ec6d100042 quota_rule3=Trash:ignore
quota_rule2=*:messages=2684354 quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
fts_solr=debug url=http://ftsvs01:8080/solr/ gid=8
indexer-worker: Debug: Added userdb setting:
plugin/_session=talk15_590ec6d100042
indexer-worker: Debug: Added userdb setting: plugin/fts=solr
indexer-worker: Debug: Added userdb setting: plugin/fts_solr=debug
url=http://ftsvs01:8080/solr/
indexer-worker: Debug: Added userdb setting:
mail=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ha
m...@rp-auth-test.com/
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule=*:storage=5242880k
indexer-worker: Debug: Added userdb setting:
plugin/quota_rule2=*:messages=2684354
indexer-worker: Debug: Added userdb setting: plugin/quota_rule3=Trash:ignore
indexer-worker: Debug: Added userdb setting: plugin/quota_rule4=Spam:ignore
indexer-worker(ham...@rp-auth-test.com): Debug: Effective uid=8, gid=8, home=
indexer-worker(ham...@rp-auth-test.com): Debug: Namespace inbox: type=private,
prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions
=yes
location=maildir:/mail/mailstore01/215/573/ham...@rp-auth-test.com/:INDEX=/mail/index01/215/573/ham...@rp-auth-test.com/
indexer-worker(ham...@rp-auth-test.com): Debug: maildir++:
root=/mail/mailstore01/215/573/ham...@rp-auth-test.com, index=/mail/index01/215/
573/ham...@rp-auth-test.com, indexpvt=, control=,
inbox=/mail/mailstore01/215/573/ham...@rp-auth-test.com, alt=
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache field:
pop3.order
indexer-worker(ham...@rp-auth-test.com): Debug: Ignoring unknown cache field:
binary.parts
indexer-worker(ham...@rp-auth-test.com): Warning: Created dotlock file's
timestamp is different than current time (1380294685 vs 1380294612
): /mail/index01/215/573/ham...@rp-auth-test.com/.INBOX/dovecot.index.log
indexer-worker(ham...@rp-auth-test.com): Debug: http-client: request [POST
http://ftsvs01:8080/solr/update]: Submitted
[...]
User1 index finishes and imap searches against ftsvs01
[...]
imap(ham...@rp-auth-test.com): Debug: http-client: request [GET
http://ftsvs01:8080/solr/select?fl=uid,scorerows=2sort=uid+ascq=(hdr:%22moo%22+OR+body:%22moo%22)fq=%2Bbox:42faee1f735b1e52b321386e9ade+%2Buser:%22ham...@rp-auth-test.com%22]:
Submitted
[...]
User2 gr...@rp-auth-test.com connects and does a SEARCH, index worker gets gets
UserDB settings, including fts host ftsvs02, but connects to ftsvs01 (also note
index-worker initially shows wrong user in loglines)
[...]
auth-worker(2195): Debug: dict(gr...@rp-auth-test.com): lookup
shared/userdb/gr...@rp-auth-test.com
auth-worker(2195): Debug: dict(gr...@rp-auth-test.com): result:
{uid:8,fts:solr,quota_rule4:Spam:ignore,_session:cow80_609fed761,quota_rule3:Trash:ignore,quota_rule2:*:messages=2684354,quota_rule:*:storage=5242880k,mail:maildir:/mail/mailstore01/812/023/gr...@rp-auth-test.com/:INDEX=/mail/index01/812/023/gr...@rp-auth-test.com/,fts_solr:debug
url=http://ftsvs02:8080/solr/,gid:8}
auth: Debug: userdb out: USER 2 gr...@rp-auth-test.com uid=8
fts=solrquota_rule4=Spam:ignore _session=cow80_609fed761
quota_rule3=Trash:ignorequota_rule2=*:messages=2684354
quota_rule=*:storage=5242880k
mail=maildir:/mail/mailstore01/812/023/gr...@rp-auth-test.com/:INDEX=/mail/index01/812/023/gr...@rp-auth-test.com/
fts_solr=debug
[Dovecot] Index error copying compressed message
Hi.
Dovecot 2.2, with the zlib plugin, I think we're getting bad index entries on
IMAP COPY.
On copying a message to an empty folder, in the dovecot error log I see:
Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: Cached
message size smaller than expected (615 971)
Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: Corrupted
index cache file
/mail/index01/434/860/gr...@rp-auth-test.com/.Bup/dovecot.index.cache: Broken
physical size for mail UID 0
Sep 19 20:34:25 imap01 dovecot: imap(gr...@rp-auth-test.com): Error: read()
failed: Invalid argument (uid=0)
(Note this happens from the copy operation, not a subsequent access. Also note
the UID is always 0).
The filename for the message is:
-rw--- 2862 mail mail 615 Aug 29 15:38
1379622865.M228140P11548.imap01,S=971,W=988:2,S
S= size looks correct:
$ zcat 1379622865.M228140P11548.imap01\,S\=971\,W\=988\:2\,S |wc
17 51 971
doveadm dump says:
$ sudo -u mail doveadm -c /he/dovecot/conf/dovecot.conf dump
/mail/index01/434/860/gr...@rp-auth-test.com/.Bup/
Detected file type: index
-- INDEX: /mail/index01/434/860/gr...@rp-auth-test.com/.Bup//dovecot.index
version .. = 7.3
base header size . = 120
header size .. = 208
record size .. = 12
compat flags . = 1
index id . = 1379605150 (2013-09-19 15:39:10)
flags = 0
uid validity . = 1377629137 (2013-08-27 18:45:37)
next uid . = 14309
messages count ... = 1
seen messages count .. = 1
deleted messages count ... = 0
first recent uid . = 14308
first unseen uid lowwater = 14309
first deleted uid lowwater = 14308
log file seq . = 6
log file tail offset . = 204
log file head offset . = 204
day stamp = 1379548800 (2013-09-19 00:00:00)
day first uid[0] . = 1
day first uid[1] . = 0
day first uid[2] . = 0
day first uid[3] . = 0
day first uid[4] . = 0
day first uid[5] . = 0
day first uid[6] . = 0
day first uid[7] . = 0
-- Extension 0 --
name = maildir
hdr_size = 36
reset_id = 0
record_offset = 0
record_size . = 0
record_align = 0
header
- new_check_time = 2013-09-19 20:34:10
- new_mtime . = 2013-09-19 20:08:51
- new_mtime_nsecs ... = 79253
- cur_check_time = 2013-09-19 20:35:38
- cur_mtime . = 2013-09-19 20:35:38
- cur_mtime_nsecs = 22771
- uidlist_mtime . = 2013-09-19 20:35:38
- uidlist_mtime_nsecs = 254613000
- uidlist_size .. = 1025178
-- Extension 1 --
name = cache
hdr_size = 0
reset_id = 1379605174
record_offset = 8
record_size . = 4
record_align = 4
-- Keywords --
-- CACHE: /mail/index01/434/860/gr...@rp-auth-test.com/.Bup//dovecot.index.cache
major version = 1
minor version = 1
indexid .. = 1379605150 (2013-09-19 15:39:10)
file_seq . = 1379605174 (2013-09-19 15:39:34) (24 compressions)
continued_record_count = 0
record_count . = 0
used_file_size (old) . = 108
deleted_record_count . = 0
field_header_offset .. = 32 (0x88808080 nontranslated)
-- Cache fields --
# Name Type Size Dec Last used
0: flagsbit 4 tmp 2013-09-19 20:07
1: hdr.Message-ID hdr - tmp 2013-09-19 20:07
2: hdr.X-HE-Tag hdr - tmp 2013-09-19 20:07
-- RECORDS: 1
RECORD: seq=1, uid=14308, flags=0x08 (Seen)
- ext 3 cache : 0 ()
$ sudo -u mail dovecot -c /he/dovecot/conf/dovecot.conf -n
# 2.2.4.3 (12e60e803a54+): /he/dovecot/conf/dovecot.conf
# OS: Linux 3.4.46-dom0-2.0.0 x86_64 Debian 7.0
debug_log_path = syslog
disable_plaintext_auth = no
first_valid_uid = 8
info_log_path = syslog
lock_method = dotlock
log_path = /var/run/dovecot/log-fifo
log_timestamp =
mail_fsync = always
mail_gid = mail
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = zlib quota tc_mail_log notify tc_proc stats
mail_uid = mail
maildir_very_dirty_syncs = yes
mmap_disable = yes
namespace inbox {
inbox = yes
location =
prefix =
}
passdb {
args = host=localhost port=1143
username=%L{user}::%L{service}::%L{rip}::%L{session}
driver = imap
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
flag_change append
mail_log_fields = uid box msgid flags hetag
memcached_servers = 10.5.47.223,10.5.47.222
quota = dict:User quota::proxy:/var/run/auth_proxy_dovecot/quotasocket:quota
stats_command_min_time = 1 mins
stats_domain_min_time = 12 hours
stats_ip_min_time = 12 hours
stats_memory_limit = 16 M
stats_refresh = 30 secs
stats_session_min_time = 15 mins
stats_track_cmds = yes
stats_user_min_time = 1 hours
zlib_save = gz
zlib_save_level = 6
}
protocols = imap pop3
service anvil {
[Dovecot] Custom quota setup
(Dovecot 2.2-ee) We have a weird quota requirement, we have file storage that we manage through our own APIs but want that usage to come out of the user's mail quota. The usage is in a maildirsize like file uncreatively called filestoresize in the user's maildir. In the past we've been doing this by modifying the quota plugin and re-compiling, but it seems like it should be possible to do this via configuration. Is there a way to add a quota setting pointing at this file for additional usage (not limits)? The feature is used infrequently so it would probably be acceptable use the dirsize backend, but I can't figure out how to configure that to point at a certain directory. We pass custom quota rules for each user in our userdb, and use a custom dict proxy program, so that program could read the file and pass a setting at log in time too (if, for example, there was a setting that said offset the user's quota usage by X amount) Thanks.
[Dovecot] feature request: IMAP passdb prefetch
It'd be useful for us if the IMAP passdb could be used as a prefetch userdb. The remote IMAP server could respond with something like * OK key=value * OK key=value SEQ OK [CAPABILITY ...] Logged in. Or * OK some JSON key value pairs SEQ OK [CAPABILITY ...] Logged in. Would anyone else find this useful?
[Dovecot] feature request: IMAP passdb prefetch
On 2.8.2013, at 18.20, Richard Platel rpla...@tucows.com wrote: It'd be useful for us if the IMAP passdb could be used as a prefetch userdb. The remote IMAP server could respond with something like * OK key=value * OK key=value SEQ OK [CAPABILITY ...] Logged in. Or * OK some JSON key value pairs SEQ OK [CAPABILITY ...] Logged in. Would anyone else find this useful? Uh. Why not simply something completely different like HTTP-based passdb? I can't find info for HTTP on the wiki, can we set that up as a success/fail passdb? We do our own password auth (so we can support custom hash types, etc).
Re: [Dovecot] Custom quota setup
On 2013-08-02, at 11:34 AM, Timo Sirainen t...@iki.fi wrote: On 2.8.2013, at 18.15, Richard Platel rpla...@tucows.com wrote: We pass custom quota rules for each user in our userdb, and use a custom dict proxy program, so that program could read the file and pass a setting at log in time too (if, for example, there was a setting that said offset the user's quota usage by X amount) That I think would work. That's future development though? No such setting exists now?
Re: [Dovecot] Custom quota setup
On 2013-08-02, at 12:05 PM, Timo Sirainen t...@iki.fi wrote: On 2.8.2013, at 18.56, Richard Platel rpla...@tucows.com wrote: On 2013-08-02, at 11:34 AM, Timo Sirainen t...@iki.fi wrote: On 2.8.2013, at 18.15, Richard Platel rpla...@tucows.com wrote: We pass custom quota rules for each user in our userdb, and use a custom dict proxy program, so that program could read the file and pass a setting at log in time too (if, for example, there was a setting that said offset the user's quota usage by X amount) That I think would work. That's future development though? No such setting exists now? I'm not exactly sure what you thought of, but my idea was simply that you'd add your own dict proxy in the middle which hooks into the GET command, and increases its value by reading the filestoresize. You can already do that by pointing to a UNIX socket different from the normal dict server socket, similar as in http://wiki2.dovecot.org/AuthDatabase/Dict If, for example, the user has 2GB quota, and 750MB of mail, and 500MB of files, we'd like the mail client to show that the user has 2GB quota, and 1.25GB used, .75GB free. I see how via the dict userdb proxy I could reduce the user's quota allowance by the filestorage amount: quota_rule=*:storage=user allowed storage - filestore size, and the client would show the correct free amount, but not the correct total or used amount. Is this what you mean, or is there some other setting or some other dict proxy entirely that I'm missing?
[Dovecot] Renaming not supported across conflicting directory permissions
Is it possible to disable this check in 2.1.7? (without a patch and re-compile?)
Re: [Dovecot] Fixing bad maildir message sizes
the message size, either from the last 4 bytes if compressed, or file size
sub _uncompressed_size_quick($)
{
my ($fn) = @_;
my $gzid = chr(0x1f) . chr(0x8b);
my ($flag, $buf);
open(my $fh, '', $fn) or return undef;
return undef unless (sysread($fh, $flag, 2) == 2);
unless ($flag eq $gzid) # not a compressed file, return the size-on-disk
{
return sysseek($fh, 0, 2);
}
# gziped file, size is in last 4 bytes
return undef unless (sysseek($fh, -4, 2));
return undef unless (sysread($fh, $buf, 4));
return unpack('V', $buf);
}
# get the S= and W= size by reading the whole file.
sub _uncompressed_size($)
{
my $fn = $_[0];
my $fh = IO::Zlib-new($fn, rb) || IO::File-new( $fn);
return undef unless $fh;
my $sz = 0; #uncompressed size
my $wsz = 0; #uncompressed size with /n converted to /r/n
my $read;
my $chunk = 4096; # TODO tune
my $buf;
my $cusp = 0;
while ($read = read($fh, $buf, $chunk))
{
$sz += $read;
$wsz += $read;
$wsz += () = $buf =~ /(?!\r)\n/sg; #count \ns not preceded by an \r
if ($cusp)
{
# last chunk ended with \r and this chunk starts with \n, so we counted
# an /n we shouldn't have above
$wsz -=1 if ($buf =~ /^\n/s);
}
$cusp = $buf =~ /\r$/s;
}
return ($sz, $wsz);
}
unless(caller())
{
my ($maildir, $folder, $mail, $lockbin);
GetOptions(
maildir=s = \$maildir,
folder=s = \$folder,
help = sub {pod2usage(-verbose =1 )},
man = sub {pod2usage(-verbose =2 )})
|| pod2usage(-verbose = 1);
$lockbin = $ENV{MAILDIRLOCK_BIN};
pod2usage(-verbose=1) unless ($lockbin ($maildir || $folder || $mail));
die Could not execute maildirlock [$lockbin] unless -x $lockbin;
system($lockbin /dev/null 21 );
die Could not execute maildirlock [$lockbin], .
maybe you need to set LD_LIBRARY_PATH unless ($? 8) == 1;
if ($maildir)
{
print Fixing maildir: [$maildir]\n;
fixmaildir($maildir, \*STDOUT);
}
if ($folder)
{
print Fixing folder: [$folder]\n;
fixfolder($folder, \*STDOUT);
}
}
1;
On 2013-01-22, at 7:01 AM, Timo Sirainen t...@iki.fi wrote:
On 21.1.2013, at 21.54, Richard Platel rpla...@tucows.com wrote:
As stated in my previous message, we have some old compressed maildir
messages with incorrect sizes in the filename. These messages cause dovecot
2.x problems.
I'm trying to write a script to crawl all our messages, check the actual
message size and if necessary, fix the filename. However, when I do this,
dovecot gives the message a new UID on next login. If I change the filename
in dovecot-uidlist, dovecot still gives a new UID on next login. If I
change dovecot-uidlist and delete the index, then the UID is preserved.
I don't really understand why deleting dovecot.index* would make a difference
here, except maybe as a workaround in case the user has that mailbox
selected, because the filenames could be cached in memory.
http://dovecot.org/tools/maildir-size-fix.pl
http://dovecot.org/tools/maildir-size-check.sh
Those scripts kind of do what you want, except not fully, so it would be nice
to get one fully finished one :) The best way for the script to would would
be to:
* scan through a maildir, figure out what needs to be renamed to what, but
don't actually do it
* lock the maildir with dovecot-uidlist.lock (src/util/maildirlock comes with
dovecot)
* doveadm kick user's imap/pop3 sessions, and even better if it was possible
to kill -9 any pending processes
* rename the files and update dovecot-uidlist
* delete dovecot-uidlist.lock
This separately for each folder.
[Dovecot] Fixing bad maildir message sizes
Hello. As stated in my previous message, we have some old compressed maildir messages with incorrect sizes in the filename. These messages cause dovecot 2.x problems. I'm trying to write a script to crawl all our messages, check the actual message size and if necessary, fix the filename. However, when I do this, dovecot gives the message a new UID on next login. If I change the filename in dovecot-uidlist, dovecot still gives a new UID on next login. If I change dovecot-uidlist and delete the index, then the UID is preserved. Re-indexing our millions of mailboxes is not a great solution for us. Is there a good way to fix this?
[Dovecot] Zlib maildir reindex bug?
Hi Running dovecot 2.1.7 (from debian repo). We have some old compressed maildir messages with the wrong S= size in the filename (our fault). If I delete index files, log in to the mailbox and try to FETCH the bad message, dovecot complains about the incorrect message size, attempts to fix the filename and disconnects the client. However, it changes the filesize to the size of the message on disk, not the message's uncompressed size. For subsequent clients, the FETCH works, but dovecot logs an error (and maybe re-indexes the mailbox?) and the message UID changes. Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Cached message size larger than expected (1612 1556) Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Maildir filename has wrong S value, renamed the file from /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=1612:2,Sab to /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted index cache file /mail/index01/505/236/b...@confidence.com/.Sent Items/dovecot.index.cache: Broken physical size for mail UID 1 Jan 14 18:40:56 imap01 dovecot: imap(b...@confidence.com): Error: read(/mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=1612:2,Sab) failed: Input/output error (uid=1) Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Cached message size smaller than expected (931 1556) Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Maildir filename has wrong S value, renamed the file from /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted index cache file /mail/index01/505/236/b...@confidence.com/.Sent Items/dovecot.index.cache: Broken physical size for mail UID 2 Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Cached message size smaller than expected (931 1556) Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Maildir filename has wrong S value, renamed the file from /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab Jan 14 18:41:25 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted index cache file /mail/index01/505/236/b...@confidence.com/.Sent Items/dovecot.index.cache: Broken physical size for mail UID 2 Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Cached message size smaller than expected (931 1556) Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Maildir filename has wrong S value, renamed the file from /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted index cache file /mail/index01/505/236/b...@confidence.com/.Sent Items/dovecot.index.cache: Broken physical size for mail UID 2 Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Cached message size smaller than expected (931 1556) Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Maildir filename has wrong S value, renamed the file from /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab to /mail/mailstore01/505/236/b...@confidence.com/.Sent Items/cur/1168058702.93517743273399070372.fetchmail01,S=931:2,Sab Jan 14 18:41:53 imap01 dovecot: imap(b...@confidence.com): Error: Corrupted index cache file /mail/index01/505/236/b...@confidence.com/.Sent Items/dovecot.index.cache: Broken physical size for mail UID 2
[Dovecot] Plugin help, number of messages in mailbox
Hi, We use Dovecot for IMAP and POP (but not LDA), we want to do something when a user has an INBOX that becomes empty, or becomes not empty (set a flag in memcached, but that's not really important). I'm writing a plugin (for Dovecot 2.1.7). On mailbox_open() I can use mailbox_get_status() to get a count of messages in the mailbox, and then decrement this in expunge() or increment it in mailbox save_finish() (for IMAP APPEND or COPY commands). However in expunge() and mailbox_save_finish, even after calling the super function, mailbox_get_status doesn't update the number of messages in the mailbox. This is a problem if (for example) there are concurrent POP sessions. Two POP sessions could get all the messages in INBOX, one could logout, calling expunge a few times, eventually causing my plugin to note that the inbox is empty, then our LDA could deliver a message, mark the INBOX not empty, then the other POP session could log out, call expunge and cause my plugin to mark the INBOX empty, when it's not. So in summation: how can a plugin be notified of changes to a mailbox, and then accurately get the real number of messages in that mailbox?
[Dovecot] Custom auth process in dovecot 2
Hi,
I'm trying to upgrade from dovecot-1.1.x to 2.1.7.
We have our own custom auth server process (because we want to do our own
password validation and for other reasons) that listens on a UNIX domain socket
and speaks the dovecot auth protocol.
In dovecot 1.1 we could configure this with
auth external {
socket connect {
master {
path = /var/run/dovecot/auth.sock
}
}
}
as per http://wiki.dovecot.org/MainConfig
I haven't been able to figure out how to do this in 2.1.7, is it possible?
[Dovecot] 1.1.5 abort with mal-formed address in header
Hi
An email with a mal-formed address in a header like:
To:([EMAIL PROTECTED]
(Starting with a comment, with no closing ')' ) causes dovecot 1.1.5
to panic and abort.
This is similar to the problem fixed by: http://hg.dovecot.org/
dovecot-1.1/rev/04fdaa2f831e
This patch seems to resolve the problem:
--- dovecot-1.1.5/src/lib-mail/message-address.c.orig 2008-10-23
22:17:10.243827000 +
+++ dovecot-1.1.5/src/lib-mail/message-address.c2008-10-23
22:20:20.050173000 +
@@ -315,7 +315,7 @@
ctx.fill_missing = fill_missing;
ret = rfc822_skip_lwsp(ctx.parser);
- if (ret == 0) {
+ if (ret = 0) {
/* no addresses */
return NULL;
}
Richard Platel
[EMAIL PROTECTED]
[Dovecot] maildir, zlib and mtime/internal date
On the wiki: http://wiki.dovecot.org/Plugins/Zlib I think there should be a step 5.0. along the lines of get and remember the original message file's mtime And a step 5.4 like Using the touch command or some other method, set the now compressed message's mtime back to the mtime of the original message file. To preserve the message's internal time in case the indices ever get rebuilt. Does this sound right?
Re: [Dovecot] Patch for zlib and maildir for 1.0.13
There was a bug in this patch, causing a seg fault with 0 or 1 byte messages (which shouldn't really happen, but, hey). Patched patch attached. zlib-plugin.c.patch Description: Binary data On Apr 3, 2008, at 3:04 PM, Richard Platel wrote: When using the zlib plugin with maildir and copying with hardlinks, if a compressed message is copied, the 'Z' suffix on the file isn't copied, so the new message isn't uncompressed when it's fetched. I wasn't smart enough to figure out a clean way to carry a file suffix through a copy, so I changed how the zlib-plugin detects if a message is compressed. This patch peeks at the first two bytes of the message looking for the zlib header. -- Richard Platel Tucowszlib-plugin.c.patch
[Dovecot] Patch for zlib and maildir for 1.0.13
When using the zlib plugin with maildir and copying with hardlinks, if a compressed message is copied, the 'Z' suffix on the file isn't copied, so the new message isn't uncompressed when it's fetched. I wasn't smart enough to figure out a clean way to carry a file suffix through a copy, so I changed how the zlib-plugin detects if a message is compressed. This patch peeks at the first two bytes of the message looking for the zlib header. -- Richard Platel Tucows zlib-plugin.c.patch Description: Binary data
[Dovecot] Patch for assertion failure in 1.0.10
Hi FETCHing a largish (~30K) zlib compressed maildir message causes an assertion failure on an attempt to grow the stream buffer to smaller than its current size. I believe the attached patch fixes this. istream-zlib.c.patch Description: Binary data
[Dovecot] Patch for seg fault in 1.0.10
Hi SELECTING a non-existent folder in 1.0.10 with maildir causes a seg fault and client disconnect. I believe this patch fixes the problem. zlib-plugin.c.patch Description: application/applefile zlib-plugin.c.patch Description: Binary data
