Re: Cert for ip range?
On Wed, Nov 27, 2019 at 11:31 AM Aki Tuomi
wrote:
>
> > On 27/11/2019 21:28 Mark Moseley via dovecot
> wrote:
> >
> >
> > On Tue, Nov 26, 2019 at 11:22 PM Aki Tuomi via dovecot <
> dovecot@dovecot.org> wrote:
> > >
> > > On 21.11.2019 23.57, Marc Roos via dovecot wrote:
> > > > Is it possible to configure a network for a cert instead of an ip?
> > > >
> > > > Something like this:
> > > >
> > > > local 192.0.2.0 {
> > > > ssl_cert = > > > ssl_key = > > > }
> > > >
> > > > Or
> > > >
> > > > local 192.0.2.0/24 (http://192.0.2.0/24) {
> > > > ssl_cert = > > > ssl_key = > > > }
> > > >
> > > > https://wiki.dovecot.org/SSL/DovecotConfiguration
> > > >
> > > >
> > > >
> > >
> > > Local part supports that.
> > >
> > > Aki
> >
> >
> > On the same topic (though I can start a new thread if preferable), it
> doesn't appear that you can use wildcards/patterns in the 'local' name,
> unless I'm missing something--which is quite likely.
> >
> > If it's not possible currently, can I suggest adding that as a feature?
> That is, instead of having to list out all the various SNI hostnames that a
> cert should be used for (e.g. "local pop3.example.com (
> http://pop3.example.com) imap.example.com (http://imap.example.com)
> pops.example.com (http://pops.example.com) pop.example.com (
> http://pop.example.com) {" -- and on and on), it'd be handy to be
> able to just say "local *.example.com (http://example.com) {" and call it
> a day. I imagine there'd be a bit of a slowdown, since you'd have to loop
> through patterns on each connection (instead of what I assume is a hash
> lookup), esp for people with significant amounts of 'local's.
> >
>
> Actually that is supported, but you need to use v2.2.35 or later.
>
>
Ha, it literally *never* fails (that there's some option I've overlooked 10
times, before asking on the list)
'local' vs 'local_name'. Never noticed the difference before in the docs.
Might be worth adding a blurb in
https://wiki.dovecot.org/SSL/DovecotConfiguration that 'local_name' takes
'*'-style wildcard (at least in the beginning of the hostname). I'll resume
my embarrassed silence now. :)
Re: Cert for ip range?
On Tue, Nov 26, 2019 at 11:22 PM Aki Tuomi via dovecot
wrote:
>
> On 21.11.2019 23.57, Marc Roos via dovecot wrote:
> > Is it possible to configure a network for a cert instead of an ip?
> >
> > Something like this:
> >
> > local 192.0.2.0 {
> > ssl_cert = > ssl_key = > }
> >
> > Or
> >
> > local 192.0.2.0/24 {
> > ssl_cert = > ssl_key = > }
> >
> > https://wiki.dovecot.org/SSL/DovecotConfiguration
> >
> >
> >
>
> Local part supports that.
>
> Aki
>
On the same topic (though I can start a new thread if preferable), it
doesn't appear that you can use wildcards/patterns in the 'local' name,
unless I'm missing something--which is quite likely.
If it's not possible currently, can I suggest adding that as a feature?
That is, instead of having to list out all the various SNI hostnames that a
cert should be used for (e.g. "local pop3.example.com imap.example.com
pops.example.com pop.example.com {" -- and on and on), it'd be handy
to be able to just say "local *.example.com {" and call it a day. I imagine
there'd be a bit of a slowdown, since you'd have to loop through patterns
on each connection (instead of what I assume is a hash lookup), esp for
people with significant amounts of 'local's.
Re: Quota and maildir does not work with subfolders of INBOX
On Mon, Sep 9, 2019 at 8:57 PM Niels Kobschätzki via dovecot < dovecot@dovecot.org> wrote: > On 9/9/19 6:18 PM, @lbutlr via dovecot wrote: > > On 9 Sep 2019, at 09:27, Niels Kobschätzki > wrote: > >> The moment I remove those folders, the size gets calculated correctly. > Unfortunately those folders are generated by some clients automatically > afaik (like .INBOX.Trash) > >> That sounds like a misconfiguration of the IMAP client. Someone has > gone in and improperly set INBOX as the IMAP path Prefix in their MUA. > > The thing is that it worked before. Even when the user misconfigured > their client in such a way, the quota-plugin shouldn't just throw some > dice to get to a arbitrarily high quota the user has used instead of the > right amount. > > > I used to have this problem with some users until I implemented repeated > and consistent application of a clue bat. > > Some users is in my case (as far as I guess) like 0.5% > > > I don’t know of a server-side setting to prevent users from screwing up > this setting, but maybe? > > Wouldn't that break existing accounts? > > Does it sound like this? https://www.dovecot.nl/pipermail/dovecot/2019-March/115214.html If so, in a direct email, Timo suggested using the 'count' quota (instead of the Maildir++ quota). I've not yet been able to test that to verify, due to the large amount of mailboxes and the reliance on maildirsize file for some of our tools.
Re: [BUG?] Double quota calulation when special folder is present
On Tue, Apr 9, 2019 at 9:52 PM Aki Tuomi wrote:
>
> On 10 April 2019 05:00 Mark Moseley via dovecot
> wrote:
>
>
> On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
>
> I've bisected this down to this commit:
>
> git diff
> 76
Re: Using userdb/passdb data in director_username_hash
On Fri, Apr 12, 2019 at 11:14 AM Aki Tuomi
wrote:
>
> > On 12 April 2019 21:09 Mark Moseley via dovecot
> wrote:
> >
> >
> > TL;DR:
> >
> > Can director_username_hash use %{userdb:...} or %{passdb:...} ?
> >
> >
> >
> > This is on Ubuntu Precise, running dovecot 2.2.36. It's a fully
> production, director-ized env, so assume everything is working correctly.
> Happy to post doveconf if it's relevant but wanted to ask a general
> question first.
> >
> > I was curious if there's a way to get userdb/passdb data
> into director_username_hash. Currently, we've got default hashing (on %u).
> I'm returning a SQL field called 'real_username' (the owner of the mailbox,
> so almost never the same as %u). I'd like (for mdbox reasons) to hash on
> that rather than %u.
> >
> > My test SQL is returning (this is just a chunk -- it's duplicated for
> testing):
> > UserName AS userdb_real_username, UserName AS real_username
> >
> > I can see in my director boxes that it's at least picking up the latter:
> >
> > passdb out: PASS1user=tesbox@mailbox.comproxy=yreal_username=testuser
> >
> > Is it possible to inject 'real_username' into director_username_hash?
> That is, I'd rather hash on 'testuser' instead of 'testbed'.
> >
> > I've been trying different permutations on my director boxes with no
> luck.
> >
> > director_username_hash = %{userdb:real_username}
> > director_username_hash = %{passdb:real_username}
> > director_username_hash = %{userdb:userdb_real_username}
> > director_username_hash = %{passdb:userdb_real_username}
> >
> > With any of those settings, every mailbox gets hashed to the same
> backend, so I'm guessing it's not getting anything useful (i.e. probably
> resolving to the same empty string and hashing on that -- or perhaps is
> just hashing on the literal string, e.g. "%{userdb:real_username}" ).
> >
> > And I'm not even sure if director_username_hash has access to any
> passdb/userdb data. Is there a debug setting that would show what string
> director is using to do the hashing?
> >
> > Current debug settings are:
> >
> > auth_debug = yes
> > auth_debug_passwords = yes
> > mail_debug = yes
> >
> > but not a peep as to the string that director is hashing on.
>
> Hi!
>
> The only variables usable on director_username_hashing are (u)ser,
> user(n)ame and (d)omain.
>
>
Ok, thanks for the info!
Using userdb/passdb data in director_username_hash
TL;DR:
Can director_username_hash use %{userdb:...} or %{passdb:...} ?
This is on Ubuntu Precise, running dovecot 2.2.36. It's a fully production,
director-ized env, so assume everything is working correctly. Happy to post
doveconf if it's relevant but wanted to ask a general question first.
I was curious if there's a way to get userdb/passdb data
into director_username_hash. Currently, we've got default hashing (on %u).
I'm returning a SQL field called 'real_username' (the owner of the mailbox,
so almost never the same as %u). I'd like (for mdbox reasons) to hash on
that rather than %u.
My test SQL is returning (this is just a chunk -- it's duplicated for
testing):
UserName AS userdb_real_username, UserName AS real_username
I can see in my director boxes that it's at least picking up the latter:
passdb out: PASS 1 user=tes...@mailbox.com proxy=y real_username=testuser
Is it possible to inject 'real_username' into director_username_hash? That
is, I'd rather hash on 'testuser' instead of 'testbed'.
I've been trying different permutations on my director boxes with no luck.
director_username_hash = %{userdb:real_username}
director_username_hash = %{passdb:real_username}
director_username_hash = %{userdb:userdb_real_username}
director_username_hash = %{passdb:userdb_real_username}
With any of those settings, every mailbox gets hashed to the same backend,
so I'm guessing it's not getting anything useful (i.e. probably resolving
to the same empty string and hashing on that -- or perhaps is just hashing
on the literal string, e.g. "%{userdb:real_username}" ).
And I'm not even sure if director_username_hash has access to any
passdb/userdb data. Is there a debug setting that would show what string
director is using to do the hashing?
Current debug settings are:
auth_debug = yes
auth_debug_passwords = yes
mail_debug = yes
but not a peep as to the string that director is hashing on.
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley wrote:
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley
> wrote:
>
>> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
>> throw the word 'bug' around
>> (since 99 times out of 100, it's not -- it's almost always the config),
>> but I can't think of any way
>> that this could be a config issue, esp when the pre-2.2.34 version works
>> as expected.
>>
>> I noticed during troubleshooting that dovecot errors out if I try to
>> create a subfolder called
>> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
>> (i.e. a folder called
>> INBOX.INBOX.SomethingElse - resulting in a directory called
>> .INBOX.SomethingElse on the
>> filesystem, and leading to the problem described below). Is that
>> sub-subfolder creation (where
>> the top level subfolder matches the namespace name) supposed to be
>> allowed? It seems
>> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
>> INBOX.SomethingElse (as
>> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
>> (i.e.
>> INBOX.INBOX.SomethingElse) would be blocked as well.
>>
>>
>> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
>> dovecot@dovecot.org> wrote:
>>
>>> Hello,
>>>
>>> we're operating dovecot on a small server. Some years ago, we migrated
>>> from courier IMAP to dovecot. Therefore, we defined our default
>>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>>> this in some migration docs those days. Generally, everything worked as
>>> expected.
>>>
>>> Our only namespace is configured like this:
>>>
>>> namespace inbox {
>>> separator = .
>>> prefix = INBOX.
>>> inbox = yes
>>> }
>>>
>>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>>> I suppose this is correct. But I found a special corner case today when
>>> it comes to quota calculation.
>>>
>>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>>> of foo) exists, the whole mailbox is counted twice in quota
>>> recalculation. Just creating .INBOX does nothing but a subfolder
>>> triggers the problem.
>>>
>>> This is my shell view (replaced username and file path and deleted
>>> unnecessary debug output)
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 14697 17
>>> $ maildirmake .INBOX.foo
>>> $ sudo doveadm -D quota recalc -u
>>> [...]
>>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/home/.../test
>>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>>> indexpvt=, control=, inbox=/home/.../test, alt=
>>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>>> inbox=, alt=
>>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>>> unset - skipping
>>> doveadm(): Debug: Quota root User quota: Recalculated relative
>>> rules with bytes=268435456 count=0. Now grace=26843545
>>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>>> /home/.../test: mode=0700 gid=default
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 29394 34
>>>
>>>
>>> So the used quota has exactly been doubled by just creating an empty
>>> subfolder.
>>>
>>> Do you have any pointers for fixing my configuration or is this a bug in
>>> dovecot?
>>>
>>>
>> I coincidentally resurrected a months-old thread with this same issue a
>> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
>> 2.2.36.
>>
>> The original poster (who also narrowed it down to something in 2.2.34)
>> mentioned a workaround that does indeed work, namely setting
>> mailbox_list_index=no:
>>
>> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>>
>> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
>> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
>> or src/lib-storage/list/mailbox-list-fs-iter.c or
>> src/lib-storage/list/mailbox-list-index-iter.c
>> or src/lib-storage/list/mailbox-list-index.c?
>>
>> The latter few have some added strcmp's against "INBOX". Then again,
>> there's a lot of new code in the diffs under src/lib-storage that
>> references INBOX specifically.
>>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
I've bisected this down to this commit:
git diff
7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
b/src/lib-storage/list/mailbox-list-index-iter.c
index c9afc7a..49cd941 100644
--- a/src/lib-storage/list/mailbox-list-index-iter.c
+++ b/src/lib-storage/list/mailbox-list-index-iter.c
@@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
mailbox_list_index_iterate_context *ctx)
if
Re: Where to report (potential) Dovecot bugs
On Tue, Apr 9, 2019 at 2:35 PM John Fawcett via dovecot wrote: > On 09/04/2019 22:03, Mark Moseley via dovecot wrote: > > I'm curious if this is still the right place to report potential bugs > > with Dovecot. > > > > Is there a Dovecot bug tracker somewhere? > > https://www.dovecot.org/bugreport.html > > Yes, I was aware of that. I probably should've said: I'm curious if this is still the right place to report potential bugs, or is the wiki out-of-date?
Where to report (potential) Dovecot bugs
I'm curious if this is still the right place to report potential bugs with Dovecot. Is there a Dovecot bug tracker somewhere?
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley wrote:
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
>> Hello,
>>
>> we're operating dovecot on a small server. Some years ago, we migrated
>> from courier IMAP to dovecot. Therefore, we defined our default
>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>> this in some migration docs those days. Generally, everything worked as
>> expected.
>>
>> Our only namespace is configured like this:
>>
>> namespace inbox {
>> separator = .
>> prefix = INBOX.
>> inbox = yes
>> }
>>
>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>> I suppose this is correct. But I found a special corner case today when
>> it comes to quota calculation.
>>
>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>> of foo) exists, the whole mailbox is counted twice in quota
>> recalculation. Just creating .INBOX does nothing but a subfolder
>> triggers the problem.
>>
>> This is my shell view (replaced username and file path and deleted
>> unnecessary debug output)
>>
>> $ cat maildirsize
>> 268435456S
>> 14697 17
>> $ maildirmake .INBOX.foo
>> $ sudo doveadm -D quota recalc -u
>> [...]
>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/home/.../test
>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>> indexpvt=, control=, inbox=/home/.../test, alt=
>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt=
>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>> unset - skipping
>> doveadm(): Debug: Quota root User quota: Recalculated relative
>> rules with bytes=268435456 count=0. Now grace=26843545
>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>> /home/.../test: mode=0700 gid=default
>>
>> $ cat maildirsize
>> 268435456S
>> 29394 34
>>
>>
>> So the used quota has exactly been doubled by just creating an empty
>> subfolder.
>>
>> Do you have any pointers for fixing my configuration or is this a bug in
>> dovecot?
>>
>>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
Can the Dovecot team confirm whether this is indeed a bug or not? I've not
yet been able to test 2.3.x to see if the problem exists there as well.
[BUG?] Double quota calulation when special folder is present
Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw
the word 'bug' around
(since 99 times out of 100, it's not -- it's almost always the config), but
I can't think of any way
that this could be a config issue, esp when the pre-2.2.34 version works as
expected.
I noticed during troubleshooting that dovecot errors out if I try to create
a subfolder called
'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
(i.e. a folder called
INBOX.INBOX.SomethingElse - resulting in a directory called
.INBOX.SomethingElse on the
filesystem, and leading to the problem described below). Is that
sub-subfolder creation (where
the top level subfolder matches the namespace name) supposed to be allowed?
It seems
odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
INBOX.SomethingElse (as
a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
(i.e.
INBOX.INBOX.SomethingElse) would be blocked as well.
On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot
wrote:
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
I coincidentally resurrected a months-old thread with this same issue a few
days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.
The original poster (who also narrowed it down to something in 2.2.34)
mentioned a workaround that does indeed work, namely setting
mailbox_list_index=no:
> doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
or src/lib-storage/list/mailbox-list-fs-iter.c or
src/lib-storage/list/mailbox-list-index-iter.c
or src/lib-storage/list/mailbox-list-index.c?
The latter few have some added strcmp's against "INBOX". Then again,
there's a lot of new code in the diffs under src/lib-storage that
references INBOX specifically.
Re: Double quota calulation when special folder is present
On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot
wrote:
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
I coincidentally resurrected a months-old thread with this same issue a few
days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.
The original poster (who also narrowed it down to something in 2.2.34)
mentioned a workaround that does indeed work, namely setting
mailbox_list_index=no:
> doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
or src/lib-storage/list/mailbox-list-fs-iter.c or
src/lib-storage/list/mailbox-list-index-iter.c
or src/lib-storage/list/mailbox-list-index.c?
The latter few have some added strcmp's against "INBOX". Then again,
there's a lot of new code in the diffs under src/lib-storage that
references INBOX specifically.
Re: Inbox quota usage doubled when mailbox_list_index enabled, under some circumstances
On Tue, Aug 14, 2018 at 12:31 PM Chris Dillon
wrote:
> I’ve had the opportunity to test the same configuration with a fresh build
> of the git master branch (2.4.devel) and the issue also occurs there. I
> see that "mailbox_list_index = yes" is now enabled by default. It can
> still be disabled via "mailbox_list_index = no" which allows the quota to
> be calculated correctly.
>
> ==
> root@ubuntu1804:~# dovecot -n
> # 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS
> # Hostname: ubuntu1804
> mail_location = maildir:~/Maildir
> mail_plugins = quota
> namespace inbox {
> inbox = yes
> location =
> prefix = INBOX.
> separator = .
> }
> passdb {
> driver = pam
> }
> plugin {
> quota = maildir:Mailbox
> }
> userdb {
> driver = passwd
> }
> ==
>
> (To summarize from my previous message -- other than "mailbox_list_index =
> yes", second most important part of replication is that there is at least
> one email in the real inbox and at least one sub-folder named "INBOX" in
> maildir format)
>
> root@ubuntu1804:~# ls -ld
> /home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S
> /home/myuser/Maildir/.INBOX.Test/
> -rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28
> '/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S'
> drwxrwxr-x 5 myuser myuser 87 Aug 14 18:56
> /home/myuser/Maildir/.INBOX.Test/
> =
>
> (In the following example usage is doubled, there is only one email)
>
> root@ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u
> myuser
> Quota name TypeValue Limit
> %
> MailboxSTORAGE 24830 -
> 0
> MailboxMESSAGE 2 -
> 0
> ==
>
> (In the following example it works correctly with mailbox_list_index
> disabled)
>
> root@ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota recalc -u
> myuser; doveadm quota get -u myuser
> Quota name TypeValue Limit
> %
> MailboxSTORAGE 12415 -
> 0
> MailboxMESSAGE 1 -
> 0
> ==
>
> Best Regards
We recently upgraded from 2.2.32 to 2.2.36.1 and ran into the same issue as
above. I was about to start compiling all the intermediate versions to
pinpoint where it started and happened upon this post first. We are in the
same boat where some users have *sub*folders starting with INBOX, resulting
in directory names like ".INBOX.Event" (and our namespace root is INBOX)
and quota calculation double-counts INBOX's vsize.
Just to be clear too: At least in my case, it's *not* double counting,
e.g., INBOX.Event. But if the above conditions are met, it's
double-counting *INBOX*, because it now sees a folder called INBOX.INBOX
(which does *not* exist on the filesystem).
I hadn't gotten as far as Chris did (this just bubbled up today), but his
solution works here too, i.e. passing -o 'mailbox_list_index=no' to
doveadm quota recalc.
Also, if I rename the directories to something else (e.g. from above,
rename ".INBOX.Event" to ".notINBOX.Event"), a quota recalc works just
fine. The presence of a directory called .INBOX. is triggering
this.
I'm able to create new subfolders called INBOX. with clients
like Apple Mail, which was a bit surprising (Apple Mail however choked when
I tried to just create a subfolder called 'INBOX'). We've got millions of
mailboxes, so educating users is a non-starter :)
Any fix for this from the dovecot devs?
Re: doveadm import with subfolder oddity
On Mon, Feb 4, 2019 at 1:59 PM Mark Moseley wrote:
> This has got to be something weird in my config. And the standard
> disclaimer of '"happy to post doveconf -n, but wanted to see if this is
> normal first" :)
>
> Background: Ubuntu Xenial, running 2.2.36. Mailbox type is mdbox and I've
> got a period separator in my inbox namespace:
>
> namespace {
> hidden = no
> inbox = yes
> list = yes
> location =
> mailbox Spam {
> auto = no
> autoexpunge = 1 weeks
> special_use = \Junk
> }
> mailbox Trash {
> auto = no
> special_use = \Trash
> }
> prefix = INBOX.
> separator = .
> subscriptions = yes
> type = private
> }
>
> If I do a import for a regular folder under INBOX, it works just fine:
>
> doveadm import -u testbox2@testing.local -U testbox1@testing.local
> mdbox:~/mdbox INBOX all mailbox Sent
>
> ... returns happily, message count gets incremented
>
> If I try to do the same with a subfolder (and a subfolder that most
> definitely exists on both source and destination side), I get an error:
>
> doveadm import -u testbox2@testing.local -U testbox1@testing.local
> mdbox:~/mdbox INBOX all mailbox Sub.Sub1
> doveadm(testbox2@testing.local): Error: remote(10.1.17.98:4000): Mailbox
> Sub.Sub1: Mailbox sync failed: Mailbox doesn't exist: Sub.Sub1
>
> If I use / instead of . in my query, it works:
>
> doveadm import -u testbox2@testing.local -U testbox1@testing.local
> mdbox:~/mdbox INBOX all mailbox Sub/Sub1
>
> ... returns happily and message count gets incremented.
>
> Since we're using '.' as our separator, that was a bit unexpected :)
>
> Ironically, if I'm doing a IMAPc 'import', it works just fine with a query
> of 'all mailbox Sub.Sub1'. It's only when importing from a local src and
> local dest (i.e. source_location == mdbox:~/mdbox) that it fails. With
> source_location set to 'imapc:', it works. I imagine that's due to using
> straight IMAP on the source side.
>
> Likely a misconfig on my part? Expected behavior?
>
> I can see in the strace that the error is triggered when doveadm is
> looking at the source mailbox. It looks
> for mdbox/mailboxes/Sub.Sub1/dbox-Mails first, then falls back
> to mdbox/mailboxes/Sub/Sub1/dbox-Mails (which it finds). Then a little bit
> later in the strace, it again looks for mdbox/mailboxes/Sub.Sub1/dbox-Mails
> (which it doesn't find) but doesn't try mdbox/mailboxes/Sub/Sub1/dbox-Mails
> this time, and then spits out 'Mailbox Sub.Sub1: Mailbox sync failed:
> Mailbox doesn't exist: Sub.Sub1'. With a query of 'all mailbox Sub/Sub1',
> the stat() is for mdbox/mailboxes/Sub/Sub1/dbox-Mails which it finds and
> uses happily.
>
> Having to substitute the '.'s for '/'s in the 'mailbox' part of the query
> isn't an awful workaround, but it very much feels like I'm doing something
> wrong. This is a production setup, so everything else is otherwise working
> fine. But I've only just begun working with 'doveadm import', so I might be
> turning up some issues with my config.
>
> Thanks! Sorry I'm so verbose :)
>
Has anyone else seen similar behavior? It's hardly a tough kludge to regex
's/\./\//g' (even if it makes for an ugly regex), but it seems like
something's not quite right.
Re: Doveadm service as non-root user
On Mon, Feb 4, 2019 at 12:04 PM Mark Moseley wrote: > > On Fri, Feb 1, 2019 at 11:37 PM Aki Tuomi > wrote: > >> >> On 01 February 2019 at 23:16 Mark Moseley < moseleym...@gmail.com> >> wrote: >> >> >> Running: Ubuntu xenial, dovecot 2.2.36 >> >> I've been working on moving our user base from maildir to mdbox and >> trying >> to come up with solutions for things like moving emails around. In the >> past, with maildir, our support guys could just mv the files around and >> done. For mdbox, I've been working on getting things set up to use >> doveadm. >> >> One weirdness I've seen is that in imports (i.e. doveadm import), mail >> gets >> copied correctly but the resulting files are left with root ownership (I >> don't have 'service doveadm' 'user' set, so I guess it defaults to root). >> It's typically new m.* files as well as the dovecot.list.index >> and dovecot.list.index.log files. >> >> Looking at strace, no chown is done on them, nor was there setuid. The >> import had no trouble finding the correct user in the db, so I know that >> it >> knows the correct UID (I can see it just fine in debug logs too). And it >> will happily import to existing m.* files with no permissions issues (but >> considering it's running as root, I wouldn't expect it to). >> >> I've seen this using 'import' via IMAPc as well as with both src and dest >> on the same server. I can see this behavior in both scenarios. We have a >> single shared UID for mail, so especially in that "src/dest on same >> server" >> case, it's not a matter of UID-mismatch. >> >> It's a director setup, so all doveadm commands are coming through the >> director. If I run the import directly on the backend (which obviously >> would be a bad idea in real life), the ownership of new m.* files seems >> to >> be correct (I can see it setuid'ing to the correct UID from userdb in >> strace). If I run the import on the director, I can get a new root-owned >> file every time it rolls over to the next m.* file. >> >> Two questions: >> >> * Is that a bug? Is this expected behavior? Seems like the expected thing >> would be to use the UID from userdb and either do a setuid (just like >> running 'doveadm import' locally did) or chown'ing any new files to the >> correct UID. I always always assume misconfiguration (vs bug, since it's >> almost never a bug) but I'm baffled on this one. >> >> * I see that it's possible to set a user for service doveadm and the wiki >> even suggests that it's a good idea in a single UID setup. If there are >> no >> mailboxes with any other UIDs, *will setting 'service doveadm' to the >> same >> UID possibly break anything*? I can't think of why it would, but I want >> to >> be duly diligent. Plus I'm a little leery about closing the door to ever >> having additional UIDs for mailboxes. >> >> Happy to provide 'doveconf -n' but wanted to check first, before spending >> 15 minutes gently obfuscating it :) >> >> >> Can you try >> >> doveadm import -U victim -u victim ... ? >> --- >> Aki Tuomi >> > > > Is that to test a generic 'import from sourceUser to dest user' (i.e. > victim isn't literally the same in both -u and -U) or are you looking for a > test where 'sourceUser' is the same email account as the destination? > > I just want to make sure I'm understanding right. The original tests (that > result in the root-owned files) were all -U userA -u userB (i.e. different > email accounts for src and dest), if you're asking about the former. > > If you're asking about the latter, I ran that and got the same result, a > root-owned dovecot.list.index.log and dovecot.list.index and freshly > created m.* files. The message count in the destination mailbox increases > by the right number (no surprise since it's running as root), so the import > itself is working. > > I should add that in both cases (different src/dest email account and same > src/dest), the import works ok -- or at least increments the count in the > index. It just leaves the email account in a broken state. Re-chown'ing it > to the current permissions makes it happy again and the newly imported > messages show up. > Any chance Aki's hit-the-nail-on-the-head answer got lost in the ether due to the DMARC snafu? :) I'm going forward for now with running doveadm as the unix user that owns all the mailbox, so no urgency, but it's still a bit perplexing (and if it's a bug, good to stomp out).
