Re: [BUG?] Double quota calulation when special folder is present
> On 6 Aug 2019, at 21.08, Mark Moseley via dovecot wrote: > >> >> I've bisected this down to this commit: >> >> git diff >> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa >> >> Prior to this commit, anything updating the quota would do the right thing >> for any .INBOX. folders (i.e. not double count the contents of >> "INBOX" against the quota). After this commit, anything updating quota (new >> mail, quota recalc, etc) does the double counting of INBOX. > > Thank you for the bisect! We'll look into this. > > Hi. I was curious if there were any fixes for this? We're still affected by > this (and I imagine others are too but don't realize it). Thanks! Looks like this happens only with Maildir++ quota. As a workaround you could switch to dict-file or "count" quota. Anyway added to internal tracking as DOP-1336.
Re: [BUG?] Double quota calulation when special folder is present
On Tue, Apr 9, 2019 at 9:52 PM Aki Tuomi wrote:
>
> On 10 April 2019 05:00 Mark Moseley via dovecot
> wrote:
>
>
> On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
>
> I've bisected this down to this commit:
>
> git diff
> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
>
>
> diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
> b/src/lib-storage/list/mailbox-list-index-iter.c
> index c9afc7a..49cd941 100644
> --- a/src/lib-storage/list/mailbox-list-index-iter.c
> +++ b/src/lib-storage/list/mailbox-list-index-iter.c
> @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
> mailbox_list_index_iterate_context *ctx)
>
Re: [BUG?] Double quota calulation when special folder is present
On 10 April 2019 05:00 Mark Moseley via dovecot wrote:
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <
moseleym...@gmail.com> wrote:
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <
moseleym...@gmail.com> wrote:
Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around
(since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way
that this could be a config issue, esp when the pre-2.2.34 version works as expected.
I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called
'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called
INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the
filesystem, and leading to the problem described below). Is that sub-subfolder creation (where
the top level subfolder matches the namespace name) supposed to be allowed? It seems
odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as
a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e.
INBOX.INBOX.SomethingElse) would be blocked as well.
On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
dovecot@dovecot.org> wrote:
Hello,
we're operating dovecot on a small server. Some years ago, we migrated
from courier IMAP to dovecot. Therefore, we defined our default
Namespace "inbox" with prefix "INBOX." to have this compatible. I found
this in some migration docs those days. Generally, everything worked as
expected.
Our only namespace is configured like this:
namespace inbox {
separator = .
prefix = INBOX.
inbox = yes
}
Regularly, there is no folder named INBOX or .INBOX in the file system,
I suppose this is correct. But I found a special corner case today when
it comes to quota calculation.
When - for whatever reason - a folder .INBOX.foo (for arbitrary values
of foo) exists, the whole mailbox is counted twice in quota
recalculation. Just creating .INBOX does nothing but a subfolder
triggers the problem.
This is my shell view (replaced username and file path and deleted
unnecessary debug output)
$ cat maildirsize
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley wrote:
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley
> wrote:
>
>> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
>> throw the word 'bug' around
>> (since 99 times out of 100, it's not -- it's almost always the config),
>> but I can't think of any way
>> that this could be a config issue, esp when the pre-2.2.34 version works
>> as expected.
>>
>> I noticed during troubleshooting that dovecot errors out if I try to
>> create a subfolder called
>> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
>> (i.e. a folder called
>> INBOX.INBOX.SomethingElse - resulting in a directory called
>> .INBOX.SomethingElse on the
>> filesystem, and leading to the problem described below). Is that
>> sub-subfolder creation (where
>> the top level subfolder matches the namespace name) supposed to be
>> allowed? It seems
>> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
>> INBOX.SomethingElse (as
>> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
>> (i.e.
>> INBOX.INBOX.SomethingElse) would be blocked as well.
>>
>>
>> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
>> dovecot@dovecot.org> wrote:
>>
>>> Hello,
>>>
>>> we're operating dovecot on a small server. Some years ago, we migrated
>>> from courier IMAP to dovecot. Therefore, we defined our default
>>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>>> this in some migration docs those days. Generally, everything worked as
>>> expected.
>>>
>>> Our only namespace is configured like this:
>>>
>>> namespace inbox {
>>> separator = .
>>> prefix = INBOX.
>>> inbox = yes
>>> }
>>>
>>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>>> I suppose this is correct. But I found a special corner case today when
>>> it comes to quota calculation.
>>>
>>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>>> of foo) exists, the whole mailbox is counted twice in quota
>>> recalculation. Just creating .INBOX does nothing but a subfolder
>>> triggers the problem.
>>>
>>> This is my shell view (replaced username and file path and deleted
>>> unnecessary debug output)
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 14697 17
>>> $ maildirmake .INBOX.foo
>>> $ sudo doveadm -D quota recalc -u
>>> [...]
>>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/home/.../test
>>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>>> indexpvt=, control=, inbox=/home/.../test, alt=
>>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>>> inbox=, alt=
>>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>>> unset - skipping
>>> doveadm(): Debug: Quota root User quota: Recalculated relative
>>> rules with bytes=268435456 count=0. Now grace=26843545
>>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>>> /home/.../test: mode=0700 gid=default
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 29394 34
>>>
>>>
>>> So the used quota has exactly been doubled by just creating an empty
>>> subfolder.
>>>
>>> Do you have any pointers for fixing my configuration or is this a bug in
>>> dovecot?
>>>
>>>
>> I coincidentally resurrected a months-old thread with this same issue a
>> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
>> 2.2.36.
>>
>> The original poster (who also narrowed it down to something in 2.2.34)
>> mentioned a workaround that does indeed work, namely setting
>> mailbox_list_index=no:
>>
>> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>>
>> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
>> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
>> or src/lib-storage/list/mailbox-list-fs-iter.c or
>> src/lib-storage/list/mailbox-list-index-iter.c
>> or src/lib-storage/list/mailbox-list-index.c?
>>
>> The latter few have some added strcmp's against "INBOX". Then again,
>> there's a lot of new code in the diffs under src/lib-storage that
>> references INBOX specifically.
>>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
I've bisected this down to this commit:
git diff
7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
b/src/lib-storage/list/mailbox-list-index-iter.c
index c9afc7a..49cd941 100644
--- a/src/lib-storage/list/mailbox-list-index-iter.c
+++ b/src/lib-storage/list/mailbox-list-index-iter.c
@@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
mailbox_list_index_iterate_context *ctx)
if
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley wrote:
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
>> Hello,
>>
>> we're operating dovecot on a small server. Some years ago, we migrated
>> from courier IMAP to dovecot. Therefore, we defined our default
>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>> this in some migration docs those days. Generally, everything worked as
>> expected.
>>
>> Our only namespace is configured like this:
>>
>> namespace inbox {
>> separator = .
>> prefix = INBOX.
>> inbox = yes
>> }
>>
>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>> I suppose this is correct. But I found a special corner case today when
>> it comes to quota calculation.
>>
>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>> of foo) exists, the whole mailbox is counted twice in quota
>> recalculation. Just creating .INBOX does nothing but a subfolder
>> triggers the problem.
>>
>> This is my shell view (replaced username and file path and deleted
>> unnecessary debug output)
>>
>> $ cat maildirsize
>> 268435456S
>> 14697 17
>> $ maildirmake .INBOX.foo
>> $ sudo doveadm -D quota recalc -u
>> [...]
>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/home/.../test
>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>> indexpvt=, control=, inbox=/home/.../test, alt=
>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt=
>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>> unset - skipping
>> doveadm(): Debug: Quota root User quota: Recalculated relative
>> rules with bytes=268435456 count=0. Now grace=26843545
>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>> /home/.../test: mode=0700 gid=default
>>
>> $ cat maildirsize
>> 268435456S
>> 29394 34
>>
>>
>> So the used quota has exactly been doubled by just creating an empty
>> subfolder.
>>
>> Do you have any pointers for fixing my configuration or is this a bug in
>> dovecot?
>>
>>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
Can the Dovecot team confirm whether this is indeed a bug or not? I've not
yet been able to test 2.3.x to see if the problem exists there as well.
