Re: [BUG?] Double quota calulation when special folder is present
> On 6 Aug 2019, at 21.08, Mark Moseley via dovecot wrote: > >> >> I've bisected this down to this commit: >> >> git diff >> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa >> >> Prior to this commit, anything updating the quota would do the right thing >> for any .INBOX. folders (i.e. not double count the contents of >> "INBOX" against the quota). After this commit, anything updating quota (new >> mail, quota recalc, etc) does the double counting of INBOX. > > Thank you for the bisect! We'll look into this. > > Hi. I was curious if there were any fixes for this? We're still affected by > this (and I imagine others are too but don't realize it). Thanks! Looks like this happens only with Maildir++ quota. As a workaround you could switch to dict-file or "count" quota. Anyway added to internal tracking as DOP-1336.
Re: [BUG?] Double quota calulation when special folder is present
On Tue, Apr 9, 2019 at 9:52 PM Aki Tuomi wrote:
>
> On 10 April 2019 05:00 Mark Moseley via dovecot
> wrote:
>
>
> On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley < moseleym...@gmail.com>
> wrote:
>
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
>
> I've bisected this down to this commit:
>
> git diff
> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
>
>
> diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
> b/src/lib-storage/list/mailbox-list-index-iter.c
> index c9afc7a..49cd941 100644
> --- a/src/lib-storage/list/mailbox-list-index-iter.c
> +++ b/src/lib-storage/list/mailbox-list-index-iter.c
> @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
> mailbox_list_index_iterate_context *ctx)
>
Re: [BUG?] Double quota calulation when special folder is present
On 10 April 2019 05:00 Mark Moseley via dovecot wrote:
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <
moseleym...@gmail.com> wrote:
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <
moseleym...@gmail.com> wrote:
Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around
(since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way
that this could be a config issue, esp when the pre-2.2.34 version works as expected.
I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called
'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called
INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the
filesystem, and leading to the problem described below). Is that sub-subfolder creation (where
the top level subfolder matches the namespace name) supposed to be allowed? It seems
odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as
a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e.
INBOX.INBOX.SomethingElse) would be blocked as well.
On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
dovecot@dovecot.org> wrote:
Hello,
we're operating dovecot on a small server. Some years ago, we migrated
from courier IMAP to dovecot. Therefore, we defined our default
Namespace "inbox" with prefix "INBOX." to have this compatible. I found
this in some migration docs those days. Generally, everything worked as
expected.
Our only namespace is configured like this:
namespace inbox {
separator = .
prefix = INBOX.
inbox = yes
}
Regularly, there is no folder named INBOX or .INBOX in the file system,
I suppose this is correct. But I found a special corner case today when
it comes to quota calculation.
When - for whatever reason - a folder .INBOX.foo (for arbitrary values
of foo) exists, the whole mailbox is counted twice in quota
recalculation. Just creating .INBOX does nothing but a subfolder
triggers the problem.
This is my shell view (replaced username and file path and deleted
unnecessary debug output)
$ cat maildirsize
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley wrote:
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley
> wrote:
>
>> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
>> throw the word 'bug' around
>> (since 99 times out of 100, it's not -- it's almost always the config),
>> but I can't think of any way
>> that this could be a config issue, esp when the pre-2.2.34 version works
>> as expected.
>>
>> I noticed during troubleshooting that dovecot errors out if I try to
>> create a subfolder called
>> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
>> (i.e. a folder called
>> INBOX.INBOX.SomethingElse - resulting in a directory called
>> .INBOX.SomethingElse on the
>> filesystem, and leading to the problem described below). Is that
>> sub-subfolder creation (where
>> the top level subfolder matches the namespace name) supposed to be
>> allowed? It seems
>> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
>> INBOX.SomethingElse (as
>> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
>> (i.e.
>> INBOX.INBOX.SomethingElse) would be blocked as well.
>>
>>
>> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
>> dovecot@dovecot.org> wrote:
>>
>>> Hello,
>>>
>>> we're operating dovecot on a small server. Some years ago, we migrated
>>> from courier IMAP to dovecot. Therefore, we defined our default
>>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>>> this in some migration docs those days. Generally, everything worked as
>>> expected.
>>>
>>> Our only namespace is configured like this:
>>>
>>> namespace inbox {
>>> separator = .
>>> prefix = INBOX.
>>> inbox = yes
>>> }
>>>
>>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>>> I suppose this is correct. But I found a special corner case today when
>>> it comes to quota calculation.
>>>
>>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>>> of foo) exists, the whole mailbox is counted twice in quota
>>> recalculation. Just creating .INBOX does nothing but a subfolder
>>> triggers the problem.
>>>
>>> This is my shell view (replaced username and file path and deleted
>>> unnecessary debug output)
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 14697 17
>>> $ maildirmake .INBOX.foo
>>> $ sudo doveadm -D quota recalc -u
>>> [...]
>>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/home/.../test
>>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>>> indexpvt=, control=, inbox=/home/.../test, alt=
>>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>>> inbox=, alt=
>>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>>> unset - skipping
>>> doveadm(): Debug: Quota root User quota: Recalculated relative
>>> rules with bytes=268435456 count=0. Now grace=26843545
>>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>>> /home/.../test: mode=0700 gid=default
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 29394 34
>>>
>>>
>>> So the used quota has exactly been doubled by just creating an empty
>>> subfolder.
>>>
>>> Do you have any pointers for fixing my configuration or is this a bug in
>>> dovecot?
>>>
>>>
>> I coincidentally resurrected a months-old thread with this same issue a
>> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
>> 2.2.36.
>>
>> The original poster (who also narrowed it down to something in 2.2.34)
>> mentioned a workaround that does indeed work, namely setting
>> mailbox_list_index=no:
>>
>> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>>
>> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
>> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
>> or src/lib-storage/list/mailbox-list-fs-iter.c or
>> src/lib-storage/list/mailbox-list-index-iter.c
>> or src/lib-storage/list/mailbox-list-index.c?
>>
>> The latter few have some added strcmp's against "INBOX". Then again,
>> there's a lot of new code in the diffs under src/lib-storage that
>> references INBOX specifically.
>>
>
> Can the Dovecot team confirm whether this is indeed a bug or not? I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
I've bisected this down to this commit:
git diff
7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
b/src/lib-storage/list/mailbox-list-index-iter.c
index c9afc7a..49cd941 100644
--- a/src/lib-storage/list/mailbox-list-index-iter.c
+++ b/src/lib-storage/list/mailbox-list-index-iter.c
@@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
mailbox_list_index_iterate_context *ctx)
if
Re: [BUG?] Double quota calulation when special folder is present
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley wrote:
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot@dovecot.org> wrote:
>
>> Hello,
>>
>> we're operating dovecot on a small server. Some years ago, we migrated
>> from courier IMAP to dovecot. Therefore, we defined our default
>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>> this in some migration docs those days. Generally, everything worked as
>> expected.
>>
>> Our only namespace is configured like this:
>>
>> namespace inbox {
>> separator = .
>> prefix = INBOX.
>> inbox = yes
>> }
>>
>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>> I suppose this is correct. But I found a special corner case today when
>> it comes to quota calculation.
>>
>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>> of foo) exists, the whole mailbox is counted twice in quota
>> recalculation. Just creating .INBOX does nothing but a subfolder
>> triggers the problem.
>>
>> This is my shell view (replaced username and file path and deleted
>> unnecessary debug output)
>>
>> $ cat maildirsize
>> 268435456S
>> 14697 17
>> $ maildirmake .INBOX.foo
>> $ sudo doveadm -D quota recalc -u
>> [...]
>> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/home/.../test
>> doveadm(): Debug: maildir++: root=/home/.../test, index=,
>> indexpvt=, control=, inbox=/home/.../test, alt=
>> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
>> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt=
>> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
>> unset - skipping
>> doveadm(): Debug: Quota root User quota: Recalculated relative
>> rules with bytes=268435456 count=0. Now grace=26843545
>> doveadm(): Debug: Namespace INBOX.: Using permissions from
>> /home/.../test: mode=0700 gid=default
>>
>> $ cat maildirsize
>> 268435456S
>> 29394 34
>>
>>
>> So the used quota has exactly been doubled by just creating an empty
>> subfolder.
>>
>> Do you have any pointers for fixing my configuration or is this a bug in
>> dovecot?
>>
>>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
Can the Dovecot team confirm whether this is indeed a bug or not? I've not
yet been able to test 2.3.x to see if the problem exists there as well.
[BUG?] Double quota calulation when special folder is present
Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw
the word 'bug' around
(since 99 times out of 100, it's not -- it's almost always the config), but
I can't think of any way
that this could be a config issue, esp when the pre-2.2.34 version works as
expected.
I noticed during troubleshooting that dovecot errors out if I try to create
a subfolder called
'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
(i.e. a folder called
INBOX.INBOX.SomethingElse - resulting in a directory called
.INBOX.SomethingElse on the
filesystem, and leading to the problem described below). Is that
sub-subfolder creation (where
the top level subfolder matches the namespace name) supposed to be allowed?
It seems
odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
INBOX.SomethingElse (as
a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
(i.e.
INBOX.INBOX.SomethingElse) would be blocked as well.
On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot
wrote:
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
> separator = .
> prefix = INBOX.
> inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct. But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u
> [...]
> doveadm(): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
I coincidentally resurrected a months-old thread with this same issue a few
days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.
The original poster (who also narrowed it down to something in 2.2.34)
mentioned a workaround that does indeed work, namely setting
mailbox_list_index=no:
> doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
or src/lib-storage/list/mailbox-list-fs-iter.c or
src/lib-storage/list/mailbox-list-index-iter.c
or src/lib-storage/list/mailbox-list-index.c?
The latter few have some added strcmp's against "INBOX". Then again,
there's a lot of new code in the diffs under src/lib-storage that
references INBOX specifically.
