Re: [exim] exim maildirsize quota calculation in the face of symlinks
Am 16.02.22 um 14:46 schrieb Maarten van Baarsel via Exim-users:
I'd like to say thanks for the replies, and ask for guidance how to
put this on the feature-addition-list so that it won't be forgotten, I
did find the problem Cyborg was alluding to in a post from a while ago :)
I had a quick look at the code but did not see a fast path to a fix.
Maarten.
Just an idea:
Calc() ...
array = new array();
Loop:
file = openfile( ... );
if file.inode.linkcounter == 1 || array.get( file.inode.id ) ==
NULL {
array.put( file.inode.id );
count file.size
} // skip if it's a known hardlink
(note: the linkcounter check is actually obsolete, because it would not
matter. It's just for illustration. )
IMHO, it's dovecot who's causing this by adding hardlinks to files in
the first place. it's not Exims fault, even if it could avoid this
"miscount" easily.
Best regards,
Marius
OpenPGP_0x048770A738345DD3.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
Hi all, On 10-02-2022 12:10:53, Jeremy Harris via Exim-users wrote: On 10/02/2022 10:55, Maarten van Baarsel via Exim-users wrote: I was surprised by the symlink behaviour so I'm reconsidering the use of the dovecot plugin, but I still wanted to ask whether this behaviour is considered OK. I've read the appendfile docs and I could find anything explicit about symlinks. The coding probably doesn't consider the possibility of multiple paths into a single directory, as afforded by the use of symlinks, at all. I've not checked, though. "OK" is in the eye of the beholder, naturally. Handling the case would likely be a feature-addition. The fastest fix would be to not traverse symlinks, I suppose. I'd like to say thanks for the replies, and ask for guidance how to put this on the feature-addition-list so that it won't be forgotten, I did find the problem Cyborg was alluding to in a post from a while ago :) I had a quick look at the code but did not see a fast path to a fix. Maarten. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
On 10.02.2022 13:06, Jasen Betts via Exim-users wrote: > On 2022-02-10, Maarten van Baarsel via Exim-users wrote: >> Currently, I'm also using a plugin in dovecot: >> >> https://wiki.dovecot.org/Plugins/MailboxAlias >> >> However, this plugin solves the alias problem with a symlink in the >> Maildir directory, which makes Exim count the mails twice when the >> appendfile transport is calculating quota sizes. > > Have you considered using OS-provided per-user disk quotas instead? > > https://www.linux.com/training-tutorials/step-step-using-user-quotas-linux/ > > this is just an idea, I don't know how well dovecot and exim interact with > quota. > Just a note. dovecot is not ready to handle filesystem quotas if control files are also under that quota. It just breaks in various ways (reported on dovecot mailing list). Workaround is to keep control files outside quota enabled filesystem... but that's only a workaround. -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
On Thu, 10 Feb 2022, Jasen Betts via Exim-users wrote: On 2022-02-10, Maarten van Baarsel via Exim-users wrote: Currently, I'm also using a plugin in dovecot: https://wiki.dovecot.org/Plugins/MailboxAlias However, this plugin solves the alias problem with a symlink in the Maildir directory, which makes Exim count the mails twice when the appendfile transport is calculating quota sizes. Have you considered using OS-provided per-user disk quotas instead? https://www.linux.com/training-tutorials/step-step-using-user-quotas-linux/ this is just an idea, I don't know how well dovecot and exim interact with quota. I have used exim (maybe dovecot too, my memory is not clear) with filesystem quotas and would consider doing it again. Quotas are a file-system/partition feature and the details vary with the filesystem type: ext[234], XFS, NFS etc. Not all filesystems support quotas. We used soft quotas (and had much higher hard quotas); this complains but does not stop usage when the soft limit is exceeded. You can lose mail if the hard quota is exceeded and the MTA and an MUA like alpine both try to access the mailbox over NFS. If I were doing this again, I probably wouldn't network the mail-spool partition and would insist that all MUAs used IMAP (or POP). I might share it *readonly*, so that users could "grep" their mailboxes. -- Andrew C. Aitchison Kendal, UK and...@aitchison.me.uk -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
On 10/02/2022 11:48, Cyborg via Exim-users wrote: You will be much more puzzled, when dovecot starts to make hardlinks for your emailfiles and exim starts to count the quota differently than i.e. "du" does. Again, what damnfool software thought doing that was a good notion? In a shared-use directory, even. Sigh. The hack to work around it in Exim would probably be to divide the byte count, and file counter (necessitating change to floating-point), by the link count. Alternatively, deliver *via* Dovecot and give up on Exim knowing anything about quota limit. -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
On 2022-02-10, Maarten van Baarsel via Exim-users wrote: > Currently, I'm also using a plugin in dovecot: > > https://wiki.dovecot.org/Plugins/MailboxAlias > > However, this plugin solves the alias problem with a symlink in the > Maildir directory, which makes Exim count the mails twice when the > appendfile transport is calculating quota sizes. Have you considered using OS-provided per-user disk quotas instead? https://www.linux.com/training-tutorials/step-step-using-user-quotas-linux/ this is just an idea, I don't know how well dovecot and exim interact with quota. -- Jasen. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
Am 10.02.22 um 11:55 schrieb Maarten van Baarsel via Exim-users: I was surprised by the symlink behaviour so I'm reconsidering the use of the dovecot plugin, but I still wanted to ask whether this behaviour is considered OK. I've read the appendfile docs and I could find anything explicit about symlinks. You will be much more puzzled, when dovecot starts to make hardlinks for your emailfiles and exim starts to count the quota differently than i.e. "du" does. here is a report about this behaviour and how to fix the issue: https://marius.bloggt-in-braunschweig.de/2014/11/30/exim-hoehere-quota-durch-hardlinks/ use deepl.com for translation from german to your desired language. Best regards, Marius -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] exim maildirsize quota calculation in the face of symlinks
On 10/02/2022 10:55, Maarten van Baarsel via Exim-users wrote: I was surprised by the symlink behaviour so I'm reconsidering the use of the dovecot plugin, but I still wanted to ask whether this behaviour is considered OK. I've read the appendfile docs and I could find anything explicit about symlinks. The coding probably doesn't consider the possibility of multiple paths into a single directory, as afforded by the use of symlinks, at all. I've not checked, though. "OK" is in the eye of the beholder, naturally. Handling the case would likely be a feature-addition. The fastest fix would be to not traverse symlinks, I suppose. -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
[exim] exim maildirsize quota calculation in the face of symlinks
Hi,
I'm using Exim 4.90.1 on an Ubuntu 18.04 machine with dovecot 2.2.33 as
IMAP server.
Currently, I'm also using a plugin in dovecot:
https://wiki.dovecot.org/Plugins/MailboxAlias
to make sure the users with different names for the Sent and Trash
folder ("Verzonden items", "Deleted Messages") see the same name between
different user agents.
However, this plugin solves the alias problem with a symlink in the
Maildir directory, which makes Exim count the mails twice when the
appendfile transport is calculating quota sizes.
I managed to grab part of an exim -d+all delivery when a user was over
quota:
10:50:48 18038 maildir_compute_size:
path=/var/mail/virtual/cobergh.nl/cynthias.2097/Maildir/.Sent
10:50:48 18038 sum=2466048922 filecount=8701 timestamp=1644486536
[...]
10:50:48 18038 maildir_compute_size:
path=/var/mail/virtual/cobergh.nl/cynthias.2097/Maildir/.Sent Messages
10:50:48 18038 sum=2466048922 filecount=13088 timestamp=1644486536
and sure enough, one of these directories is a symlink to the other:
root@mx-1:/var/spool/mail/virtual/cobergh.nl/cynthias.2097/Maildir# ls -asF |
fgrep .Sent
4 .Sent/
0 .Sent Messages@
root@mx-1:/var/spool/mail/virtual/cobergh.nl/cynthias.2097/Maildir# stat .Sent\
Messages
File: .Sent Messages -> .Sent
Size: 5 Blocks: 0 IO Block: 4096 symbolic link
[...]
I was surprised by the symlink behaviour so I'm reconsidering the use of
the dovecot plugin, but I still wanted to ask whether this behaviour is
considered OK. I've read the appendfile docs and I could find anything
explicit about symlinks.
the transport part of the config:
virtual_delivery:
driver = appendfile
# let op: geen spatie voor de \ want dat wordt ".../Maildir "
directory =
/var/mail/virtual/$domain/${sg{$local_part}{/}{-}}.${extract{user_id}{$address_data}}/Maildir\
${if bool{$acl_m_mailIsSpam}\
{${if
match_domain{$acl_m_connectionDomainname}{+martin_domains}\
{}\
{/.Junk}\
}\
}\
{${if ! or{ {eq{$address_file}{}}
{eq{$address_file}{inbox}}}\
{/.${sg{$address_file}{/}{-}}}\
{}\
}\
}\
}
directory_mode = 2750
mode= 0640
user= ${extract{uid}{$address_data}}
group = mail
maildir_tag = ",S=$message_size,W=$message_size"
headers_remove =
Lines:Content-Length:Status:X-IMAP:X-IMAPbase:X-Keywords:X-Status:X-UID:X-UIDL
headers_add = "Lines: $body_linecount"
use_crlf
maildir_format
create_directory
delivery_date_add
envelope_to_add
return_path_add
#quota opties
quota = ${extract{quotum}{$address_data}}
quota_size_regex = S=(\d+)
quota_is_inclusive= false
maildir_quota_directory_regex = ^(?:cur|new|\.(?!Trash).*)$
quota_warn_threshold = 95%
quota_warn_message= MSG_QUOTA
maildir_use_size_file
thanks,
Maarten.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
