Re: [vchkpw] Check Space Usage
On Wednesday, October 9, 2002, at 08:43 PM, Justin R. Miller wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Said Matt Simerson on Wed, Oct 09, 2002 at 03:30:55PM -0400: Otherwise, Doug's heading down the right path. Write yourself a script that loops for each domain, and then each user within the domain, and go through and start counting up the bits. You'll likely have issues with depending on the maildirquota file so make sure that if that file doesn't exist, you fall back to a more expensive but accurate method like du. Is there an efficient way to do this if you _don't_ use quotas, ideally something other than 'du'? Efficient is a relative term. If you're asking if there's a way to do this that's nearly as efficient as using file system quotas, then the answer is an easy no. The reason quotas are so efficient is that the kernel is involved, keeping track of disk operations on a per user basis. It always has a real time idea of how much disk space is in use by a customer. There is no other highly efficient way of doing it. The next best utility (based on my personal experience, the extensive reading I've done, and advise of peers) for such a task is du or ls, both if which suffer from varying degrees of inefficiency. I haven't ever compared the speed of using du/ls to using perls File::* utilities. If using file system quota's isn't an option, experimenting with all three could be a fruitful endeavor, the results of which I'd find interesting. Matt
Re: [vchkpw] Check Space Usage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Said Matt Simerson on Mon, Oct 14, 2002 at 10:11:31AM -0400: Is there an efficient way to do this if you _don't_ use quotas, ideally something other than 'du'? Efficient is a relative term. If you're asking if there's a way to do this that's nearly as efficient as using file system quotas, then the answer is an easy no. The reason quotas are so efficient is that the kernel is involved, keeping track of disk operations on a per user basis. It always has a real time idea of how much disk space is in use by a customer. There is no other highly efficient way of doing it. The next best utility (based on my personal experience, the extensive reading I've done, and advise of peers) for such a task is du or ls, both if which suffer from varying degrees of inefficiency. I haven't ever compared the speed of using du/ls to using perls File::* utilities. If using file system quota's isn't an option, experimenting with all three could be a fruitful endeavor, the results of which I'd find interesting. I meant vpopmail quotas, not filesystem quotas. How exactly could you sue fs quotas when everything is owned by vpopmail.vchkpw? - -- [!] Justin R. Miller [EMAIL PROTECTED] Encrypted email preferred (key 0xC9C40C31) -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQE9qxyb94d6K8nEDDERAh4aAKCUDsi/UsfIhq7FGe4wTJQbsT7mmACggLZ7 DlfJ2OC0BgyCW3CYotYK7jc= =QSgg -END PGP SIGNATURE-
[vchkpw] Check Space Usage
Is there a way to use vuserinfo to produce a report of user space utilization, per user, for all users in the domain? Most of the scripting stuff I have tried takes HOURS with 50,000 users.
Re: [vchkpw] Check Space Usage
Michael Funk wrote: Is there a way to use vuserinfo to produce a report of user space utilization, per user, for all users in the domain? Most of the scripting stuff I have tried takes HOURS with 50,000 users. vuserinfo doesn't store that data.. you pretty much just have to either count up the mails, or look in the maildirquota file if you use those kinds of quotas. I think parsing the quota file would be a bit easier, but the perl script I use to look for abusive customers also takes quite a while with a large amount of uses. I've spent a bit of time optimizing my script, so if you want to post it, I'll take a look and see if it can sped up. --Doug
Re: [vchkpw] Check Space Usage
This is a valid argument for having each domain created/owned by an unprivileged system user. When you do that, you have handy tools like repquota to help you manage your disk space usage. When used in conjunction with a perl script, I can format and report the disk space for 10,000 domains in about 6 seconds. I've further taken that report and for all domains that are within 90% of their quota, check the disk usage for each user within the domain and report the disk space hogs. That report takes a couple minutes to run but is invaluable. Otherwise, Doug's heading down the right path. Write yourself a script that loops for each domain, and then each user within the domain, and go through and start counting up the bits. You'll likely have issues with depending on the maildirquota file so make sure that if that file doesn't exist, you fall back to a more expensive but accurate method like du. Matt On Wednesday, October 9, 2002, at 08:27 AM, Doug Clements wrote: Michael Funk wrote: Is there a way to use vuserinfo to produce a report of user space utilization, per user, for all users in the domain? Most of the scripting stuff I have tried takes HOURS with 50,000 users. vuserinfo doesn't store that data.. you pretty much just have to either count up the mails, or look in the maildirquota file if you use those kinds of quotas. I think parsing the quota file would be a bit easier, but the perl script I use to look for abusive customers also takes quite a while with a large amount of uses. I've spent a bit of time optimizing my script, so if you want to post it, I'll take a look and see if it can sped up. --Doug
Re: [vchkpw] Check Space Usage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Said Matt Simerson on Wed, Oct 09, 2002 at 03:30:55PM -0400: Otherwise, Doug's heading down the right path. Write yourself a script that loops for each domain, and then each user within the domain, and go through and start counting up the bits. You'll likely have issues with depending on the maildirquota file so make sure that if that file doesn't exist, you fall back to a more expensive but accurate method like du. Is there an efficient way to do this if you _don't_ use quotas, ideally something other than 'du'? - -- [!] Justin R. Miller [EMAIL PROTECTED] Encrypted email preferred (key 0xC9C40C31) -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQE9pM0594d6K8nEDDERAiZtAJwJCG/Msbh2sBhOURk14/rFhV9S9wCeLzKK 9UlVmADxqekIwcH1xtG7RAo= =qPLj -END PGP SIGNATURE-
Re: [vchkpw] Check Space Usage
- Original Message -
From: Justin R. Miller [EMAIL PROTECTED]
To: vpopmail list [EMAIL PROTECTED]
Sent: Wednesday, October 09, 2002 5:43 PM
Subject: Re: [vchkpw] Check Space Usage
Is there an efficient way to do this if you _don't_ use quotas, ideally
something other than 'du'?
You can open a pipe to 'ls' since the mail sizes are in the name. Quite
easy, though it's up for debate if this is more efficient than du.
This also doesn't take into account other files in the Maildir that aren't
actually messages, but depending on your point of view, these files
shouldn't be included in the user quota anyways.
Snippet of ls pipe (haven't converted to native opendir and readdir yet,
which is probably a bit faster since it saves a process being launched):
open (IN, ls $homeDir/Maildir/$dir |);
while ($line = IN){
chomp $line;
($garbage,$size) = split(',', $line);
($garbage,$size) = split('=', $line);
# $filename = $homeDir/Maildir/$dir . $line;
$MailBoxSize += $size;
}
Even better would be native opendir() and readdir(), since it saves a
process being launched. At this point, savings are likely to be minimal,
even with thousands of mailboxes.
Depending on how you do backups and what your needs are for calculating used
space, you might be better analyzing last night's backups instead of live
data. We copy the main data store every night to a seperate disk as part of
our backup, so running analysis on this disk would be quite a bit faster
than on the live store. If you don't need live quota management, this
might be acceptable.
--Doug
