Hash: SHA1

Tom Collins wrote:
> On Jan 22, 2009, at 1:43 AM, Matt Brookings wrote:
>> Where do you come up with 128bit from?  On most systems a 'long long'
>> is going
>> to be a 64bit integer.  That's what I'm currently using.
> Sorry, you're right on that.
> Why isn't it enough to keep the current quota system, and just update
> all related code to use a long long (64 bits) for the byte counter? 
> Will quotas ever grow that huge?  In the next 10 years?

I'm not replacing the current quota system, and I agree, we should update
all the quota code in vpopmail to be 64bits.

> You could even keep the 32 bit numbers, and just update all
> quota-checking code to round sizes up to the nearest KB -- now you're
> able to have quotas up to 2000 GB (2TB).  Since a single email isn't
> going to be 2GB, each entry in maildirsize can still be a 32-bit int and
> you may not have to modify qmail or the POP/IMAP servers.  You only need
> to update the code that rebuilds the file and checks for an overquota
> condition.

Here's what I'm doing.  I have already created a daemon which answers requests
about user and domain storage.  It has a set of rules by which it operates.

  1) Always respond quickly, even if the information is not up to date
     This is to be sure that processes requesting information on user usage,
     like vdelivermail, or an SMTP daemon, never bottleneck on the daemon
     if the system is under heavy load, or fail even if the daemon is not 
     or fails.

     This is supported by both the server and client code.  Neither are allowed
     to cause a bottleneck.

  2) If a request will take too long to fill and there is no information 
     it must return a zero usage response so that delivery will not fail.  It's
     more important to never fail erroneously than to keep quotas enforced in 

     Again, supported by both the server and client.

  3) All updates can be deferred into a queue to be handled at reasonable I/O 
     If a request will take a very long period of time, or is known to use 
large amounts
     of disk I/O, for instance, opening a directory with 150k messages in it, 
the last
     known information on this directory will be used and the directory can be 
tagged as
     an extreme I/O hog to further defer future updates.

     This will have it's own set of rules for dealing with a growing queue of 
     updates, etc.

  4) Do not enforce, or care, about quotas
     The goal of the daemon is not to provide quota enforcement, it merely 
     information on user usage.  Comparing usage to quotas, and enforcing 
quotas is entirely
     up to the program using the client API.

The client API is very easy to use.  Here is an example:

     // storage_t is a typedef of a 64bit unsigned integer defined by the 
server code
     storage_t user_usage = 0, domain_usage = 0;

     cHandle = client_connect();
     // Error checking here

     client_query(cHandle, "u...@domain", strlen("u...@domain"), &user_usage, 

     if (user_usage >= user_quota) fail();
     if (domain_usage >= domain_quota) fail();

I've been running this code on a few production servers where vdelivermail and 
qmail-smtpd were
patched not to enforce quotas, but simply to run queries on every user it sees. 
 Most queries
are responded to in 0.010 seconds or less with accuracy to within 15 seconds of 
actual disk usage.

There's still lots of work to be done on it, but I hope to release it in a beta 
form after
I return from vacation Feb 2nd.

My current TODO:
  * Disk updates must be deferrable; maybe with threading
  * Statistics gathering like average disk poll time, biggest poll time, 
shortest poll time, etc

As you can see, the deferring updates is not yet implemented.  The performance 
I'm currently
seeing is acceptable, but it would probably not cut it on a *very* large system 
with many absent
users and catchalls collecting gigs of email in a single directory.

I hope to release a beta to developers once the deferring code is ready.

Opinions on this? :)
- --
    Matt Brookings <m...@inter7.com>       GnuPG Key D9414F70
    Software developer                     Systems technician
    Inter7 Internet Technologies, Inc.     (815)776-9465
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


Reply via email to