John Simpson wrote:
On 2006-12-22, at 1006, Christopher Chan wrote:
John Simpson wrote:


http://qmail.jms1.net/patches/validrcptto.cdb.shtml

There is a better patch for vpopmail support in qmail. A mysql patch that goes straight the vpopmail mysql database but I am not sure of its location. The writer even rebuffed one of Inter7's developers when someone floated the idea of qmail supporting vpopmail's mysql tables and the developer said he would write it since he was not aware of the patch's existence. So I believe the Inter7 guy drop it right then and there or maybe not. I believe it is this one here and the writer was Italian: http://www.interazioni.it/opensource/chkusr/

that's all well and good, IF your incoming mail always arrives on the same machine where vpopmail is running, IF you don't mind re-compiling qmail everytime vpopmail is upgraded, and IF you keep your user information in a mysql database.

If it ain't broke...I don't see why people would want to upgrade vpopmail unless there is a security fix or a feature that they need. Realistically, it cannot be as bad as you make it.


most ISPs handle a large enough volume of email that they have several internet-facing servers which handle the flood of incoming mail, and forward the legitimate messages to an internal machine which contains the mailboxes. the one mailbox machine will be running vpopmail, but the other internet-facing servers (i call them "mailhubs") are generally not running vpopmail, which means they are not able to check recipients or process AUTH commands against the vpopmail information.

i've seen people get around this using mysql, both by having the mailhubs connect across the network to a database server, and by setting up mysql servers on the mailhubs and replicating the data. but what if the company isn't using mysql in the first place? (i spent eight years building and running ISPs with this exact scenario- multiple mailhubs, no mysql. yes, we had a customer database- but that was for billing, and it wasn't directly involved with the mechanics of the systems themselves.)

my validrcptto.cdb and auth.cdb patches get around these problems by storing the list of valid recipient addresses and the list of valid userid/password pairs in cdb files, and just copying those files from the mailbox server out to the mailhubs whenever they change. PLUS, the fact that they're cdb files means that the lookups happen without the added overhead of having to open a connection to a mysql server (whose connection pool might become overloaded in case of a spam flood.)

I will pit my four years in my previous job as a MTA admin in a SME email service provider that handles in total over 40 million mailboxes against your eight years running and building ISPs. The same two stage delivery system is used too. I have dealt with both types of environments. An older system built cdb files for deployment to the frontline mailhubs. The newer systems had mysql servers for the frontline mailhubs. I get the impression you have not seen mysql connection pooling in acton. When I joined, they used sendmail frontline hubs patched to support mysql databases with cdb support being later added by me for the older system. The sendmail patch had no connection pooling support and so it would open a new mysql connection to the mysql server and yes, this meant that the mysql server would become overloaded in the case of a spam flood. There was, for example, one set of 5 frontline servers handling up to 600 connections each using one mysql server and another set of 4 frontline servers also doing 600 connections using another mysql server. I did a trial with postfix with its mysql and mysql connection pooling support because I got tired of manually taking care of the queues due to the mysql servers being overloaded and due to the many security holes that were being discovered in sendmail 8.12.x. postfix replaced sendmail very soon after the first trial run.

mysql connection pooling makes a huge difference. Those two mysql servers under the sendmail system would be pushed till they had only 10% cpu idle resources being reported and they were still not delivering because mysql just cannot handle a large number of connections that are being set up and torn down at the same time. With the postfix system using postfix's builtin connection pooling support, just ONE mysql server is enough to the mysql query load for all eleven boxes at full load (all connections available taken and a tcp syn queue backlog of over 1024 per box) without breaking a sweat. Lowest cpu idle registered on the mysql server is 80%. Connection pooling is king. The bottleneck now is not the mysql server but the mailhubs themselves. So it appears to me that your comment "(whose connection pool might become overloaded)" indicates you have no idea what I meant by 'connection pooling'. This is software on the mailhub side, not the mysql server side, that addresses mysql's poor connection handling performance.


the "down side" is that you have to write some scripts to generate the validrcptto.cdb and auth.cdb files in the first place, and copy them out to the mailhubs. however, my web site also has working "mkvalidrcptto" and "mkauth" scripts, along with a web page which explains how to use ssh to push the files out to your mailhubs... so while it may not be "brain dead easy", it's certainly not as difficult as setting up and maintaining replication between mysql servers.

Are you serious? mysql replication is very straight forward. It works so well that sometimes you even don't think about it until you notice disk space getting low (if you did not organize automatic deleting of binary logs). As for generating cdbs, it is "brain dead easy". Create master file, rsync to host and then generate cdb on host. You apparently do the way it was formerly done too at the outfit; generate cdb and then scp the cdb file across to relevant boxes. What did you do to ensure that it is an atomic operation on the push/copy out to mailhub?


for my needs and my clients' needs, my patches are the best solution. they may not be for everybody, which is why i'll explain the differences between validrcptto.cdb and chkusr, but i don't claim either one to be "better" than the other. different people have different needs.

Yes, so long as you do not need the 'instant' creation of accounts or what not, your system will do fine for those who have a controlled generation of the cdb files.


postfix trumps chkusr/chkuser just as chkusr/chkuser trumps the cdb check.

everybody has their own opinions... mine happen to be the exact opposite of what you've written here.

Yes, these are all opinions. Food for the goose is poison for the gander as they say.


First, chkusr vs rcptto.cdb. tcpserver + qmail-smtpd means a fresh fork for each new connection. The cdb rcptto means a disk access for each rcpt to check and regular rebuilds of the cdb database. chkusr/chkuser helps by keeping I/O of disk (okay we can contest whether looking up cdbs is better than looking up mysql tables or not but I think it is fair game to say that mysql lookups are more likely to be disk I/O free) and by not needing regular rebuilds of a cdb file. In fact, it offers instant/real-time user existence checks.

until you build in the overhead of mysql replication (or even worse, qmail-smtpd connecting to a mysql server across the network.)

open() takes less CPU and less time than mysql_connect(), even if the mysql server is on the same machine (because open() only involves qmail-smtpd and the kernel, while mysql_connect() also involves mysqld, which may already be busy with other clients, witness the complains about this very issue on the courier-imap list.)

and in the case of a file like validrcptto.cdb, which would be used constantly on a busy server, the file's data blocks would be in the kernel's disk cache 99% of the time, so there is almost never any wait for a disk to rotate- any disk reads are satisfied from the kernel's disk cache. if anything, i think the chances of a single file already being cached in the kernel's disk cache are higher than the chances of a mysql server having the right rows from the right table in memory, plus be idle at the right time and be able to answer qmail-smtpd's queries immediately.

believe me, i have tested these scenarios with several clients... and validrcptto.cdb has always ended up being faster than any mysql scenario, both with and without replication. and yes, these were on busy servers- one mailbox server with 120K mailboxes across 4500 domains, and four mailhubs, located at three sites in different parts of florida, handling an average of 1.2 million SMTP connections per hour.

For your traffic patterns, cdb will probably do. The outfit I worked for handled on a daily average, 200 million SMTP connections or over 8 million connections hourly. It was not acceptable to spend minutes pushing the cdb file across for the mailhubs and probably still is. (Please don't give me the get proper hardware. If I could have gotten more servers or replacements that had better disk i/o...)


postfix improves on this by 1) no new fork for each connection (except perhaps initially if handling hundreds or thousands of connections right after startup of postfix) and 2) by using mysql connection pooling which means you don't hammer the mysql server into the ground with the constant setting up and breaking down of connections. This is without including all the great anti-spam features that postfix provides too.

i'll be honest, i can't really comment on this. my only exposure to postfix has been removing it from my OSX machines in order to make way for qmail. i can't and won't say that one is "better" than the other, since i haven't ever administered a postfix system. i know that postfix has its collection of zealots, just as qmail does.

ROTFL. I have done sendmail, postfix and qmail. qmail is the best in that it is simple and elegant. I had colleagues who would not touch qmail with a ten foot pole. They did not care to delve into the internals of qmail and qmail is a pain if you have to go in the clear out spam. sendmail and postfix are much better in the queue management area. Stopping qmail-send to scrounge out spam and then making sure you delete the stuff properly and do not end up with a corrupt queue is not their cup of tea since it is something they have to do regularly (yes...partly free webmail provider). Don't give tell me about qmHandle. That script is broken and will leave you with corrupt bounce messages under certain conditions besides being awfully slow.


I probably know/understand qmail better than you do.

i wouldn't bet on it, but then i have better things to do with my life than play "mine is bigger than yours" with some random person on a mailing list. i know what has worked for me since 1998, and what i will continue using. if you don't want to use my patches, or if you don't want to use qmail, it's no sweat off my back... i just threw it out there as a suggestion, trying to mention that there are other options. there was no need for you to get nasty about why you don't want to use my patches, i truly don't care if you use them or not.

? I see I have got off on the wrong foot. I thought postfix was the 'other option'. qmail is not an option. qmail is a must whether patched or not. I'd like to see vpopmail get an option to run without needing the presence of qmail. I don't fancy telling others to install a 'stub qmail' so that they can benefit from vpopmail without having to build their own virtual mail backend.

Reply via email to