John Simpson wrote:
On 2006-12-22, at 1006, Christopher Chan wrote:
John Simpson wrote:
http://qmail.jms1.net/patches/validrcptto.cdb.shtml
There is a better patch for vpopmail support in qmail. A mysql patch
that goes straight the vpopmail mysql database but I am not sure of
its location. The writer even rebuffed one of Inter7's developers when
someone floated the idea of qmail supporting vpopmail's mysql tables
and the developer said he would write it since he was not aware of the
patch's existence. So I believe the Inter7 guy drop it right then and
there or maybe not. I believe it is this one here and the writer was
Italian: http://www.interazioni.it/opensource/chkusr/
that's all well and good, IF your incoming mail always arrives on the
same machine where vpopmail is running, IF you don't mind re-compiling
qmail everytime vpopmail is upgraded, and IF you keep your user
information in a mysql database.
If it ain't broke...I don't see why people would want to upgrade
vpopmail unless there is a security fix or a feature that they need.
Realistically, it cannot be as bad as you make it.
most ISPs handle a large enough volume of email that they have several
internet-facing servers which handle the flood of incoming mail, and
forward the legitimate messages to an internal machine which contains
the mailboxes. the one mailbox machine will be running vpopmail, but the
other internet-facing servers (i call them "mailhubs") are generally not
running vpopmail, which means they are not able to check recipients or
process AUTH commands against the vpopmail information.
i've seen people get around this using mysql, both by having the
mailhubs connect across the network to a database server, and by setting
up mysql servers on the mailhubs and replicating the data. but what if
the company isn't using mysql in the first place? (i spent eight years
building and running ISPs with this exact scenario- multiple mailhubs,
no mysql. yes, we had a customer database- but that was for billing, and
it wasn't directly involved with the mechanics of the systems themselves.)
my validrcptto.cdb and auth.cdb patches get around these problems by
storing the list of valid recipient addresses and the list of valid
userid/password pairs in cdb files, and just copying those files from
the mailbox server out to the mailhubs whenever they change. PLUS, the
fact that they're cdb files means that the lookups happen without the
added overhead of having to open a connection to a mysql server (whose
connection pool might become overloaded in case of a spam flood.)
I will pit my four years in my previous job as a MTA admin in a SME
email service provider that handles in total over 40 million mailboxes
against your eight years running and building ISPs. The same two stage
delivery system is used too. I have dealt with both types of
environments. An older system built cdb files for deployment to the
frontline mailhubs. The newer systems had mysql servers for the
frontline mailhubs. I get the impression you have not seen mysql
connection pooling in acton. When I joined, they used sendmail frontline
hubs patched to support mysql databases with cdb support being later
added by me for the older system. The sendmail patch had no connection
pooling support and so it would open a new mysql connection to the mysql
server and yes, this meant that the mysql server would become overloaded
in the case of a spam flood. There was, for example, one set of 5
frontline servers handling up to 600 connections each using one mysql
server and another set of 4 frontline servers also doing 600 connections
using another mysql server. I did a trial with postfix with its mysql
and mysql connection pooling support because I got tired of manually
taking care of the queues due to the mysql servers being overloaded and
due to the many security holes that were being discovered in sendmail
8.12.x. postfix replaced sendmail very soon after the first trial run.
mysql connection pooling makes a huge difference. Those two mysql
servers under the sendmail system would be pushed till they had only 10%
cpu idle resources being reported and they were still not delivering
because mysql just cannot handle a large number of connections that are
being set up and torn down at the same time. With the postfix system
using postfix's builtin connection pooling support, just ONE mysql
server is enough to the mysql query load for all eleven boxes at full
load (all connections available taken and a tcp syn queue backlog of
over 1024 per box) without breaking a sweat. Lowest cpu idle registered
on the mysql server is 80%. Connection pooling is king. The bottleneck
now is not the mysql server but the mailhubs themselves. So it appears
to me that your comment "(whose connection pool might become
overloaded)" indicates you have no idea what I meant by 'connection
pooling'. This is software on the mailhub side, not the mysql server
side, that addresses mysql's poor connection handling performance.
the "down side" is that you have to write some scripts to generate the
validrcptto.cdb and auth.cdb files in the first place, and copy them out
to the mailhubs. however, my web site also has working "mkvalidrcptto"
and "mkauth" scripts, along with a web page which explains how to use
ssh to push the files out to your mailhubs... so while it may not be
"brain dead easy", it's certainly not as difficult as setting up and
maintaining replication between mysql servers.
Are you serious? mysql replication is very straight forward. It works so
well that sometimes you even don't think about it until you notice disk
space getting low (if you did not organize automatic deleting of binary
logs). As for generating cdbs, it is "brain dead easy". Create master
file, rsync to host and then generate cdb on host. You apparently do the
way it was formerly done too at the outfit; generate cdb and then scp
the cdb file across to relevant boxes. What did you do to ensure that it
is an atomic operation on the push/copy out to mailhub?
for my needs and my clients' needs, my patches are the best solution.
they may not be for everybody, which is why i'll explain the differences
between validrcptto.cdb and chkusr, but i don't claim either one to be
"better" than the other. different people have different needs.
Yes, so long as you do not need the 'instant' creation of accounts or
what not, your system will do fine for those who have a controlled
generation of the cdb files.
postfix trumps chkusr/chkuser just as chkusr/chkuser trumps the cdb
check.
everybody has their own opinions... mine happen to be the exact opposite
of what you've written here.
Yes, these are all opinions. Food for the goose is poison for the gander
as they say.
First, chkusr vs rcptto.cdb. tcpserver + qmail-smtpd means a fresh
fork for each new connection. The cdb rcptto means a disk access for
each rcpt to check and regular rebuilds of the cdb database.
chkusr/chkuser helps by keeping I/O of disk (okay we can contest
whether looking up cdbs is better than looking up mysql tables or not
but I think it is fair game to say that mysql lookups are more likely
to be disk I/O free) and by not needing regular rebuilds of a cdb
file. In fact, it offers instant/real-time user existence checks.
until you build in the overhead of mysql replication (or even worse,
qmail-smtpd connecting to a mysql server across the network.)
open() takes less CPU and less time than mysql_connect(), even if the
mysql server is on the same machine (because open() only involves
qmail-smtpd and the kernel, while mysql_connect() also involves mysqld,
which may already be busy with other clients, witness the complains
about this very issue on the courier-imap list.)
and in the case of a file like validrcptto.cdb, which would be used
constantly on a busy server, the file's data blocks would be in the
kernel's disk cache 99% of the time, so there is almost never any wait
for a disk to rotate- any disk reads are satisfied from the kernel's
disk cache. if anything, i think the chances of a single file already
being cached in the kernel's disk cache are higher than the chances of a
mysql server having the right rows from the right table in memory, plus
be idle at the right time and be able to answer qmail-smtpd's queries
immediately.
believe me, i have tested these scenarios with several clients... and
validrcptto.cdb has always ended up being faster than any mysql
scenario, both with and without replication. and yes, these were on busy
servers- one mailbox server with 120K mailboxes across 4500 domains, and
four mailhubs, located at three sites in different parts of florida,
handling an average of 1.2 million SMTP connections per hour.
For your traffic patterns, cdb will probably do. The outfit I worked for
handled on a daily average, 200 million SMTP connections or over 8
million connections hourly. It was not acceptable to spend minutes
pushing the cdb file across for the mailhubs and probably still is.
(Please don't give me the get proper hardware. If I could have gotten
more servers or replacements that had better disk i/o...)
postfix improves on this by 1) no new fork for each connection (except
perhaps initially if handling hundreds or thousands of connections
right after startup of postfix) and 2) by using mysql connection
pooling which means you don't hammer the mysql server into the ground
with the constant setting up and breaking down of connections. This is
without including all the great anti-spam features that postfix
provides too.
i'll be honest, i can't really comment on this. my only exposure to
postfix has been removing it from my OSX machines in order to make way
for qmail. i can't and won't say that one is "better" than the other,
since i haven't ever administered a postfix system. i know that postfix
has its collection of zealots, just as qmail does.
ROTFL. I have done sendmail, postfix and qmail. qmail is the best in
that it is simple and elegant. I had colleagues who would not touch
qmail with a ten foot pole. They did not care to delve into the
internals of qmail and qmail is a pain if you have to go in the clear
out spam. sendmail and postfix are much better in the queue management
area. Stopping qmail-send to scrounge out spam and then making sure you
delete the stuff properly and do not end up with a corrupt queue is not
their cup of tea since it is something they have to do regularly
(yes...partly free webmail provider). Don't give tell me about qmHandle.
That script is broken and will leave you with corrupt bounce messages
under certain conditions besides being awfully slow.
I probably know/understand qmail better than you do.
i wouldn't bet on it, but then i have better things to do with my life
than play "mine is bigger than yours" with some random person on a
mailing list. i know what has worked for me since 1998, and what i will
continue using. if you don't want to use my patches, or if you don't
want to use qmail, it's no sweat off my back... i just threw it out
there as a suggestion, trying to mention that there are other options.
there was no need for you to get nasty about why you don't want to use
my patches, i truly don't care if you use them or not.
? I see I have got off on the wrong foot. I thought postfix was the
'other option'. qmail is not an option. qmail is a must whether patched
or not. I'd like to see vpopmail get an option to run without needing
the presence of qmail. I don't fancy telling others to install a 'stub
qmail' so that they can benefit from vpopmail without having to build
their own virtual mail backend.
