On Wed, May 30, 2001 at 01:44:13PM -0400, rick pim wrote:
> 4.x has a variety of performance improvements over v3. i'd like to
> implement as many of them as are safe in our situation. on our
> systems, users have shell accounts. "most" users who use POP don't use
> the shell access at all, let alone for mail. of those that do, "most"
> don't use shell access and POP at the same time. our biggest system
> has 15-20k users and about 100k pop connections per day.
This sounds quite similar to our configuration that I just posted
about. In our case, just implemented, v4 + server-mode (selectively) +
fast-update was a tremendous performance increase.
> the most obvious performance wins are server mode and fast-update.
> fast-update is listed in the manual as potentially breaking biff.
> if this is the worst problem then, given our setup, i'm more than willing
> to enable it. are there any other consequences to using fast-update?
There are two gotchas here, which I got after a week or two of puzzling
through things.
1) fast-update *only* ever applies if you are using server mode. I had
to read the code to understand this; I couldn't find a note to that
effect in the documentation that I could see. (Read the comments at
the the header of pop_updt.c, and then grep for the fast update
variable if you want to confirm this.) So first worry about making
server mode safe to use, and if you do then you will pretty certainly
find fast-update safe.
2) fast-update requires the poptemp (pop drop) files to be located on
the same partition as the corresponding mail spool files. If you have
previously located them on a separate partition (as we had) for greater
performance copying mailboxes to the poptemp files in non-server mode,
you will need to reorganize the location of your poptemp directory; in
our case because /var/mail is its own partition, we created a separate
directory under /var/mail for poptemp files. Again, I had to read the
code to get clear on this. Server mode still will recopy the spool
file under some circumstances to merge changes made by the pop session
with independent changes during it, e.g. new mail arriving. In
fast-update mode, it then relinks the recopied file on place of the
spool instead of copying it again.
I may try to write some proposed verbiage about both these points for
the documentation because it really confused me when I was trying to
set it up.
> the same holds for server mode: in a sitation like this, there are
> potential issues with server mode, but the documentation doesn't
> detail them. exactly how serious _are_ they? what's the worst case?
> is it something serious (lost mail, vanishing mailboxes) or just
> cosmetic?
Server mode is the biggest win, because it prevents qpopper from
recopying the *entire* mailbox on every single POP session.
In *theory* and in the worst case, I think that in server mode lost
mail could result in some combination of circumstances, like a user
reading mail with a carelessly implemented shell mail client, plus
using POP at the same time, plus mail arrives during the POP session.
I haven't read a definitive statement about this, and I might be overly
pessimistic, but popper is written to be extremely conservative about
these possibilities in non server mode. In server mode it is still
careful about its spool handling but less conservative in its
assumptions, basically assuming that the only way the spool will get
updated while popping is by new mail arriving.
I think your best course is to find some way to segregate users who may
actually *use* their shell from those who are certain not to log in and
run shell programs, and automatically enable server mode for all of the
latter. That's what we did and as of day 2 it's going very well. One
of v4's big advantages is that it has several facilities to do this,
based on the group a user is in or based on individual qpopper.options
files. Then if you have server-mode applying only to non-shell users,
fast-update is also safe for all of them. If you can put all your
shell users in certain groups, or all non-shell users in certain
groups, you're set. I ended up coding our own based on our
circumstances, where we set it based on the assigned login shell - if
they have a dummy shell of /usr/local/bin/no_shell or /sbin/nologin,
qpopper goes into server mode.
> other performance wins in v4 would be nice to hear comments on.
In combination with the above features, the UIDL cache file is a huge
win. As long as no new mail has come in for a given user, they can POP
it every 2 minutes with absolutely 0 performance impact, even if they
have a huge mailbox, because all qpopper needs to do is check the time
stamp on the files and then read the (tiny) UIDL cache file.
Same as with v3, turning off reverse lookups is a big win as always for
most servers.
If you have 15-20K users, if you can get other mail programs to comply
with it (clients, etc.) then you might consider the mailhash algorithms
to cut down on directory lookup times. This is minor compared to the
other improvements, though.
One last meta-issue is that if you write some scripts to analyse your
POP logs, you will probably find that, unless you have very strict mail
quotas, a dozen or so users out of your 15K are generating most of the
load on your mail server through having big mailboxes, leaving their
mail on the server, and checking them frequently. If you deal with
them administratively, you can significantly reduce your load. Log the
"Stats" and "Timing" lines, total the fields by user, and look for
users whose numbers stand out.
Good luck!
-- Clifton
--
Clifton Royston -- LavaNet Systems Architect -- [EMAIL PROTECTED]
WWJD? "JWRTFM!" - Scott Dorsey (kludge) "JWG" - Eddie Aikau
