On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote:
I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort.
For folks that are seeing huge jumps in memory, instead of gradual growth, how are you calling SA?
Thanks, Michael
I, too, am seeing the 250MB spamd processes. I've been doing some research on this and I'll share what I know.
I'm using SA 3.0.0 with spamd and the qmail-spamc wrapper which really just pipes the email into spamc. It's pretty common that one of the machines doing the filtering will have a spamd process jump to 250MB or more of memory usage. I have several machines doing filtering and each will have a problem several times per hour. I am not using Bayes at all, so this doesn't have anything to do with that. Since I'm using spamc, I've got its default behavior which is to not scan any message over 250KB.
First, I've been told that perl never really frees memory, even when it isn't using it any more. So, if a single message causes a spamd to jump up to hundreds of MB's, it will stay that way until the process exits entirely. Setting the --max-conn-per-child to a lower number will help make sure these don't stick around too long. With the default of 200 and a bunch of spamd processes, it could be hours before a spamd dies on a server that isn't busy. Pick a time limit that seems reasonable to you, say five minutes, and adjust --max-conn-per-child to make the processes die about that often.
I captured an exact copy of one of the messages that was being scanned when this happened. This particular message took something like 10 minutes to scan. We deliver to maildirs, so it was easy to get a copy of it, although it's possible there was an extra header line by the time I had it. Rescanning the same message by calling spamc didn't cause the problem. The scan completed in just a couple of seconds.
One interesting thing is that I have some machines that do this and a couple that don't. The ones that don't were built slightly later than the ones that do. Actually, the SpamAssassin installs were done at the same time and are identical, but other system software isn't exactly the same vintage. The machines that are having problems all have perl 5.8.0. The machines that don't all run perl 5.8.4. There might be small differences in the versions of other libraries, too. Maybe this is exercising a perl bug?
This extreme memory usage does cause real problems for us. If two spamd's get messed up at the same time, the machine starts thrashing to swap. (The machines all have 1 GB of RAM). When that happens, mail processing slows way down, which unfortunately means it takes even longer for the spamd's to process enough messages to go ahead and die.
One last note, when the spamd is processing a trouble message, it seems like it is usually running 99% CPU. It might drop CPU usage after that message, but like I said earlier it doesn't drop memory usage until it dies. Why a small message would cause spamd to go to 99% CPU for minutes, I don't know.
JT
