Re: `top` process memory usage: SIZE vs RES
[EMAIL PROTECTED] wrote: [...] J 1.) Where is my Free memory going? given what you say custom-python-qmail-scanner-clamd-qmail-queue This whole scenario is very memory intensive. First you have each email pythonized and then qmail-scanner is *very* memory intensive, as it has initially a very heavy duty perl script for each email before being passed off to clamd. Clamd is a separate issue, since the only clamav command actually run from the pipeline (and thus under the restrictions of softlimit) is the clamdscan client, which is NOT memory intensive. Yes, clamd contributes to the overall memory footprint, but I'm only concerned with getting softlimit set properly at this point. My machine can always revert to swap, but the second softlimit is exceeded the email will be temporarily defered, which I consider a Bad Thing. Having said that, yes, it is still a very memory intensive pipeline. I took some time to profile the memory usage a few days ago, and it looks like the most memory the pipeline should ever use at any given point in time is ~12780K, with the following processes running: USER PID PPID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND qmaild 24716 24553 0.0 0.2 920 460 ?? I 7:39PM 1:08.07 /var/qmail/bin/qmail-smtpd qmaild 24718 24716 0.0 0.3 884 488 ?? I 7:39PM 0:08.63 /usr/local/bin/qmail-qfilter /var/qmail/queue-filters/block-forged-sender.py -s qmailq 24730 24718 9.2 2.1 5052 3988 ?? S 7:41PM 0:55.87 /usr/bin/suidperl -T /dev/fd/4//var/qmail/bin/qmail-scanner-queue.pl (perl) qmailq 24739 24730 69.7 2.1 5052 3988 ?? R 7:43PM 0:06.55 /usr/bin/suidperl -T /dev/fd/4//var/qmail/bin/qmail-scanner-queue.pl (perl) qmailq 24740 24739 14.4 0.2 872 400 ?? R 7:43PM 0:01.28 /var/qmail/bin/qmail-queue (qmail-scanner is silly. For some reason it spawns a copy of itself, possibly to hand the message off to qmail-queue.) But even with the softlimit set to 15M, my huge test message to a server with only about 80M of free RAM (before sending the message. Free Memory dropped to ~500k while handling the message) somehow managed to exceed the softlimit. The exact same message, sent to a machine with ~600M of free RAM and an identical mail server setup, passed through the pipeline without tripping the softlimit. From what I have seen while watching a huge message pass down the pipeline, none of the processes in the pipeline increase memory usage in proportion to email size. They're all relatively static. So I'm a little confused about why the softlimit would be tripped on a box that had less RAM (128M) but pass through successfully on a box with more RAM (1G). Would the act of using more swap effectively increase a process's: data segment usage? stack segment usage? locked physical pages per process? total of all segments per process? These are the things that softlimit limits (according to `man softlimit`), and I admittedly don't understand how any of the above translates to memory usage as shown by VSZ and RSS under `ps`, or SIZE and RES under `top`. Any ideas? Maybe running vmstat -w 1 would give you a different perspective also. I'll check it out. -- Jesse Guardiani, Systems Administrator WingNET Internet Services, P.O. Box 2605 // Cleveland, TN 37320-2605 423-559-LINK (v) 423-559-5145 (f) http://www.wingnet.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
Dan Nelson wrote: In the last episode (Sep 11), Jesse Guardiani said: 1.) Where is my Free memory going? I can't account for it in the SIZE and RES columns of the various processes. These are relatively constant. Disk cache. I thought it might be something like that. My large test messages are being written to disk over and over and over as the message travels down the pipline. Makes a great case for installing a RAM disk. :) 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core OK. To clarify, you mean core kernel memory here? If so, how is that significant? Why should I care? In other words, why would I ever want to know that? Thanks. -- Jesse Guardiani, Systems Administrator WingNET Internet Services, P.O. Box 2605 // Cleveland, TN 37320-2605 423-559-LINK (v) 423-559-5145 (f) http://www.wingnet.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
In the last episode (Sep 12), Jesse Guardiani said: Dan Nelson wrote: In the last episode (Sep 11), Jesse Guardiani said: 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core OK. To clarify, you mean core kernel memory here? If so, how is that significant? Why should I care? In other words, why would I ever want to know that? core meaning physical memory; user memory in this case. Processes can lock kernel memory, but there's no easy way of listing that (it's usually a small amount held in pipe or socket buffers and is short-lived). The name core came from when memory bits were ferrite rings magnetized by wires running through them. http://www.columbia.edu/acis/history/core.html -- Dan Nelson [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
Jesse Guardiani [EMAIL PROTECTED] writes: Dan Nelson wrote: In the last episode (Sep 11), Jesse Guardiani said: 1.) Where is my Free memory going? I can't account for it in the SIZE and RES columns of the various processes. These are relatively constant. Disk cache. I thought it might be something like that. My large test messages are being written to disk over and over and over as the message travels down the pipline. Makes a great case for installing a RAM disk. :) No, probably not. The OS disk-caching is probably *more* efficient than letting the data go into a RAM disk at each stage. Considerably so, in fact. 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core OK. To clarify, you mean core kernel memory here? No, it's not in kernel space. Core just refers to RAM: the term is held over from the days when main memory was constructed out of little magnetic cores in a wire matrix. If so, how is that significant? Why should I care? If your system starts swapping heavily, that will often be the clue that tells you why. Just one example. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
Dan Nelson wrote: In the last episode (Sep 12), Jesse Guardiani said: Dan Nelson wrote: In the last episode (Sep 11), Jesse Guardiani said: 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core OK. To clarify, you mean core kernel memory here? If so, how is that significant? Why should I care? In other words, why would I ever want to know that? core meaning physical memory; user memory in this case. OK. And how does core, or user memory differ from SIZE memory then? If X = SIZE - RES, where is X stored? Processes can lock kernel memory, but there's no easy way of listing that (it's usually a small amount held in pipe or socket buffers and is short-lived). The name core came from when memory bits were ferrite rings magnetized by wires running through them. http://www.columbia.edu/acis/history/core.html Wow. That's a really cool bit of history. I don't quite understand how a core is switched, but I'm sure it must have worked. -- Jesse Guardiani, Systems Administrator WingNET Internet Services, P.O. Box 2605 // Cleveland, TN 37320-2605 423-559-LINK (v) 423-559-5145 (f) http://www.wingnet.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
In the last episode (Sep 12), Jesse Guardiani said: Dan Nelson wrote: In the last episode (Sep 12), Jesse Guardiani said: Dan Nelson wrote: In the last episode (Sep 11), Jesse Guardiani said: 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core OK. To clarify, you mean core kernel memory here? If so, how is that significant? Why should I care? In other words, why would I ever want to know that? core meaning physical memory; user memory in this case. OK. And how does core, or user memory differ from SIZE memory then? If X = SIZE - RES, where is X stored? You don't need to store it, since you know SIZE and RES :) X is any memory mapped into process space that is not in physical memory at the moment; it could be dirty or private pages swapped to disk, or program code that can be pulled from the binary on the filesystem if necessary, or malloced memory that hasn't been written to yet. -- Dan Nelson [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
In the last episode (Sep 11), Jesse Guardiani said: 1.) Where is my Free memory going? I can't account for it in the SIZE and RES columns of the various processes. These are relatively constant. Disk cache. 2.) What, exactly, is RES? `man top` describes it as this: RES is the current amount of resident memory, but does that mean RES is included in SIZE? Or does that mean that RES should be counted in addition to SIZE? RES the amount of SIZE that it currently in core, and thus should never exceed SIZE (I don't see any processes on my system that do at least). -- Dan Nelson [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: `top` process memory usage: SIZE vs RES
Hello Jesse, Thursday, September 11, 2003, 5:15:31 PM, you wrote: J I am stress testing a FreeBSD 4.7-RELEASE server (it's a pre- J production test server) by sending huge email messages to it J via SMTP. I too am putting together a 4.8 box as we speak for a client. J I'm running qmail-1.03 built from source, with the QMAILQUEUE J patch, qmail-qfilter, a custom Python script that runs under J qmail-qfilter, and qmail-scanner with ClamAV. J I test the server by sending a 59M or a 99M email from a J remote machine (connected via fxp0). J As I watch the email travel down the qmail-smtpd-qmail-qfilter- custom-python-qmail-scanner-clamd-qmail-queue pipeline, J I watch the memory usage with `top`. J Memory is critical in this type of application, since I run my J qmail-smtpd pipeline under DJB's softlimit program. I MUST J know how much memory to allocate for the upper limit of each J pipeline, otherwise qmail-smtpd will terminate the transfer J with a 451 SMTP error. right. J Anyway, as I watch `top`, I never see more than 15M being used J by the various pipeline programs at any given point in time, J but my Free Memory constantly declines until it reaches about J 526k. J The Questions: J == J 1.) Where is my Free memory going? given what you say custom-python-qmail-scanner-clamd-qmail-queue This whole scenario is very memory intensive. First you have each email pythonized and then qmail-scanner is *very* memory intensive, as it has initially a very heavy duty perl script for each email before being passed off to clamd. Multiply this with the default SMTP concurrency limit of 20, and you have a lot of memory usage. Clamd is known to be a memory piggie too. I switched from that to using fprot which uses less memory and is faster processing. J I can't account for it J in the SIZE and RES columns of the various processes. J These are relatively constant. Maybe running vmstat -w 1 would give you a different perspective also. -- Best regards, Gary ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
