Re: loss of shared memory in parent httpd (2)
Eric Frazier wrote: Hi, This may be totaly ignorate crap, but I noticed this when I was reading the ps man page on BSD 4.5 about sys/proc.h flags This one I noticed.. P_SYSTEM 0x00200System proc: no sigs, stats or swapping Could this mean what I think it means? That a process with this flag set, won't be swaped out?? I've spent some time with our friend google and here is what I came up with (it's hard to search for things which can be called in many different ways): I've searched for P_SYSTEM and it seems that it's a *BSD thing and also when you set it you don't get sigs delivered. e.g. see the discussion here: http://mail-index.netbsd.org/tech-kern/1998/09/08/0004.html There is also: madvise(2) - give advice about use of memory Has anybody tried to use it? Can this help? There is some discussion here: http://lists.insecure.org/linux-kernel/2001/Oct/0877.html Here is another observation and explanation of the swapping/mem unsharing phenomena on linux. http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0324.html http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0307.html Finally, apparently it's relatively easy to patch the linux kernel to enabling mlock for non-root processes: http://www.uwsg.iu.edu/hypermail/linux/kernel/9608.2/0280.html At 03:55 PM 3/12/02 +0100, Elizabeth Mattijsen wrote: Oops. Premature sending... I have two ideas that might help: - reduce number of global variables used, less memory pollution by lexicals - make sure that you have the most up-to-date (kernel) version of your OS. Newer Linux kernels seem to be a lot savvy at handling shared memory than older kernels. Again, I wish you strength in fixing this problem... Elizabeth Mattijsen http://www.kwinternet.com/eric (250) 655 - 9513 (PST Time Zone) -- _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
RE: loss of shared memory in parent httpd
Hi all, On Sat, 16 Mar 2002, Bill Marrs wrote: leads ones to wonder if some of our assumptions or tools used to monitor memory are inaccurate or we're misinterpreting them. Well 'top' on Linux is rubbish for sure. 73, Ged.
RE: loss of shared memory in parent httpd
Hi, I had hoped that FreeBSD would be immune, but it seems not. I have been bashing it with http_load and all of a sudden(after a LOT of bashing and swaping) all of my processes had zero shared. It did take me days of fiddling to run into this though. Thanks, Eric At 04:16 PM 3/16/02 -0500, Ed Grimm wrote: I believe I have the answer... The problem is that the parent httpd swaps, and any new children it creates load the portion of memory that was swaped from swap, which does not make it copy-on-write. The really annoying thing - when memory gets tight, the parent is the most likely httpd process to swap, because its memory is 99% idle. This issue aflicts Linux, Solaris, and a bunch of other OSes. The solution is mlockall(2), available under Linux, Solaris, and other POSIX.1b compliant OSes. I've not experimented with calling it from perl, and I've not looked at Apache enough to consider patching it there, but this system call, if your process is run as root, will prevent any and all swapping of your process's memory. If your process is not run as root, it returns an error. The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) Ed On Thu, 14 Mar 2002, Bill Marrs wrote: It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. -bill http://www.kwinternet.com/eric (250) 655 - 9513 (PST Time Zone)
Re: loss of shared memory in parent httpd (2)
Hi, This may be totaly ignorate crap, but I noticed this when I was reading the ps man page on BSD 4.5 about sys/proc.h flags This one I noticed.. P_SYSTEM 0x00200System proc: no sigs, stats or swapping Could this mean what I think it means? That a process with this flag set, won't be swaped out?? Thanks, Eric At 03:55 PM 3/12/02 +0100, Elizabeth Mattijsen wrote: Oops. Premature sending... I have two ideas that might help: - reduce number of global variables used, less memory pollution by lexicals - make sure that you have the most up-to-date (kernel) version of your OS. Newer Linux kernels seem to be a lot savvy at handling shared memory than older kernels. Again, I wish you strength in fixing this problem... Elizabeth Mattijsen http://www.kwinternet.com/eric (250) 655 - 9513 (PST Time Zone)
RE: loss of shared memory in parent httpd
I believe I have the answer... The problem is that the parent httpd swaps, and any new children it creates load the portion of memory that was swaped from swap, which does not make it copy-on-write. The really annoying thing - when memory gets tight, the parent is the most likely httpd process to swap, because its memory is 99% idle. This issue aflicts Linux, Solaris, and a bunch of other OSes. The solution is mlockall(2), available under Linux, Solaris, and other POSIX.1b compliant OSes. I've not experimented with calling it from perl, and I've not looked at Apache enough to consider patching it there, but this system call, if your process is run as root, will prevent any and all swapping of your process's memory. If your process is not run as root, it returns an error. The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) Ed On Thu, 14 Mar 2002, Bill Marrs wrote: It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. -bill
Re: loss of shared memory in parent httpd
Yes, this is my theory also. I figured this out a while back, and started a thread on this list, but since then haven't had enough time to investigate it further. The thread is here: http://mathforum.org/epigone/modperl/wherdtharvoi which includes some helpful hints from Doug on how to call mlockall() from the mod_perl parent process. HTH.. I'm very curious to know if this works. -Adi Ed Grimm wrote: I believe I have the answer... The problem is that the parent httpd swaps, and any new children it creates load the portion of memory that was swaped from swap, which does not make it copy-on-write. The really annoying thing - when memory gets tight, the parent is the most likely httpd process to swap, because its memory is 99% idle. This issue aflicts Linux, Solaris, and a bunch of other OSes. The solution is mlockall(2), available under Linux, Solaris, and other POSIX.1b compliant OSes. I've not experimented with calling it from perl, and I've not looked at Apache enough to consider patching it there, but this system call, if your process is run as root, will prevent any and all swapping of your process's memory. If your process is not run as root, it returns an error. The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) Ed On Thu, 14 Mar 2002, Bill Marrs wrote: It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. -bill
RE: loss of shared memory in parent httpd
The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) In my case, I'm using Linux 2.4.17, when I turn off swap and turn it back on again, it restores the shared memory of both the parent and the children Apache processes. This seems counter-intuitive, as it would seem the kernel memory manager would have to bend over backwards to accomplish this re-binding of the swapped-out shared memory pages. Thus, it leads ones to wonder if some of our assumptions or tools used to monitor memory are inaccurate or we're misinterpreting them. -bill
RE: loss of shared memory in parent httpd
It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. -bill
Re: loss of shared memory in parent httpd
On Thu, 14 Mar 2002 07:25:27 -0500, Bill Marrs [EMAIL PROTECTED] said:
It's copy-on-write. The swap is a write-to-disk.
There's no such thing as sharing memory between one process on disk(/swap)
and another in memory.
agreed. What's interesting is that if I turn swap off and back on
again, the sharing is restored! So, now I'm tempted to run a crontab
every 30 minutes that turns the swap off and on again, just to keep
the httpds shared. No Apache restart required!
Funny, I'm doing this for ages and I never really knew why, you just
found the reason, Thank You! My concerns were similar to yours but on
a smaller scale, so I did not worry that much, but I'm running a
swapflusher regularly.
Make sure you have a recent kernel, because all old kernels up to
2.4.12 or so were extremely unresponsive during swapoff. With current
kernels, this is much, much faster and nothing to worry about.
Let me show you the script I use for the job. No rocket science, but
it's easy to do it wrong. Be careful to maintain equality of priority
among disks:
use strict;
$|=1;
print Running swapon -a, just in case...\n;
system swapon -a;
print Running swapon -s\n;
open S, swapon -s |;
my(%prio);
PARTITION: while (S) {
print;
next if /^Filename/;
chop;
my($f,$t,$s,$used,$p) = split;
my $disk = $f;
$disk =~ s/\d+$//;
$prio{$disk} ||= 5;
$prio{$disk}--;
if ($used == 0) {
print Unused, skipping\n;
next PARTITION;
}
print Turning off\n;
system swapoff $f;
print Turning on with priority $prio{$disk}\n;
system swapon -p $prio{$disk} $f;
}
system swapon -s;
Let me know if you see room for improvements,
Regards,
--
andreas
[OT]RE: loss of shared memory in parent httpd
Call me an idiot.
How is it even remotely possible that turning off swap restores memory
shared between processes? Is the Linux kernel going from process to process
comparing pages of memory as they re-enter RAM? Oh, those two look
identical, they'll get shared?
-Incredulous
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 14, 2002 8:24 AM
To: Bill Marrs
Cc: [EMAIL PROTECTED]
Subject: Re: loss of shared memory in parent httpd
On Thu, 14 Mar 2002 07:25:27 -0500, Bill Marrs [EMAIL PROTECTED]
said:
It's copy-on-write. The swap is a write-to-disk.
There's no such thing as sharing memory between one process on
disk(/swap)
and another in memory.
agreed. What's interesting is that if I turn swap off and back on
again, the sharing is restored! So, now I'm tempted to run a crontab
every 30 minutes that turns the swap off and on again, just to keep
the httpds shared. No Apache restart required!
Funny, I'm doing this for ages and I never really knew why, you just
found the reason, Thank You! My concerns were similar to yours but on
a smaller scale, so I did not worry that much, but I'm running a
swapflusher regularly.
Make sure you have a recent kernel, because all old kernels up to
2.4.12 or so were extremely unresponsive during swapoff. With current
kernels, this is much, much faster and nothing to worry about.
Let me show you the script I use for the job. No rocket science, but
it's easy to do it wrong. Be careful to maintain equality of priority
among disks:
use strict;
$|=1;
print Running swapon -a, just in case...\n;
system swapon -a;
print Running swapon -s\n;
open S, swapon -s |;
my(%prio);
PARTITION: while (S) {
print;
next if /^Filename/;
chop;
my($f,$t,$s,$used,$p) = split;
my $disk = $f;
$disk =~ s/\d+$//;
$prio{$disk} ||= 5;
$prio{$disk}--;
if ($used == 0) {
print Unused, skipping\n;
next PARTITION;
}
print Turning off\n;
system swapoff $f;
print Turning on with priority $prio{$disk}\n;
system swapon -p $prio{$disk} $f;
}
system swapon -s;
Let me know if you see room for improvements,
Regards,
--
andreas
--
This message is intended only for the personal and confidential use of the designated
recipient(s) named above. If you are not the intended recipient of this message you
are hereby notified that any review, dissemination, distribution or copying of this
message is strictly prohibited. This communication is for information purposes only
and should not be regarded as an offer to sell or as a solicitation of an offer to buy
any financial product, an official confirmation of any transaction, or as an official
statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or
error-free. Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such. All information is subject to
change without notice.
Re: [OT]RE: loss of shared memory in parent httpd
How is it even remotely possible that turning off swap restores memory shared between processes? Is the Linux kernel going from process to process comparing pages of memory as they re-enter RAM? Oh, those two look identical, they'll get shared? This is a good point. I really have no clue how the kernel deals with swapping/sharing, so I can only speculate. I could imagine that it's possible for it to do this, if the pages are marked properly, they could be restored. But, I'll admit, it seems unlikely. ...and, I had this thought before. Maybe this apparent loss of shared memory is an illusion. It appears to make the amount of memory that the httpds use grow very high, but perhaps it is a kind of shared-swap, and thus the calculation I'm using to determine overall memory usage would need to also factor out swap. ...in which case, there's no problem at all. But, I do see an albeit qualitative performance increase and CPU load lowering when I get the httpds to stay shared (and unswapped). So, I think it does matter. Though, if you think about it, it sort of makes sense. Some portion of the shared part of the httpd is also not being used much, so it gets swapped out to disk. But, if those pages really aren't being used, then there shouldn't be a performance hit. If they are being used, then they'd get swapped back in. ...which sort of disproves my qualitative reasoning that swap/unshared is bad. my head hurts, maybe I should join a kernel mailing list and see is someone there can help me (and if I can understand them). -bill
RE: loss of shared memory in parent httpd
On Thu, 14 Mar 2002, Bill Marrs wrote: It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, what? doesn't seem to me like you are agreeing, and the original quote doesn't make sense either (because a shared page is a shared page, it can only be in one spot until/unless it gets copied). a shared page is swapped to disk. It then gets swapped back in, but for some reason the kernel seems to treat swapping a page back in as copying the page which doesn't seem logical ... anyone here got a more direct line with someone like Alan Cox? That is, _unless_ you copy all the swap space back in (e.g. swapoff)..., but that is probably a very different operation than demand paging. the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... the kernel doesn't want to swap out data in any case... if it does, it means memory pressure is reasonably high. AFAIK the kernel would far rather drop executable code pages which it can just go re-read ... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. or reduce MaxRequestsPerChild ? Stas also has some tools for causing children to exit early if their memory usage goes above some limit. I'm sure it's in the guide. -bill -- [EMAIL PROTECTED] | Courage is doing what you're afraid to do. http://BareMetal.com/ | There can be no courage unless you're scared. | - Eddie Rickenbacker
Re: [OT]RE: loss of shared memory in parent httpd
Bill Marrs wrote: You actually can do this. See the mergemem project: http://www.complang.tuwien.ac.at/ulrich/mergemem/ I'm interested in this, but it involves a kernel hack and the latest version is from 29-Jan-1999, so I got cold feet. It was a student project. And unless someone tells me differently wasn't picked up by community. In any case I've mentioned this as a proof of concept. Of course I'd love to see a working tool too. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: loss of shared memory in parent httpd
I just wanted to mention that the theory that my loss of shared memory in the parent is related to swapping seems to be correct. When the lack of sharing occurs, it is correlated with my httpd processes showing a SWAP (from top/ps) of 7.5MB, which is roughly equal to the amount of sharing that I lose. I've been lowering my MaxClients setting (from 25 to 10, so far) in hopes of finding a new balance where SWAP is not used, and more RAM is on order. Thanks -bill
RE: loss of shared memory in parent httpd
It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. You'll also want to look into tuning your paging algorithm. This will hold swapping at bay longer. The paging algorithm USED to be configurable with a bunch of kernal params (params.(h|c)) stuff like PAGEHANDSFREE but I really don't know anymore. HTH, -Josh -Original Message- From: Ask Bjoern Hansen [mailto:[EMAIL PROTECTED]] Sent: Tuesday, March 12, 2002 7:17 PM To: Graham TerMarsch Cc: Bill Marrs; [EMAIL PROTECTED] Subject: Re: loss of shared memory in parent httpd On Tue, 12 Mar 2002, Graham TerMarsch wrote: [...] We saw something similar here, running on Linux servers. Turned out to be that if the server swapped hard enough to swap an HTTPd out, then you basically lost all the shared memory that you had. I can't explain all of the technical details and the kernel-ness of it all, but from watching our own servers here this is what we saw on some machines that experienced quite a high load. Our quick solution was first to reduce some the number of Mod_perls that we had running, using the proxy-front-end/modperl-back-end technique, You should always do that. :-) and then supplemented that by adding another Gig of RAM to the machine. And yes, once you've lost the shared memory, there isn't a way to get it back as shared again. And yes, I've also seen that when this happens that it could full well take the whole server right down the toilet with it (as then your ~800MB of shared memory becomes ~800MB of _physical_ memory needed, and that could throw the box into swap city). I forwarded this mail to one of the CitySearch sysadmins who had told about seeing this. He is seeing the same thing (using kernel 2.4.17), except that if he disables swap then the processes will get back to reporting more shared memory. So maybe it's really just GTop or the kernel reporting swapped stuff in an odd way. No, I can't explain the nitty gritty either. :-) Someone should write up a summary of this thread and ask in a technical linux place, or maybe ask Dean Gaudet. - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do(); -- This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
Re: loss of shared memory in parent httpd
Stas Bekman wrote: Bill Marrs wrote: One more piece of advice: I find it easier to tune memory control with a single parameter. Setting up a maximum size and a minumum shared size is not as effective as setting up a maximum *UNSHARED* size. After all, it's the amount of real memory being used by each child that you care about, right? Apache::SizeLimit has this now, and it would be easy to add to GTopLimit (it's just $SIZE - $SHARED). Doing it this way helps avoid unnecessary process turnover. I agree. For me, with my ever more bloated Perl code, I find this unshared number to be easier to keep a lid on. I keep my apache children under 10MB each unshared as you say. That number is more stable that the SIZE/SHARED numbers that GTopLimmit offers. But, I have the GTopLimit sources, so I plan to tweak them to allow for an unshared setting. I think I bugged Stas about this a year ago and he had a reason why I was wrong to think this way, but I never understood it. I don't remember why I was argueing against it :) But in any case, I'll simply add this third option, so you can control it by either SIZE/SHARED or UNSHARED. it's on CPAN now. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: loss of shared memory in parent httpd
At 09:18 AM 3/12/02 -0500, Bill Marrs wrote: If anyone has any ideas what might cause the httpd parent (and new children) to lose a big chunk of shared memory between them, please let me know. I've seen this happen many times. One day it works fine, the next you're in trouble. And in my experience, it's not a matter of why this avalanche effect happens, but is it more a matter of why didn't it happen before? You may not have realised it that you were just below a threshold and now you're over it. And the change can be as small as the size of a heavily used template that suddenly gets over an internal memory allocation border, which in turn causes Perl to allocate more, which in turn causes memory to become unshared. I have been thinking about a perl/C routine that would internally use all of the memory that was already allocated by Perl. Such a routine would need to be called when the initial start of Apache is complete so that any child that is spawned has a saturated memory pool, so that any new variables would need to use newly allocated memory, which would be unshared. But at least all of that memory would be used for new variables and not have the tendency to pollute old memory segments. I'm not sure whether my assessment of the problem is correct. I would welcome any comments on this. I have two ideas that might help: - - other than making sure that you have the most up-to-date (kernel) version of your OS. Older Linux kernels seem to have this problem a lot more than newer kernels. I wish you strength in fixing this problem... Elizabeth Mattijsen
Re: loss of shared memory in parent httpd (2)
Oops. Premature sending... I have two ideas that might help: - reduce number of global variables used, less memory pollution by lexicals - make sure that you have the most up-to-date (kernel) version of your OS. Newer Linux kernels seem to be a lot savvy at handling shared memory than older kernels. Again, I wish you strength in fixing this problem... Elizabeth Mattijsen
Re: loss of shared memory in parent httpd
Bill Marrs wrote: I'm a heavy mod_perl user, running 3 sites as virtual servers, all with lots of custom Perl code. My httpd's are huge(~50mb), but with the help of a startup file I'm able to get them sharing most of their memory(~43). With the help of GTopLimit, I'm able to keep the memory usage under control. But... recently, something happened, and things have changed. After some random amount of time (1 to 40 minutes or so, under load), the parent httpd suddenly loses about 7-10mb of share between it and any new child it spawns. As you can imagine, the memory footprint of my httpds skyrockets and the delicate balance I set up is disturbed. Also, GTopLimit is no help in this case - it actually causes flailing because each new child starts with memory sharing that is out of bounds and is thus killed very quickly. Restarting Apache resets the memory usage and restores the server to smooth operation. Until, it happens again. Using GTop() to get the shared memory of each child before and after running my perl for each page load showed that it wasn't my code causing the jump, but suddenly the child, after having a good amount of shared memory in use, loses a 10MB chunk and from then on the other children follow suit. So, something I did on the server (I'm always doing stuff!) has caused this change to happen, but I've been pulling my hair out for days trying to track it down. I am now getting desperate. One of the recent things I did was to run tux (another web server) to serve my images, but I don't see how that could have any effect on this. If anyone has any ideas what might cause the httpd parent (and new children) to lose a big chunk of shared memory between them, please let me know. I assume that you are on linux (tux :). look at the output of 'free' -- how much swap is used before you start the server and when the horrors begin? Your delicate ballance could be ruined when the system starts to swap and the load doesn't go away. Therefore what you see is normal. Notice that it's possible that you didn't add a single line of code to your webserver, but updated some other app running on the same machine which started to use more memory, and that takes the ballance off. Hope that my guess was right. If so make sure that your system never swaps. Swap is for emergency short term extra memory requirement, not for normal operation. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: loss of shared memory in parent httpd
On Tue, 12 Mar 2002 09:18:32 -0500 Bill Marrs [EMAIL PROTECTED] wrote: But... recently, something happened, and things have changed. After some random amount of time (1 to 40 minutes or so, under load), the parent httpd suddenly loses about 7-10mb of share between it and any new child it spawns. As you can imagine, the memory footprint of my httpds skyrockets and the delicate balance I set up is disturbed. Also, GTopLimit is no help Restarting Apache resets the memory usage and restores the server to smooth operation. Until, it happens again. Hi Bill I can't give you a decent answer, but I have noticed this as well, and my impression is that this happens when your httpd's are swapped out (when your system runs short of free memory) - or perhaps just when the parent httpd is swapped out (been a while - can't remember exactly what symptoms I observed). I think the whole phenomenon is a side-effect of normal memory management - I'm sure someone on the list will have a proper explanation. Bye Paolo
Re: loss of shared memory in parent httpd
Elizabeth Mattijsen wrote: At 09:18 AM 3/12/02 -0500, Bill Marrs wrote: If anyone has any ideas what might cause the httpd parent (and new children) to lose a big chunk of shared memory between them, please let me know. I've seen this happen many times. One day it works fine, the next you're in trouble. And in my experience, it's not a matter of why this avalanche effect happens, but is it more a matter of why didn't it happen before? You may not have realised it that you were just below a threshold and now you're over it. And the change can be as small as the size of a heavily used template that suddenly gets over an internal memory allocation border, which in turn causes Perl to allocate more, which in turn causes memory to become unshared. I have been thinking about a perl/C routine that would internally use all of the memory that was already allocated by Perl. Such a routine would need to be called when the initial start of Apache is complete so that any child that is spawned has a saturated memory pool, so that any new variables would need to use newly allocated memory, which would be unshared. But at least all of that memory would be used for new variables and not have the tendency to pollute old memory segments. I'm not sure whether my assessment of the problem is correct. I would welcome any comments on this. Nope Elizabeth, your explanation is not so correct. ;) Shared memory is not about sharing the pre-allocated memory pool (heap memory). Once you re-use a bit of preallocated memory the sharing goes away. Shared memory is about 'text'/read-only memory pages which never get modified and pages that can get modified but as long as they aren't modified they are shared. Unfortunately (in this aspect, but fortunately for many other aspects) Perl is not a strongly-typed (or whatever you call it) language, therefore it's extremely hard to share memory, because in Perl almost everything is data. Though as you could see Bill was able to share 43M out of 50M which is damn good! _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: loss of shared memory in parent httpd
At 11:46 PM 3/12/02 +0800, Stas Bekman wrote: I'm not sure whether my assessment of the problem is correct. I would welcome any comments on this. Nope Elizabeth, your explanation is not so correct. ;) Too bad... ;-( Shared memory is not about sharing the pre-allocated memory pool (heap memory). Once you re-use a bit of preallocated memory the sharing goes away. I think the phrase is Copy-On-Write, right? And since RAM is allocated in chunks, let's assume 4K for the sake of the argument, changing a single byte in such a chunk causes the entire chunk to be unshared. In older Linux kernels, I believe to have seen that when a byte gets changed in a chunk of any child, that chunk becomes changed for _all_ children. Newer kernels only unshare it for that particular child. Again, if I'm not mistaken and someone please correct me if I'm wrong... Since Perl is basically all data, you would need to find a way of localizing all memory that is changing to as few memory chunks as possible. My idea was just that: by filling up all used memory before spawning children, you would use op some memory, but that would be shared between all children and thus not so bad. But by doing this, you would hopefully cause all changing data to be localized to newly allocated memory by the children. Wish someone with more Perl guts experience could tell me if that really is an idea that could work or not... Shared memory is about 'text'/read-only memory pages which never get modified and pages that can get modified but as long as they aren't modified they are shared. Unfortunately (in this aspect, but fortunately for many other aspects) Perl is not a strongly-typed (or whatever you call it) language, therefore it's extremely hard to share memory, because in Perl almost everything is data. Though as you could see Bill was able to share 43M out of 50M which is damn good! As a proof of concept I have run more than 100 200MB+ children on a 1 GB RAM machine and had sharing go up so high causing the top number of bytes field for shared memory to cycle through its 32-bit range multiple times... ;-) . It was _real_ fast (had all of its data that it needed as Perl hashes and lists) and ran ok until something would start an avalanche effect and it would all go down in a whirlwind of swapping. So in the end, it didn't work reliably enough ;-( But man, was it fast when it ran... ;-) Elizabeth Mattijsen
Re: loss of shared memory in parent httpd
At 12:43 AM 3/13/02 +0800, Stas Bekman wrote: Doug has plans for a much improved opcode tree sharing for mod_perl 2.0, the details are kept as a secret so far :) Can't wait to see that! This topic is covered (will be) in the upcoming mod_perl book, where we include the following reference materials which you may find helpful for understanding the shared memory concepts. Ah... ok... can't wait for that either... ;-) Don't you love mod_perl for what it makes you learn :) Well, yes and no... ;-) Elizabeth Mattijsen
Re: loss of shared memory in parent httpd
Elizabeth Mattijsen wrote: Since Perl is basically all data, you would need to find a way of localizing all memory that is changing to as few memory chunks as possible. That certainly would help. However, I don't think you can do that in any easy way. Perl doesn't try to keep compiled code on separate pages from variable storage. - Perrin
Re: loss of shared memory in parent httpd
On Tuesday 12 March 2002 06:18, you wrote: I'm a heavy mod_perl user, running 3 sites as virtual servers, all with lots of custom Perl code. My httpd's are huge(~50mb), but with the help of a startup file I'm able to get them sharing most of their memory(~43). With the help of GTopLimit, I'm able to keep the memory usage under control. But... recently, something happened, and things have changed. After some random amount of time (1 to 40 minutes or so, under load), the parent httpd suddenly loses about 7-10mb of share between it and any new child it spawns. As you can imagine, the memory footprint of my httpds skyrockets and the delicate balance I set up is disturbed. Also, GTopLimit is no help in this case - it actually causes flailing because each new child starts with memory sharing that is out of bounds and is thus killed very quickly. We saw something similar here, running on Linux servers. Turned out to be that if the server swapped hard enough to swap an HTTPd out, then you basically lost all the shared memory that you had. I can't explain all of the technical details and the kernel-ness of it all, but from watching our own servers here this is what we saw on some machines that experienced quite a high load. Our quick solution was first to reduce some the number of Mod_perls that we had running, using the proxy-front-end/modperl-back-end technique, and then supplemented that by adding another Gig of RAM to the machine. And yes, once you've lost the shared memory, there isn't a way to get it back as shared again. And yes, I've also seen that when this happens that it could full well take the whole server right down the toilet with it (as then your ~800MB of shared memory becomes ~800MB of _physical_ memory needed, and that could throw the box into swap city). -- Graham TerMarsch Howling Frog Internet Development, Inc. http://www.howlingfrog.com
Re: loss of shared memory in parent httpd
Bill Marrs wrote: But... recently, something happened, and things have changed. After some random amount of time (1 to 40 minutes or so, under load), the parent httpd suddenly loses about 7-10mb of share between it and any new child it spawns. One possible reason is that a perl memory structure in there might be changing. Perl is able to grow variables dynamically by allocating memory in buckets, and it tends to be greedy when grabbing more. You might trigger another large allocation by something as simple as implicitly converting a string to a number, or adding one element to an array. Over time, I always see the parent process lose some shared memory. My advice is to base your tuning not on the way it looks right after you start it, but on the way it looks after serving pages for a few hours. Yes, you will underutilize the box just after a restart, but you will also avoid overloading it when things get going. I also recommend restarting your server every 24 hours, to reset things. One more piece of advice: I find it easier to tune memory control with a single parameter. Setting up a maximum size and a minumum shared size is not as effective as setting up a maximum *UNSHARED* size. After all, it's the amount of real memory being used by each child that you care about, right? Apache::SizeLimit has this now, and it would be easy to add to GTopLimit (it's just $SIZE - $SHARED). Doing it this way helps avoid unnecessary process turnover. - Perrin
Re: loss of shared memory in parent httpd
Thanks for all the great advice. A number of you indicated that it's likely due to my apache processes being partially swapped to disk. That seems likely to me. I haven't had a chance to prove that point, but when it does it again and I'm around, I plan to test it with free/top (top has a SWAP column which should show if my apaches are swapped out at all). I am in the process of getting a memory upgrade, so that should ease this problem. Meanwhile, I can set MaxClients lower and see if that keeps me out of trouble as well. I suspect adding the tux server disrupted the balance I had (apparently, I was tuned pretty close to my memory limits!) Yes, I am running on Linux... One more piece of advice: I find it easier to tune memory control with a single parameter. Setting up a maximum size and a minumum shared size is not as effective as setting up a maximum *UNSHARED* size. After all, it's the amount of real memory being used by each child that you care about, right? Apache::SizeLimit has this now, and it would be easy to add to GTopLimit (it's just $SIZE - $SHARED). Doing it this way helps avoid unnecessary process turnover. I agree. For me, with my ever more bloated Perl code, I find this unshared number to be easier to keep a lid on. I keep my apache children under 10MB each unshared as you say. That number is more stable that the SIZE/SHARED numbers that GTopLimmit offers. But, I have the GTopLimit sources, so I plan to tweak them to allow for an unshared setting. I think I bugged Stas about this a year ago and he had a reason why I was wrong to think this way, but I never understood it. -bill
Re: loss of shared memory in parent httpd
On Tue, 12 Mar 2002, Graham TerMarsch wrote: [...] We saw something similar here, running on Linux servers. Turned out to be that if the server swapped hard enough to swap an HTTPd out, then you basically lost all the shared memory that you had. I can't explain all of the technical details and the kernel-ness of it all, but from watching our own servers here this is what we saw on some machines that experienced quite a high load. Our quick solution was first to reduce some the number of Mod_perls that we had running, using the proxy-front-end/modperl-back-end technique, You should always do that. :-) and then supplemented that by adding another Gig of RAM to the machine. And yes, once you've lost the shared memory, there isn't a way to get it back as shared again. And yes, I've also seen that when this happens that it could full well take the whole server right down the toilet with it (as then your ~800MB of shared memory becomes ~800MB of _physical_ memory needed, and that could throw the box into swap city). I forwarded this mail to one of the CitySearch sysadmins who had told about seeing this. He is seeing the same thing (using kernel 2.4.17), except that if he disables swap then the processes will get back to reporting more shared memory. So maybe it's really just GTop or the kernel reporting swapped stuff in an odd way. No, I can't explain the nitty gritty either. :-) Someone should write up a summary of this thread and ask in a technical linux place, or maybe ask Dean Gaudet. - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do();
Re: loss of shared memory in parent httpd
No, I can't explain the nitty gritty either. :-) Someone should write up a summary of this thread and ask in a technical linux place, or maybe ask Dean Gaudet. I believe this is a linux/perl issue... stand alone daemons exhibit the same behaviour... e.g. if you've got a parent PERL daemon that fork()s ... swapping in data from a child does _not_ have any affect on other copies of that memory. I'm sure swapping in the memory of the parent before fork()ing would be fine Admittedly, my experience is from old linux kernels (2.0), but I would not be suprised if current ones are similar. I'm sure it is the same on some other platforms, but I haven't used much else for a long time. -- [EMAIL PROTECTED] | Put all your eggs in one basket and http://BareMetal.com/ | WATCH THAT BASKET! web hosting since '95 | - Mark Twain
Re: loss of shared memory in parent httpd
Bill Marrs wrote: One more piece of advice: I find it easier to tune memory control with a single parameter. Setting up a maximum size and a minumum shared size is not as effective as setting up a maximum *UNSHARED* size. After all, it's the amount of real memory being used by each child that you care about, right? Apache::SizeLimit has this now, and it would be easy to add to GTopLimit (it's just $SIZE - $SHARED). Doing it this way helps avoid unnecessary process turnover. I agree. For me, with my ever more bloated Perl code, I find this unshared number to be easier to keep a lid on. I keep my apache children under 10MB each unshared as you say. That number is more stable that the SIZE/SHARED numbers that GTopLimmit offers. But, I have the GTopLimit sources, so I plan to tweak them to allow for an unshared setting. I think I bugged Stas about this a year ago and he had a reason why I was wrong to think this way, but I never understood it. I don't remember why I was argueing against it :) But in any case, I'll simply add this third option, so you can control it by either SIZE/SHARED or UNSHARED. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
