Danger: Rant ahead. Proceed with caution. On Sat, 23 Mar 2002, Stas Bekman wrote:
> See the discussion on the [EMAIL PROTECTED] list, > http://marc.theaimsgroup.com/?t=101659730800001&r=1&w=2 where it was > said that it's a very bad idea to use mlock and variants. Moreover the > memory doesn't get unshared when the parent pages are paged out, it's > the reporting tools that report the wrong information and of course > mislead the the size limiting modules which start killing the > processes. As a conclusion to this thread I've added the following > section to the performance chapter of the guide: > > =head3 Potential Drawbacks of Memory Sharing Restriction > > It's very important that the system won't be heavily engaged in > swapping process. Some systems do swap in and out every so often even > if they have plenty of real memory available and it's OK. The > following applies to conditions when there is hardly any free memory > available. > > So if the system uses almost all of its real memory (including the > cache), there is a danger of parent's process memory pages being > swapped out (written to a swap device). If this happens the memory > usage reporting tools will report all those swapped out pages as > non-shared, even though in reality these pages are still shared on > most OSs. When these pages are getting swapped in, the sharing will be My Solaris 2.6 box, while in this situation, was swapping hard, as measured by my ears, by iostat, and by top (both iowait and the memory stats). Note that mlockall does not restrict memory sharing, it restricts swapping a certain portion of memory. This will prevent this memory from ever being needlessly unshared. In the discussion you referred to, all of the people saying this was a bad idea were using terms like, "I think". None of them had the situation themselves, so have a difficult time coming to terms with it. None of them had related former experience using this. Something like this really needs to be tested by someone who has the issue, and has the ability to do benchmarks with real data streams. If they find it seems to work well, then they should test it on production systems. Anyone else talking about it is simply that much hot air, myself included. (I *could* test it, but I don't have enough of a problem to put a priority on it. If we were waiting for me to get time, we'd be waiting a long time.) Yes, I agree, it's better to never swap. But if we take the attitude that we won't use tools to help us when times are tight, then get rid of swap entirely. Locking memory is all about being selective about what you will and won't swap. Yes, I agree, it'd be better to mlock those bits of memory that you really care about, but that's hard to do when that memory is allocated by software you didn't write. (In this case, I'd really like to mlock all the memory that perl allocated but did not free in BEGIN sections (including, of course, use blocks). I would also like to compact that first, but that could be even more difficult.) As far as the logic regarding 'let the OS decide' - the admin of the system has the ability to have a much better understanding of how the system resources are used. If I have one section of memory which is used 95% of the time by 75% of my active processes, I really don't want that memory to swap out just because another program that'll only run for a minute wants a bit more memory, if it can take that memory from anywhere else. When doing individual page-ins, memory managers tend to worry only about those processes that they are trying to make runable now; they're not going to go and load that page back on to every other page map that shares it just because they also use it. So even though that memory is loaded back into memory, all those processes will still have to swap it back. For them to do otherwise would be irresponsible, unless the system administrator clearly doesn't know how to system administrate, or has chosen not to. The OS is supposed to handle the typical case; having one segment of memory used by dozens of processes actively is not the typical case. This does not happen on end-user machines; this only happens on servers. Theoretically speaking, servers are run by people who can analyze and tune; mlock and mlockall are tools available to them to do such tuning. > reported back to normal after a certain amount of time. If a big chunk > of the memory shared with child processes is swapped out, it's most > likely that C<Apache::SizeLimit> or C<Apache::GTopLimit> will notice > that the shared memory floor threshold was crossed and as a result > kill those processes. If many of the parent process' pages are swapped > out, and the newly created child process is already starting with > shared memory below the limit, it'll be killed immediately after > serving a single request (assuming that we the > C<$CHECK_EVERY_N_REQUESTS> is set to one). This is a very bad > situation which will eventually lead to a state where the system won't > respond at all, as it'll be heavily engaged in swapping process. Yes, this is why we want to lock the memory. Ed
