Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On 2000-10-11 19:53:50 -0700, [EMAIL PROTECTED] wrote: > On other machines I'd set RLIMIT_DATA and my OOM problems went away, > but on linux this didn't work RLIMIT_DATA appears to only be checked for aout format executables. Looking at the 2.4.0-test10pre1 sources for fs/binfmt_aout.c and fs/binfmt_elf.c you'll note the difference in load_aout_binary() and load_elf_binary(), both just above the comment of "OK, This is the point of no return" Does putting a similar check to the aout one make sense for ELF? I'm just trying to avoid Rik having to pull his hair out implementing a system that conceptually already exists in the kernel (nasty processes being terminated before they do some damage). Especially when that existing system is far more configurable. Cheers, -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On 2000-10-11 11:45:06 -0400, Bruce A. Locke wrote: > This manpage shows me functions and structs. What were you expecting from the system call section of the Linux Programmer's Manual? Dancing girls? (h...) > I'm assuming you want these used by the offending program or the shell > under which the program is being called. That's usually what happens. > In the first case, a person might not have source to the program and > if thats the case, it doesn't help much. Closed-source software is *so* 20th century... ;-) Anyway, when run from the shell it'll inherit its parent's limits (which leads to your next question...) > And in the second case, if the shell sets it, does it affect children > of a process (aka fork()'d)? Certainly. Maybe if more distributions took Debian's stance and set the default limits so anal that you frequently can't even read email let alone recompile the kernel without getting the process terminated for tripping one limit or another, then more people would know this functionality exists and set the limits more appropriately. -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On 2000-10-11 12:48:54 -0400, Andrew Pimlott wrote: > No way should a desktop user be responsible for micro-managing the > resource usage of his applications. That's right. The systems administrator should, and will set appropriate limits for users on his/her system that apply from login. This is how the systems I first used were configured (lucky me had a damn fine sysadmin), and so this is how I configure mine. > The only thing that knows what's right for Netscape is Netscape. I would disagree with this, I believe this is exactly the root of people's problems with Netscape (and the same theory should apply to other apps). The application doesn't know what's _right_ - it knows what it _wants_. Big difference. -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On Thu, Oct 12, 2000 at 01:58:49AM +1100, Matthew Hawkins wrote: > On 2000-10-11 10:33:39 -0400, Bruce A. Locke wrote: > > > > Your making the deadly assumption that all applications behave themselves > > exactly the same all the time. Oops... netscape decided to freak out and > > take up all your memory... guess its the admins fault. > > Yep, for not setting appropriate resource limits. No way should a desktop user be responsible for micro-managing the resource usage of his applications. How can he decide what's reasonable for Netscape to consume? Shouldn't Netscape be allowed to take up most of memory, if it's the only major application and the memory will improve its performance? The only thing that knows what's right for Netscape is Netscape. If Netscape were clever and kind, perhaps it would estimate what's reasonable and set limits on itself, adjusting them from time to time based on user behavior and environmental factors. But Netscape's a pretty mature program, and it doesn't do this; it can hardly be expected of the zillions of immature (and probably leaky) applications a user might run. So, we inevitably need an automated low-memory or out-of-memory algorithm. I tend to think it may need to be more adjustable than Rik's--people will be much more comfortable if they can say "spare this simulation at all cost!" or "kill off one of these processes in an emergency" or "this system has no business coming within 90% of RAM+swap capacity, so start killing things at that point--oh, and mail me". Some of this has no place in the kernel, obviously. But Rik has a good start, and perhaps his work will be part of a more complete solution. Andrew - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
- Received message begins Here - > > On 2000-10-11 09:45:30 -0500, Jesse Pollard wrote: > > Until user memory resource quotas are included in the kernel, there will be > > nothing else that can be done. Even with resource quotas, if the total of > > active users exceeds the resource then the same/equivalent situation occurs. > > So setrlimit() with RLIMIT_DATA, RLIMIT_STACK, RLIMIT_RSS, > RLIMIT_MEMLOCK, RLIMIT_AS et al is a null op? > > If so, I wish to register a complaint ;-) Not exactly. As I have seen it, each process gets a copy of these limits. A single process cannot exceed the limit, but the sum of all processes can. One of the problems is cause by COW: given trivially small limits (1 MB) The first process allocates and initializes up to one MB, then forks. The second process begins updating data - .5MB. Neither process exceeds the limits, but the sum is now 1.5MB. If this is repeated enough, then the system can go OOM, with none of the processes at or over the limits set. Another problem occurs on multi-user servers. Each user logs in and gets "reasonable" rlimit values - each user uses one medium sized process. If the #users * rlimits exceeds the system capacity then OOM could occur, and still none may have exceeded the rlimit. I've always treated rlimit values as "suggestions" to the user process to aid in debugging. (this is more applicable to the ulimits though). The users process will not exeed the value, and when they do it is a strong suggestion that a bug may be present. (I first saw this with a leakey X server.) There have been some patches (the beancounter stuff) that does relate to resource control, but a more integrated resource accounting will make it work better. I do believe it should be available as an option, especially for multi-user servers, clusters, and other large systems. It isn't that usefull on single user workstations. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On Thu, 12 Oct 2000, Matthew Hawkins wrote:
>
> Seriously, am I missing something obvious or is it far simpler just to
> keel over and die if the system goes OOM? I mean, seriously, if the
> administrator lets it get to that state then he/she/it deserves a dead
> system. It's akin to having your car run out of petrol - you don't
> start shooting passengers because their extra load made the engine chew
> more. You pack up your kitty and go to the nearest petrol station and
> buy more, plug it into the car then learn from the experience so this
> fringe case of it happening doesn't happen again. I don't really see
> much difference between a car going "OOP" and a computer going OOM.
> Should we start deleting files according to some randomly-chosen
> heueristic if a filesystem goes "OOS" ?
Excellent point. However, the idea is to kill an attacker if your 'car'
is being hijacked.
Whatever is being designed should ideally have zero impact on the usual
performance and only come into play if something runs away, deliberately
or by accident.
If Linux doesn't track down and kill deliberate attempts to kill the
system, there will always be those who say; "Linux is no good because
a user can readily kill it". Of course we could track down and
kill those who say this, but it'd get messy.
FYI, a fork() bomb on my Sun Workstation does not kill it. Also
malloc()ing and writing all over the place doesn't kill it either.
Script started on Wed Oct 11 10:41:38 2000
# cat xxx.c
main()
{
for(;;)
fork();
}
# gcc -o xxx xxx.c
# ./xxx
^C
# # ^C
# ps
PID TTY TIME CMD
24800 pts/10:00 xxx
24335 pts/10:00 sh
24688 pts/10:00 xxx
24690 pts/10:00 xxx
24692 pts/10:00 xxx
24694 pts/10:00 xxx
24696 pts/10:00 xxx
24697 pts/10:00 xxx
24699 pts/10:00 xxx
24701 pts/10:00 xxx
24703 pts/10:00 xxx
24704 pts/10:00 xxx
24706 pts/10:00 xxx
24708 pts/10:00 xxx
24710 pts/10:00 xxx
24712 pts/10:00 xxx
24714 pts/10:00 xxx
24716 pts/10:00 xxx
24717 pts/10:00 xxx
24719 pts/10:00 xxx
24720 pts/10:00 xxx
24721 pts/10:00 xxx
24722 pts/10:00 xxx
24723 pts/10:00 xxx
24724 pts/10:00 xxx
24725 pts/10:00 xxx
24726 pts/10:00 xxx
24727 pts/10:00 xxx
24728 pts/10:00 xxx
24729 pts/10:00 xxx
24730 pts/10:00 xxx
24731 pts/10:00 xxx
24732 pts/10:00 xxx
24733 pts/10:00 xxx
24734 pts/10:00 xxx
24735 pts/10:00 xxx
24736 pts/10:00 xxx
24737 pts/10:00 xxx
24738 pts/10:00 xxx
24739 pts/10:00 xxx
24740 pts/10:00 xxx
24741 pts/10:00 xxx
24742 pts/10:00 xxx
24743 pts/10:00 xxx
24744 pts/10:00 xxx
24801 pts/10:00 ps
24687 pts/10:00 xxx
24689 pts/10:00 xxx
24691 pts/10:00 xxx
24693 pts/10:00 xxx
24695 pts/10:00 xxx
24698 pts/10:00 xxx
24700 pts/10:00 xxx
24702 pts/10:00 xxx
24705 pts/10:00 xxx
24707 pts/10:00 xxx
24709 pts/10:00 xxx
24711 pts/10:00 xxx
24713 pts/10:00 xxx
24715 pts/10:00 xxx
24718 pts/10:00 xxx
24653 pts/10:00 xxx
24610 pts/10:00 xxx
24614 pts/10:00 xxx
24615 pts/10:00 xxx
24616 pts/10:00 xxx
24617 pts/10:00 xxx
24618 pts/10:00 xxx
24619 pts/10:00 xxx
24620 pts/10:00 xxx
24621 pts/10:00 xxx
24622 pts/10:00 xxx
24623 pts/10:00 xxx
24624 pts/10:00 xxx
24625 pts/10:00 xxx
24626 pts/10:00 xxx
24627 pts/10:00 xxx
24628 pts/10:00 xxx
24629 pts/10:00 xxx
24630 pts/10:00 xxx
24631 pts/10:00 xxx
24632 pts/10:00 xxx
24686 pts/10:00 xxx
24685 pts/10:00 xxx
24684 pts/10:00 xxx
24683 pts/10:00 xxx
24682 pts/10:00 xxx
24681 pts/10:00 xxx
24680 pts/10:00 xxx
24679 pts/10:00 xxx
24678 pts/10:00 xxx
24677 pts/10:00 xxx
24676 pts/10:00 xxx
24675 pts/10:00 xxx
24674 pts/10:00 xxx
24673 pts/10:00 xxx
24672 pts/10:00 xxx
24671 pts/10:00 xxx
24670 pts/10:00 xxx
24669 pts/10:00 xxx
24668 pts/10:00 xxx
24667 pts/10:00 xxx
24666 pts/10:00 xxx
24665 pts/10:00 xxx
24664 pts/10:00 xxx
24663 pts/10:00 xxx
24662 pts/10:00 xxx
24661 pts/10:00 xxx
24660 pts/10:00 xxx
24659 pts/10:00 xxx
24658 pts/10:00 xxx
24657 pts/10:00 xxx
24656 pts/10:00 xxx
24655 pts/10:00 xxx
24654 pts/10:00 xxx
24652 pts/10:00 xxx
24651 pts/10:00 xxx
24650 pts/10:00 xxx
24649 pts/10:00 xxx
24648 pts/10:00 xxx
24647 pts/10:00 xxx
24646 pts/10:00 xxx
24645 pts/10:00 xxx
24644 pts/10:00 xxx
24643 pts/10:00 xxx
24642 pts/10:00 xxx
24634 pts/10:00 xxx
24633 pts/10:00 xxx
24641 pts/10:00 xxx
24640 pts/10:00 xxx
24639 pts/10:00 xxx
24638 pts/10:00 xxx
24637 pts/10:00 xxx
24636 pts/10:00 xxx
24635 pts/10:00 xxx
24613 pts/1
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On 2000-10-11 10:33:39 -0400, Bruce A. Locke wrote: > > Your making the deadly assumption that all applications behave themselves > exactly the same all the time. Oops... netscape decided to freak out and > take up all your memory... guess its the admins fault. Yep, for not setting appropriate resource limits. man 2 setrlimit Of course, if its a kernel bug that causes it I think you're SOL ;) -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On 2000-10-11 09:45:30 -0500, Jesse Pollard wrote:
> Until user memory resource quotas are included in the kernel, there will be
> nothing else that can be done. Even with resource quotas, if the total of
> active users exceeds the resource then the same/equivalent situation occurs.
So setrlimit() with RLIMIT_DATA, RLIMIT_STACK, RLIMIT_RSS,
RLIMIT_MEMLOCK, RLIMIT_AS et al is a null op?
If so, I wish to register a complaint ;-)
--
* Matthew Hawkins <[EMAIL PROTECTED]> :(){ :|:&};:
** Information Specialist, tSA Group Pty. Ltd. Ph: +61 2 6257 7111
*** 1 Hall Street, Lyneham ACT 2602 Australia. Fx: +61 2 6257 7311
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
- Received message begins Here - > > > Heh.. now all we need is some smart-arse to make something similar to > apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy > ;) > > Seriously, am I missing something obvious or is it far simpler just to > keel over and die if the system goes OOM? I mean, seriously, if the > administrator lets it get to that state then he/she/it deserves a dead > system. It's akin to having your car run out of petrol - you don't > start shooting passengers because their extra load made the engine chew > more. You pack up your kitty and go to the nearest petrol station and > buy more, plug it into the car then learn from the experience so this > fringe case of it happening doesn't happen again. I don't really see > much difference between a car going "OOP" and a computer going OOM. > Should we start deleting files according to some randomly-chosen > heueristic if a filesystem goes "OOS" ? Not deleting files, but your system may crash :) The problem with memory is that the tools are not available (ie already included in the kernel) to do anything else. In the example of running out of file space, there are quota limits. You can still run out of space, but only when the sum of all users quota allocations exceed the disk capacity. Until user memory resource quotas are included in the kernel, there will be nothing else that can be done. Even with resource quotas, if the total of active users exceeds the resource then the same/equivalent situation occurs. What is being done is still necessary, but in the long term it will end up addressing the case where a single user runs out, rather than the system as a whole. User memory resource quota control is needed in large clusters, and in large systems with multiple users. In a single user environment, resource quotas are less important than providing a consistant (and hopefully intutitive) process abort. That keeps the system going, and becomes up to the user to choose what else may need to be aborted. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
Heh.. now all we need is some smart-arse to make something similar to apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy ;) Seriously, am I missing something obvious or is it far simpler just to keel over and die if the system goes OOM? I mean, seriously, if the administrator lets it get to that state then he/she/it deserves a dead system. It's akin to having your car run out of petrol - you don't start shooting passengers because their extra load made the engine chew more. You pack up your kitty and go to the nearest petrol station and buy more, plug it into the car then learn from the experience so this fringe case of it happening doesn't happen again. I don't really see much difference between a car going "OOP" and a computer going OOM. Should we start deleting files according to some randomly-chosen heueristic if a filesystem goes "OOS" ? -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
Heh.. now all we need is some smart-arse to make something similar to apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy ;) Seriously, am I missing something obvious or is it far simpler just to keel over and die if the system goes OOM? I mean, seriously, if the administrator lets it get to that state then he/she/it deserves a dead system. It's akin to having your car run out of petrol - you don't start shooting passengers because their extra load made the engine chew more. You pack up your kitty and go to the nearest petrol station and buy more, plug it into the car then learn from the experience so this fringe case of it happening doesn't happen again. I don't really see much difference between a car going "OOP" and a computer going OOM. Should we start deleting files according to some randomly-chosen heueristic if a filesystem goes "OOS" ? -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
Your making the deadly assumption that all applications behave themselves exactly the same all the time. Oops... netscape decided to freak out and take up all your memory... guess its the admins fault. Oops... some mod_perl script decided to freak out and an apache process decides to suck all of your CPU and MEM. Crap like this does happen. An example of this is a webboard package called "Blackboard" consisting of various mod_perl scripts, apache, and mysql. It is an educational online conferencing system being used in conjunction with many college classes and thus is quite vital to the campus. Unfortunatly its buggy as hell and the memory sucking bug didn't pop up until we were a couple weeks into classes and locked into the system. A mod_perl script freaks out, the copy of apache goes nuts, and we get a bunch of lovely out of memory related messages to the console. Its times like these that an OOM killer like Rik's would be very useful. I feel Rik's OOM backported to 2.2.x would do wonders for situation. After playing with Rik's OOM system, I know it would do the right thing on this system but unfortunatly 2.4.x isn't trustworthy yet Yes, the software is buggy and should be fixed. Do I have the power to fix a broken commerical package that I'm locked into? No. The point of an OOM killer is if all hell breaks loose and you have a choice between a locked up system, a system thats slow as hell because its spending all its time swapping, or a system that kills the offender and gets back to buisness. I choose the third option. I can't think of any situation (either on desktop or server) where a system lockup or panic due to OOM would be acceptible w/ 2.4.x. On Thu, 12 Oct 2000, Matthew Hawkins wrote: Heh.. now all we need is some smart-arse to make something similar to apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy ;) Seriously, am I missing something obvious or is it far simpler just to keel over and die if the system goes OOM? I mean, seriously, if the administrator lets it get to that state then he/she/it deserves a dead system. It's akin to having your car run out of petrol - you don't start shooting passengers because their extra load made the engine chew more. You pack up your kitty and go to the nearest petrol station and buy more, plug it into the car then learn from the experience so this fringe case of it happening doesn't happen again. I don't really see much difference between a car going "OOP" and a computer going OOM. Should we start deleting files according to some randomly-chosen heueristic if a filesystem goes "OOS" ? -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/ -- Bruce A. Locke [EMAIL PROTECTED] "The Internet views censorship as damage and routes around it" www.eff.org www.peacefire.org - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
- Received message begins Here - Heh.. now all we need is some smart-arse to make something similar to apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy ;) Seriously, am I missing something obvious or is it far simpler just to keel over and die if the system goes OOM? I mean, seriously, if the administrator lets it get to that state then he/she/it deserves a dead system. It's akin to having your car run out of petrol - you don't start shooting passengers because their extra load made the engine chew more. You pack up your kitty and go to the nearest petrol station and buy more, plug it into the car then learn from the experience so this fringe case of it happening doesn't happen again. I don't really see much difference between a car going "OOP" and a computer going OOM. Should we start deleting files according to some randomly-chosen heueristic if a filesystem goes "OOS" ? Not deleting files, but your system may crash :) The problem with memory is that the tools are not available (ie already included in the kernel) to do anything else. In the example of running out of file space, there are quota limits. You can still run out of space, but only when the sum of all users quota allocations exceed the disk capacity. Until user memory resource quotas are included in the kernel, there will be nothing else that can be done. Even with resource quotas, if the total of active users exceeds the resource then the same/equivalent situation occurs. What is being done is still necessary, but in the long term it will end up addressing the case where a single user runs out, rather than the system as a whole. User memory resource quota control is needed in large clusters, and in large systems with multiple users. In a single user environment, resource quotas are less important than providing a consistant (and hopefully intutitive) process abort. That keeps the system going, and becomes up to the user to choose what else may need to be aborted. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On 2000-10-11 09:45:30 -0500, Jesse Pollard wrote:
Until user memory resource quotas are included in the kernel, there will be
nothing else that can be done. Even with resource quotas, if the total of
active users exceeds the resource then the same/equivalent situation occurs.
So setrlimit() with RLIMIT_DATA, RLIMIT_STACK, RLIMIT_RSS,
RLIMIT_MEMLOCK, RLIMIT_AS et al is a null op?
If so, I wish to register a complaint ;-)
--
* Matthew Hawkins [EMAIL PROTECTED] :(){ :|:};:
** Information Specialist, tSA Group Pty. Ltd. Ph: +61 2 6257 7111
*** 1 Hall Street, Lyneham ACT 2602 Australia. Fx: +61 2 6257 7311
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Wed, 11 Oct 2000, Bruce A. Locke wrote: Your making the deadly assumption that all applications behave themselves exactly the same all the time. Oops... netscape decided to freak out and take up all your memory... guess its the admins fault. Oops... some mod_perl script decided to freak out and an apache process decides to suck all of your CPU and MEM. that's why you have per process limits set. Eg, PAM makes this exceedingly easy with pam_limit.so - edit /etc/security/limit.conf. this prevents at least 90% of OOM situations (ie individual leaky processes). eg netscape will then pop-up "can not allocate memory" messages and stop rendering pages instead of crashing your system. --paulj - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On 2000-10-11 10:33:39 -0400, Bruce A. Locke wrote: Your making the deadly assumption that all applications behave themselves exactly the same all the time. Oops... netscape decided to freak out and take up all your memory... guess its the admins fault. Yep, for not setting appropriate resource limits. man 2 setrlimit Of course, if its a kernel bug that causes it I think you're SOL ;) -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Thu, 12 Oct 2000, Matthew Hawkins wrote:
Seriously, am I missing something obvious or is it far simpler just to
keel over and die if the system goes OOM? I mean, seriously, if the
administrator lets it get to that state then he/she/it deserves a dead
system. It's akin to having your car run out of petrol - you don't
start shooting passengers because their extra load made the engine chew
more. You pack up your kitty and go to the nearest petrol station and
buy more, plug it into the car then learn from the experience so this
fringe case of it happening doesn't happen again. I don't really see
much difference between a car going "OOP" and a computer going OOM.
Should we start deleting files according to some randomly-chosen
heueristic if a filesystem goes "OOS" ?
Excellent point. However, the idea is to kill an attacker if your 'car'
is being hijacked.
Whatever is being designed should ideally have zero impact on the usual
performance and only come into play if something runs away, deliberately
or by accident.
If Linux doesn't track down and kill deliberate attempts to kill the
system, there will always be those who say; "Linux is no good because
a user can readily kill it". Of course we could track down and
kill those who say this, but it'd get messy.
FYI, a fork() bomb on my Sun Workstation does not kill it. Also
malloc()ing and writing all over the place doesn't kill it either.
Script started on Wed Oct 11 10:41:38 2000
# cat xxx.c
main()
{
for(;;)
fork();
}
# gcc -o xxx xxx.c
# ./xxx
^C
# # ^C
# ps
PID TTY TIME CMD
24800 pts/10:00 xxx
24335 pts/10:00 sh
24688 pts/10:00 xxx
24690 pts/10:00 xxx
24692 pts/10:00 xxx
24694 pts/10:00 xxx
24696 pts/10:00 xxx
24697 pts/10:00 xxx
24699 pts/10:00 xxx
24701 pts/10:00 xxx
24703 pts/10:00 xxx
24704 pts/10:00 xxx
24706 pts/10:00 xxx
24708 pts/10:00 xxx
24710 pts/10:00 xxx
24712 pts/10:00 xxx
24714 pts/10:00 xxx
24716 pts/10:00 xxx
24717 pts/10:00 xxx
24719 pts/10:00 xxx
24720 pts/10:00 xxx
24721 pts/10:00 xxx
24722 pts/10:00 xxx
24723 pts/10:00 xxx
24724 pts/10:00 xxx
24725 pts/10:00 xxx
24726 pts/10:00 xxx
24727 pts/10:00 xxx
24728 pts/10:00 xxx
24729 pts/10:00 xxx
24730 pts/10:00 xxx
24731 pts/10:00 xxx
24732 pts/10:00 xxx
24733 pts/10:00 xxx
24734 pts/10:00 xxx
24735 pts/10:00 xxx
24736 pts/10:00 xxx
24737 pts/10:00 xxx
24738 pts/10:00 xxx
24739 pts/10:00 xxx
24740 pts/10:00 xxx
24741 pts/10:00 xxx
24742 pts/10:00 xxx
24743 pts/10:00 xxx
24744 pts/10:00 xxx
24801 pts/10:00 ps
24687 pts/10:00 xxx
24689 pts/10:00 xxx
24691 pts/10:00 xxx
24693 pts/10:00 xxx
24695 pts/10:00 xxx
24698 pts/10:00 xxx
24700 pts/10:00 xxx
24702 pts/10:00 xxx
24705 pts/10:00 xxx
24707 pts/10:00 xxx
24709 pts/10:00 xxx
24711 pts/10:00 xxx
24713 pts/10:00 xxx
24715 pts/10:00 xxx
24718 pts/10:00 xxx
24653 pts/10:00 xxx
24610 pts/10:00 xxx
24614 pts/10:00 xxx
24615 pts/10:00 xxx
24616 pts/10:00 xxx
24617 pts/10:00 xxx
24618 pts/10:00 xxx
24619 pts/10:00 xxx
24620 pts/10:00 xxx
24621 pts/10:00 xxx
24622 pts/10:00 xxx
24623 pts/10:00 xxx
24624 pts/10:00 xxx
24625 pts/10:00 xxx
24626 pts/10:00 xxx
24627 pts/10:00 xxx
24628 pts/10:00 xxx
24629 pts/10:00 xxx
24630 pts/10:00 xxx
24631 pts/10:00 xxx
24632 pts/10:00 xxx
24686 pts/10:00 xxx
24685 pts/10:00 xxx
24684 pts/10:00 xxx
24683 pts/10:00 xxx
24682 pts/10:00 xxx
24681 pts/10:00 xxx
24680 pts/10:00 xxx
24679 pts/10:00 xxx
24678 pts/10:00 xxx
24677 pts/10:00 xxx
24676 pts/10:00 xxx
24675 pts/10:00 xxx
24674 pts/10:00 xxx
24673 pts/10:00 xxx
24672 pts/10:00 xxx
24671 pts/10:00 xxx
24670 pts/10:00 xxx
24669 pts/10:00 xxx
24668 pts/10:00 xxx
24667 pts/10:00 xxx
24666 pts/10:00 xxx
24665 pts/10:00 xxx
24664 pts/10:00 xxx
24663 pts/10:00 xxx
24662 pts/10:00 xxx
24661 pts/10:00 xxx
24660 pts/10:00 xxx
24659 pts/10:00 xxx
24658 pts/10:00 xxx
24657 pts/10:00 xxx
24656 pts/10:00 xxx
24655 pts/10:00 xxx
24654 pts/10:00 xxx
24652 pts/10:00 xxx
24651 pts/10:00 xxx
24650 pts/10:00 xxx
24649 pts/10:00 xxx
24648 pts/10:00 xxx
24647 pts/10:00 xxx
24646 pts/10:00 xxx
24645 pts/10:00 xxx
24644 pts/10:00 xxx
24643 pts/10:00 xxx
24642 pts/10:00 xxx
24634 pts/10:00 xxx
24633 pts/10:00 xxx
24641 pts/10:00 xxx
24640 pts/10:00 xxx
24639 pts/10:00 xxx
24638 pts/10:00 xxx
24637 pts/10:00 xxx
24636 pts/10:00 xxx
24635 pts/10:00 xxx
24613 pts/10:00 xxx
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
- Received message begins Here - On 2000-10-11 09:45:30 -0500, Jesse Pollard wrote: Until user memory resource quotas are included in the kernel, there will be nothing else that can be done. Even with resource quotas, if the total of active users exceeds the resource then the same/equivalent situation occurs. So setrlimit() with RLIMIT_DATA, RLIMIT_STACK, RLIMIT_RSS, RLIMIT_MEMLOCK, RLIMIT_AS et al is a null op? If so, I wish to register a complaint ;-) Not exactly. As I have seen it, each process gets a copy of these limits. A single process cannot exceed the limit, but the sum of all processes can. One of the problems is cause by COW: given trivially small limits (1 MB) The first process allocates and initializes up to one MB, then forks. The second process begins updating data - .5MB. Neither process exceeds the limits, but the sum is now 1.5MB. If this is repeated enough, then the system can go OOM, with none of the processes at or over the limits set. Another problem occurs on multi-user servers. Each user logs in and gets "reasonable" rlimit values - each user uses one medium sized process. If the #users * rlimits exceeds the system capacity then OOM could occur, and still none may have exceeded the rlimit. I've always treated rlimit values as "suggestions" to the user process to aid in debugging. (this is more applicable to the ulimits though). The users process will not exeed the value, and when they do it is a strong suggestion that a bug may be present. (I first saw this with a leakey X server.) There have been some patches (the beancounter stuff) that does relate to resource control, but a more integrated resource accounting will make it work better. I do believe it should be available as an option, especially for multi-user servers, clusters, and other large systems. It isn't that usefull on single user workstations. - Jesse I Pollard, II Email: [EMAIL PROTECTED] Any opinions expressed are solely my own. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Thu, 12 Oct 2000, Matthew Hawkins wrote: Yep, for not setting appropriate resource limits. man 2 setrlimit Of course, if its a kernel bug that causes it I think you're SOL ;) This manpage shows me functions and structs. I'm assuming you want these used by the offending program or the shell under which the program is being called. In the first case, a person might not have source to the program and if thats the case, it doesn't help much. And in the second case, if the shell sets it, does it affect children of a process (aka fork()'d)? Thanks for yout time... -- Matt -- Bruce A. Locke [EMAIL PROTECTED] "The Internet views censorship as damage and routes around it" www.eff.org www.peacefire.org - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Wed, 11 Oct 2000, Paul Jakma wrote: that's why you have per process limits set. Eg, PAM makes this exceedingly easy with pam_limit.so - edit /etc/security/limit.conf. this prevents at least 90% of OOM situations (ie individual leaky processes). eg netscape will then pop-up "can not allocate memory" messages and stop rendering pages instead of crashing your system. I wasn't aware PAM settings affected daemons started up during boottime but I will check into it, thank you. BTW, you said it works only 90%, what are the other 10% of times it doesn't work? --paulj -- Bruce A. Locke [EMAIL PROTECTED] "The Internet views censorship as damage and routes around it" www.eff.org www.peacefire.org - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Wed, 11 Oct 2000, Bruce A. Locke wrote: I wasn't aware PAM settings affected daemons started up during boottime but I will check into it, thank you. daemons generally don't need to be PAM aware (unless they deal with authorising things). The script that launches it however (if started by a PAM aware app such as su) can set limits - which the daemon should inherit. BTW, you said it works only 90%, what are the other 10% of times it doesn't work? malicious processes, or a collection of processes. --paulj - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Thu, Oct 12, 2000 at 01:58:49AM +1100, Matthew Hawkins wrote: On 2000-10-11 10:33:39 -0400, Bruce A. Locke wrote: Your making the deadly assumption that all applications behave themselves exactly the same all the time. Oops... netscape decided to freak out and take up all your memory... guess its the admins fault. Yep, for not setting appropriate resource limits. No way should a desktop user be responsible for micro-managing the resource usage of his applications. How can he decide what's reasonable for Netscape to consume? Shouldn't Netscape be allowed to take up most of memory, if it's the only major application and the memory will improve its performance? The only thing that knows what's right for Netscape is Netscape. If Netscape were clever and kind, perhaps it would estimate what's reasonable and set limits on itself, adjusting them from time to time based on user behavior and environmental factors. But Netscape's a pretty mature program, and it doesn't do this; it can hardly be expected of the zillions of immature (and probably leaky) applications a user might run. So, we inevitably need an automated low-memory or out-of-memory algorithm. I tend to think it may need to be more adjustable than Rik's--people will be much more comfortable if they can say "spare this simulation at all cost!" or "kill off one of these processes in an emergency" or "this system has no business coming within 90% of RAM+swap capacity, so start killing things at that point--oh, and mail me". Some of this has no place in the kernel, obviously. But Rik has a good start, and perhaps his work will be part of a more complete solution. Andrew - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Thu, 12 Oct 2000, Matthew Hawkins wrote: On 2000-10-11 09:45:30 -0500, Jesse Pollard wrote: Until user memory resource quotas are included in the kernel, there will be nothing else that can be done. Even with resource quotas, if the total of active users exceeds the resource then the same/equivalent situation occurs. So setrlimit() with RLIMIT_DATA, RLIMIT_STACK, RLIMIT_RSS, RLIMIT_MEMLOCK, RLIMIT_AS et al is a null op? If so, I wish to register a complaint ;-) Don't send a complaint, send patches ... regards, Rik -- "What you're running that piece of shit Gnome?!?!" -- Miguel de Icaza, UKUUG 2000 http://www.conectiva.com/ http://www.surriel.com/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On 2000-10-11 12:48:54 -0400, Andrew Pimlott wrote: No way should a desktop user be responsible for micro-managing the resource usage of his applications. That's right. The systems administrator should, and will set appropriate limits for users on his/her system that apply from login. This is how the systems I first used were configured (lucky me had a damn fine sysadmin), and so this is how I configure mine. The only thing that knows what's right for Netscape is Netscape. I would disagree with this, I believe this is exactly the root of people's problems with Netscape (and the same theory should apply to other apps). The application doesn't know what's _right_ - it knows what it _wants_. Big difference. -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On 2000-10-11 11:45:06 -0400, Bruce A. Locke wrote: This manpage shows me functions and structs. What were you expecting from the system call section of the Linux Programmer's Manual? Dancing girls? (h...) I'm assuming you want these used by the offending program or the shell under which the program is being called. That's usually what happens. In the first case, a person might not have source to the program and if thats the case, it doesn't help much. Closed-source software is *so* 20th century... ;-) Anyway, when run from the shell it'll inherit its parent's limits (which leads to your next question...) And in the second case, if the shell sets it, does it affect children of a process (aka fork()'d)? Certainly. Maybe if more distributions took Debian's stance and set the default limits so anal that you frequently can't even read email let alone recompile the kernel without getting the process terminated for tripping one limit or another, then more people would know this functionality exists and set the limits more appropriately. -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
I've had to support an app running as a back-end to a webserver that would malloc() different amounts of memory depending on user input, up to multiple gigabytes of memory which vastly exceeded the 512k the machine had as main memory. The app was a program that would scan genetic sequence looking for 'repeats' in the sequence, and one sequence would malloc a hundred megs while a similar sequence of the same size would cause the algorithm to try to malloc over a gig. Part of the algorithm was actually to simply try to malloc all the memory it could and if it ran out, it would bump down the resolution that it was scanning with and try again. And it would regularly push the machine into OOM and take it down because daemons got killed long before the OOM killer got around to taking out the process that was malloc()ing all the memory. On other machines I'd set RLIMIT_DATA and my OOM problems went away, but on linux this didn't work (and i wasn't comfortable enough with kernel sources back then to manage to find RLIMIT_AS). On Thu, 12 Oct 2000, Matthew Hawkins wrote: Heh.. now all we need is some smart-arse to make something similar to apply to the _entire_ VM subsystem, and both Rik and Andrea can be happy ;) Seriously, am I missing something obvious or is it far simpler just to keel over and die if the system goes OOM? I mean, seriously, if the administrator lets it get to that state then he/she/it deserves a dead system. It's akin to having your car run out of petrol - you don't start shooting passengers because their extra load made the engine chew more. You pack up your kitty and go to the nearest petrol station and buy more, plug it into the car then learn from the experience so this fringe case of it happening doesn't happen again. I don't really see much difference between a car going "OOP" and a computer going OOM. Should we start deleting files according to some randomly-chosen heueristic if a filesystem goes "OOS" ? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On 2000-10-11 19:53:50 -0700, [EMAIL PROTECTED] wrote: On other machines I'd set RLIMIT_DATA and my OOM problems went away, but on linux this didn't work RLIMIT_DATA appears to only be checked for aout format executables. Looking at the 2.4.0-test10pre1 sources for fs/binfmt_aout.c and fs/binfmt_elf.c you'll note the difference in load_aout_binary() and load_elf_binary(), both just above the comment of "OK, This is the point of no return" Does putting a similar check to the aout one make sense for ELF? I'm just trying to avoid Rik having to pull his hair out implementing a system that conceptually already exists in the kernel (nasty processes being terminated before they do some damage). Especially when that existing system is far more configurable. Cheers, -- Matt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On Tue, Oct 10, 2000 at 05:58:46PM -0300, Rik van Riel wrote: > On Tue, 10 Oct 2000, Tom Rini wrote: > > On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote: > > > On Tue, 10 Oct 2000, Ingo Oeser wrote: > > > > > > > before you argue endlessly about the "Right OOM Killer (TM)", I > > > > did a small patch to allow replacing the OOM killer at runtime. > > > > > > > > So now you can stop arguing about the one and only OOM killer, > > > > implement it, provide it as module and get back to the important > > > > stuff ;-) > > > > > > This is definately a cool toy for people who have doubts > > > that my OOM killer will do the wrong thing in their > > > workloads. > > > > I think this can be useful for more than just a cool toy. I > > think that the main thing that this discusion has shown is no > > OOM killer will please 100% of the people 100% of the time. I > > think we should try and have a good generic OOM killer that > > kills the right process most of the time. People can impliment > > (and submit) different-style OOM killers as needed. > > Indeed, though I suspect most of the people trying this would > fall into the trap of over-engineering their OOM killer, after > which it mostly becomes less predictable ;) I was thinking more along the lines of ones w/ "safety" features that not everyone might like/need (ie /usr/local/bin/foo is always good, those sugjestions). It seems like useful functionality at little/no cost. And a neat toy for now. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote: > On Tue, 10 Oct 2000, Ingo Oeser wrote: > > > before you argue endlessly about the "Right OOM Killer (TM)", I > > did a small patch to allow replacing the OOM killer at runtime. > > > > So now you can stop arguing about the one and only OOM killer, > > implement it, provide it as module and get back to the important > > stuff ;-) > > This is definately a cool toy for people who have doubts > that my OOM killer will do the wrong thing in their > workloads. I think this can be useful for more than just a cool toy. I think that the main thing that this discusion has shown is no OOM killer will please 100% of the people 100% of the time. I think we should try and have a good generic OOM killer that kills the right process most of the time. People can impliment (and submit) different-style OOM killers as needed. Or at least get 'em on freshmeat. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote:
> > So now you can stop arguing about the one and only OOM killer,
> > implement it, provide it as module and get back to the important
> > stuff ;-)
>
> This is definately a cool toy for people who have doubts
> that my OOM killer will do the wrong thing in their
> workloads.
Thanks ;-)
But I forgot to include my changes to the mm/Makefile (to export
the API for modules).
Here is a _working_ one:
--- linux-2.4.0-test10-pre1/mm/oom_kill.c Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/oom_kill.c Tue Oct 10 16:59:27 2000
@@ -13,6 +13,8 @@
* machine) this file will double as a 'coding guide' and a signpost
* for newbie kernel hackers. It features several pointers to major
* kernel subsystems and hints as to where to find out what things do.
+ *
+ * Added oom_killer API for special needs - Ingo Oeser
*/
#include
@@ -136,7 +138,7 @@
}
/**
- * oom_kill - kill the "best" process when we run out of memory
+ * oom_kill_rik - kill the "best" process when we run out of memory
*
* If we run out of memory, we have the choice between either
* killing a random task (bad), letting the system crash (worse)
@@ -147,7 +149,9 @@
* CAP_SYS_RAW_IO set, send SIGTERM instead (but it's unlikely that
* we select a process with CAP_SYS_RAW_IO set).
*/
-void oom_kill(void)
+
+
+static void oom_kill_rik(void)
{
struct task_struct *p = select_bad_process();
@@ -207,4 +211,63 @@
/* Else... */
return 1;
+}
+
+/* Protects oom_killer against resetting during its execution */
+static rwlock_t oom_kill_lock = RW_LOCK_UNLOCKED;
+
+static oom_killer_t oom_killer = oom_kill_rik;
+
+/**
+ * oom_kill - the oom_kill wrapper for installable OOM killers
+ *
+ * Wraper around the OOM killers, that can be installed via
+ * install_oom_killer and reset_default_oom_killer.
+ *
+ * This gets called from kswapd() in linux/mm/vmscan.c when we
+ * really run out of memory.
+ */
+void oom_kill(void) {
+ read_lock(_kill_lock);
+ oom_killer();
+ read_unlock(_kill_lock);
+}
+
+/**
+ * install_oom_killer - install alternate OOM killer
+ * @new_oom_kill: the alternate OOM killer provided by the caller
+ *
+ * Since the default OOM killer (oom_kill_rik) is not suitable
+ * for everyone, we provide an interface to install custom OOM killers.
+ *
+ * You can take the most appropriate action for your application if the
+ * kernel goes OOM.
+ *
+ * Providing an NULL argument just returns the current OOM killer.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ *
+ * NOTE: We don't do refcounting on OOM killers, so be careful with
+ * modules
+ */
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill) {
+ oom_killer_t tmp;
+ write_lock(_kill_lock);
+ tmp=oom_killer;
+ if (new_oom_kill)
+ oom_killer=new_oom_kill;
+ write_unlock(_kill_lock);
+ return tmp;
+}
+
+/**
+ * reset_default_oom_killer - reset back to default OOM killer
+ *
+ * If you are going to unload the module which provided
+ * your OOM killer, you can install the default one by this.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ */
+oom_killer_t reset_default_oom_killer(void) {
+ return install_oom_killer(_kill_rik);
}
--- linux-2.4.0-test10-pre1/include/linux/swap.hTue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/include/linux/swap.hTue Oct 10 16:44:22 2000
@@ -127,8 +127,14 @@
#define read_swap_cache(entry) read_swap_cache_async(entry, 1);
/* linux/mm/oom_kill.c */
+typedef void (*oom_killer_t)(void);
+
extern int out_of_memory(void);
extern void oom_kill(void);
+
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill);
+oom_killer_t reset_default_oom_killer(void);
+
/*
* Make these inline later once they are working properly.
--- linux-2.4.0-test10-pre1/mm/Makefile Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/Makefile Tue Oct 10 16:34:06 2000
@@ -10,7 +10,8 @@
O_TARGET := mm.o
O_OBJS := memory.o mmap.o filemap.o mprotect.o mlock.o mremap.o \
vmalloc.o slab.o bootmem.o swap.o vmscan.o page_io.o \
- page_alloc.o swap_state.o swapfile.o numa.o oom_kill.o
+ page_alloc.o swap_state.o swapfile.o numa.o
+OX_OBJS := oom_kill.o
ifeq ($(CONFIG_HIGHMEM),y)
O_OBJS += highmem.o
Regards
Ingo Oeser
--
Feel the power of the penguin - run [EMAIL PROTECTED]
:x
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
[PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 & OOM handler)
[OOM killer war]
Hi there,
before you argue endlessly about the "Right OOM Killer (TM)", I
did a small patch to allow replacing the OOM killer at runtime.
You can even use modules, if you are careful (see khttpd on how
to do this without refcouting).
So now you can stop arguing about the one and only OOM killer,
implement it, provide it as module and get back to the important
stuff ;-)
PS: Patch is against test10-pre1.
Thanks for listening
Ingo Oeser
--- linux-2.4.0-test10-pre1/mm/oom_kill.c Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/oom_kill.c Tue Oct 10 16:59:27 2000
@@ -13,6 +13,8 @@
* machine) this file will double as a 'coding guide' and a signpost
* for newbie kernel hackers. It features several pointers to major
* kernel subsystems and hints as to where to find out what things do.
+ *
+ * Added oom_killer API for special needs - Ingo Oeser
*/
#include
@@ -136,7 +138,7 @@
}
/**
- * oom_kill - kill the "best" process when we run out of memory
+ * oom_kill_rik - kill the "best" process when we run out of memory
*
* If we run out of memory, we have the choice between either
* killing a random task (bad), letting the system crash (worse)
@@ -147,7 +149,9 @@
* CAP_SYS_RAW_IO set, send SIGTERM instead (but it's unlikely that
* we select a process with CAP_SYS_RAW_IO set).
*/
-void oom_kill(void)
+
+
+static void oom_kill_rik(void)
{
struct task_struct *p = select_bad_process();
@@ -207,4 +211,63 @@
/* Else... */
return 1;
+}
+
+/* Protects oom_killer against resetting during its execution */
+static rwlock_t oom_kill_lock = RW_LOCK_UNLOCKED;
+
+static oom_killer_t oom_killer = oom_kill_rik;
+
+/**
+ * oom_kill - the oom_kill wrapper for installable OOM killers
+ *
+ * Wraper around the OOM killers, that can be installed via
+ * install_oom_killer and reset_default_oom_killer.
+ *
+ * This gets called from kswapd() in linux/mm/vmscan.c when we
+ * really run out of memory.
+ */
+void oom_kill(void) {
+ read_lock(_kill_lock);
+ oom_killer();
+ read_unlock(_kill_lock);
+}
+
+/**
+ * install_oom_killer - install alternate OOM killer
+ * @new_oom_kill: the alternate OOM killer provided by the caller
+ *
+ * Since the default OOM killer (oom_kill_rik) is not suitable
+ * for everyone, we provide an interface to install custom OOM killers.
+ *
+ * You can take the most appropriate action for your application if the
+ * kernel goes OOM.
+ *
+ * Providing an NULL argument just returns the current OOM killer.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ *
+ * NOTE: We don't do refcounting on OOM killers, so be careful with
+ * modules
+ */
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill) {
+ oom_killer_t tmp;
+ write_lock(_kill_lock);
+ tmp=oom_killer;
+ if (new_oom_kill)
+ oom_killer=new_oom_kill;
+ write_unlock(_kill_lock);
+ return tmp;
+}
+
+/**
+ * reset_default_oom_killer - reset back to default OOM killer
+ *
+ * If you are going to unload the module which provided
+ * your OOM killer, you can install the default one by this.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ */
+oom_killer_t reset_default_oom_killer(void) {
+ return install_oom_killer(_kill_rik);
}
--- linux-2.4.0-test10-pre1/include/linux/swap.hTue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/include/linux/swap.hTue Oct 10 16:44:22 2000
@@ -127,8 +127,14 @@
#define read_swap_cache(entry) read_swap_cache_async(entry, 1);
/* linux/mm/oom_kill.c */
+typedef void (*oom_killer_t)(void);
+
extern int out_of_memory(void);
extern void oom_kill(void);
+
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill);
+oom_killer_t reset_default_oom_killer(void);
+
/*
* Make these inline later once they are working properly.
--
Feel the power of the penguin - run [EMAIL PROTECTED]
:x
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
[PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
[OOM killer war]
Hi there,
before you argue endlessly about the "Right OOM Killer (TM)", I
did a small patch to allow replacing the OOM killer at runtime.
You can even use modules, if you are careful (see khttpd on how
to do this without refcouting).
So now you can stop arguing about the one and only OOM killer,
implement it, provide it as module and get back to the important
stuff ;-)
PS: Patch is against test10-pre1.
Thanks for listening
Ingo Oeser
--- linux-2.4.0-test10-pre1/mm/oom_kill.c Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/oom_kill.c Tue Oct 10 16:59:27 2000
@@ -13,6 +13,8 @@
* machine) this file will double as a 'coding guide' and a signpost
* for newbie kernel hackers. It features several pointers to major
* kernel subsystems and hints as to where to find out what things do.
+ *
+ * Added oom_killer API for special needs - Ingo Oeser
*/
#include linux/mm.h
@@ -136,7 +138,7 @@
}
/**
- * oom_kill - kill the "best" process when we run out of memory
+ * oom_kill_rik - kill the "best" process when we run out of memory
*
* If we run out of memory, we have the choice between either
* killing a random task (bad), letting the system crash (worse)
@@ -147,7 +149,9 @@
* CAP_SYS_RAW_IO set, send SIGTERM instead (but it's unlikely that
* we select a process with CAP_SYS_RAW_IO set).
*/
-void oom_kill(void)
+
+
+static void oom_kill_rik(void)
{
struct task_struct *p = select_bad_process();
@@ -207,4 +211,63 @@
/* Else... */
return 1;
+}
+
+/* Protects oom_killer against resetting during its execution */
+static rwlock_t oom_kill_lock = RW_LOCK_UNLOCKED;
+
+static oom_killer_t oom_killer = oom_kill_rik;
+
+/**
+ * oom_kill - the oom_kill wrapper for installable OOM killers
+ *
+ * Wraper around the OOM killers, that can be installed via
+ * install_oom_killer and reset_default_oom_killer.
+ *
+ * This gets called from kswapd() in linux/mm/vmscan.c when we
+ * really run out of memory.
+ */
+void oom_kill(void) {
+ read_lock(oom_kill_lock);
+ oom_killer();
+ read_unlock(oom_kill_lock);
+}
+
+/**
+ * install_oom_killer - install alternate OOM killer
+ * @new_oom_kill: the alternate OOM killer provided by the caller
+ *
+ * Since the default OOM killer (oom_kill_rik) is not suitable
+ * for everyone, we provide an interface to install custom OOM killers.
+ *
+ * You can take the most appropriate action for your application if the
+ * kernel goes OOM.
+ *
+ * Providing an NULL argument just returns the current OOM killer.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ *
+ * NOTE: We don't do refcounting on OOM killers, so be careful with
+ * modules
+ */
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill) {
+ oom_killer_t tmp;
+ write_lock(oom_kill_lock);
+ tmp=oom_killer;
+ if (new_oom_kill)
+ oom_killer=new_oom_kill;
+ write_unlock(oom_kill_lock);
+ return tmp;
+}
+
+/**
+ * reset_default_oom_killer - reset back to default OOM killer
+ *
+ * If you are going to unload the module which provided
+ * your OOM killer, you can install the default one by this.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ */
+oom_killer_t reset_default_oom_killer(void) {
+ return install_oom_killer(oom_kill_rik);
}
--- linux-2.4.0-test10-pre1/include/linux/swap.hTue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/include/linux/swap.hTue Oct 10 16:44:22 2000
@@ -127,8 +127,14 @@
#define read_swap_cache(entry) read_swap_cache_async(entry, 1);
/* linux/mm/oom_kill.c */
+typedef void (*oom_killer_t)(void);
+
extern int out_of_memory(void);
extern void oom_kill(void);
+
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill);
+oom_killer_t reset_default_oom_killer(void);
+
/*
* Make these inline later once they are working properly.
--
Feel the power of the penguin - run [EMAIL PROTECTED]
esc:x
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Tue, 10 Oct 2000, Ingo Oeser wrote: before you argue endlessly about the "Right OOM Killer (TM)", I did a small patch to allow replacing the OOM killer at runtime. So now you can stop arguing about the one and only OOM killer, implement it, provide it as module and get back to the important stuff ;-) This is definately a cool toy for people who have doubts that my OOM killer will do the wrong thing in their workloads. If anyone can demonstrate that the current OOM killer is doing the wrong thing and has a replacement algorithm available, please let us know ... ;) [lets move the discussion back to a less theoretical and more practical point of view] regards, Rik -- "What you're running that piece of shit Gnome?!?!" -- Miguel de Icaza, UKUUG 2000 http://www.conectiva.com/ http://www.surriel.com/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote:
So now you can stop arguing about the one and only OOM killer,
implement it, provide it as module and get back to the important
stuff ;-)
This is definately a cool toy for people who have doubts
that my OOM killer will do the wrong thing in their
workloads.
Thanks ;-)
But I forgot to include my changes to the mm/Makefile (to export
the API for modules).
Here is a _working_ one:
--- linux-2.4.0-test10-pre1/mm/oom_kill.c Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/oom_kill.c Tue Oct 10 16:59:27 2000
@@ -13,6 +13,8 @@
* machine) this file will double as a 'coding guide' and a signpost
* for newbie kernel hackers. It features several pointers to major
* kernel subsystems and hints as to where to find out what things do.
+ *
+ * Added oom_killer API for special needs - Ingo Oeser
*/
#include linux/mm.h
@@ -136,7 +138,7 @@
}
/**
- * oom_kill - kill the "best" process when we run out of memory
+ * oom_kill_rik - kill the "best" process when we run out of memory
*
* If we run out of memory, we have the choice between either
* killing a random task (bad), letting the system crash (worse)
@@ -147,7 +149,9 @@
* CAP_SYS_RAW_IO set, send SIGTERM instead (but it's unlikely that
* we select a process with CAP_SYS_RAW_IO set).
*/
-void oom_kill(void)
+
+
+static void oom_kill_rik(void)
{
struct task_struct *p = select_bad_process();
@@ -207,4 +211,63 @@
/* Else... */
return 1;
+}
+
+/* Protects oom_killer against resetting during its execution */
+static rwlock_t oom_kill_lock = RW_LOCK_UNLOCKED;
+
+static oom_killer_t oom_killer = oom_kill_rik;
+
+/**
+ * oom_kill - the oom_kill wrapper for installable OOM killers
+ *
+ * Wraper around the OOM killers, that can be installed via
+ * install_oom_killer and reset_default_oom_killer.
+ *
+ * This gets called from kswapd() in linux/mm/vmscan.c when we
+ * really run out of memory.
+ */
+void oom_kill(void) {
+ read_lock(oom_kill_lock);
+ oom_killer();
+ read_unlock(oom_kill_lock);
+}
+
+/**
+ * install_oom_killer - install alternate OOM killer
+ * @new_oom_kill: the alternate OOM killer provided by the caller
+ *
+ * Since the default OOM killer (oom_kill_rik) is not suitable
+ * for everyone, we provide an interface to install custom OOM killers.
+ *
+ * You can take the most appropriate action for your application if the
+ * kernel goes OOM.
+ *
+ * Providing an NULL argument just returns the current OOM killer.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ *
+ * NOTE: We don't do refcounting on OOM killers, so be careful with
+ * modules
+ */
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill) {
+ oom_killer_t tmp;
+ write_lock(oom_kill_lock);
+ tmp=oom_killer;
+ if (new_oom_kill)
+ oom_killer=new_oom_kill;
+ write_unlock(oom_kill_lock);
+ return tmp;
+}
+
+/**
+ * reset_default_oom_killer - reset back to default OOM killer
+ *
+ * If you are going to unload the module which provided
+ * your OOM killer, you can install the default one by this.
+ *
+ * Returns: The OOM killer, which has been installed so far.
+ */
+oom_killer_t reset_default_oom_killer(void) {
+ return install_oom_killer(oom_kill_rik);
}
--- linux-2.4.0-test10-pre1/include/linux/swap.hTue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/include/linux/swap.hTue Oct 10 16:44:22 2000
@@ -127,8 +127,14 @@
#define read_swap_cache(entry) read_swap_cache_async(entry, 1);
/* linux/mm/oom_kill.c */
+typedef void (*oom_killer_t)(void);
+
extern int out_of_memory(void);
extern void oom_kill(void);
+
+oom_killer_t install_oom_killer(oom_killer_t new_oom_kill);
+oom_killer_t reset_default_oom_killer(void);
+
/*
* Make these inline later once they are working properly.
--- linux-2.4.0-test10-pre1/mm/Makefile Tue Oct 10 16:31:08 2000
+++ linux-2.4.0-test10-pre1-ioe/mm/Makefile Tue Oct 10 16:34:06 2000
@@ -10,7 +10,8 @@
O_TARGET := mm.o
O_OBJS := memory.o mmap.o filemap.o mprotect.o mlock.o mremap.o \
vmalloc.o slab.o bootmem.o swap.o vmscan.o page_io.o \
- page_alloc.o swap_state.o swapfile.o numa.o oom_kill.o
+ page_alloc.o swap_state.o swapfile.o numa.o
+OX_OBJS := oom_kill.o
ifeq ($(CONFIG_HIGHMEM),y)
O_OBJS += highmem.o
Regards
Ingo Oeser
--
Feel the power of the penguin - run [EMAIL PROTECTED]
esc:x
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote: On Tue, 10 Oct 2000, Ingo Oeser wrote: before you argue endlessly about the "Right OOM Killer (TM)", I did a small patch to allow replacing the OOM killer at runtime. So now you can stop arguing about the one and only OOM killer, implement it, provide it as module and get back to the important stuff ;-) This is definately a cool toy for people who have doubts that my OOM killer will do the wrong thing in their workloads. I think this can be useful for more than just a cool toy. I think that the main thing that this discusion has shown is no OOM killer will please 100% of the people 100% of the time. I think we should try and have a good generic OOM killer that kills the right process most of the time. People can impliment (and submit) different-style OOM killers as needed. Or at least get 'em on freshmeat. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Tue, 10 Oct 2000, Tom Rini wrote: On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote: On Tue, 10 Oct 2000, Ingo Oeser wrote: before you argue endlessly about the "Right OOM Killer (TM)", I did a small patch to allow replacing the OOM killer at runtime. So now you can stop arguing about the one and only OOM killer, implement it, provide it as module and get back to the important stuff ;-) This is definately a cool toy for people who have doubts that my OOM killer will do the wrong thing in their workloads. I think this can be useful for more than just a cool toy. I think that the main thing that this discusion has shown is no OOM killer will please 100% of the people 100% of the time. I think we should try and have a good generic OOM killer that kills the right process most of the time. People can impliment (and submit) different-style OOM killers as needed. Indeed, though I suspect most of the people trying this would fall into the trap of over-engineering their OOM killer, after which it mostly becomes less predictable ;) regards, Rik -- "What you're running that piece of shit Gnome?!?!" -- Miguel de Icaza, UKUUG 2000 http://www.conectiva.com/ http://www.surriel.com/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] OOM killer API (was: [PATCH] VM fix for 2.4.0-test9 OOM handler)
On Tue, Oct 10, 2000 at 05:58:46PM -0300, Rik van Riel wrote: On Tue, 10 Oct 2000, Tom Rini wrote: On Tue, Oct 10, 2000 at 12:32:50PM -0300, Rik van Riel wrote: On Tue, 10 Oct 2000, Ingo Oeser wrote: before you argue endlessly about the "Right OOM Killer (TM)", I did a small patch to allow replacing the OOM killer at runtime. So now you can stop arguing about the one and only OOM killer, implement it, provide it as module and get back to the important stuff ;-) This is definately a cool toy for people who have doubts that my OOM killer will do the wrong thing in their workloads. I think this can be useful for more than just a cool toy. I think that the main thing that this discusion has shown is no OOM killer will please 100% of the people 100% of the time. I think we should try and have a good generic OOM killer that kills the right process most of the time. People can impliment (and submit) different-style OOM killers as needed. Indeed, though I suspect most of the people trying this would fall into the trap of over-engineering their OOM killer, after which it mostly becomes less predictable ;) I was thinking more along the lines of ones w/ "safety" features that not everyone might like/need (ie /usr/local/bin/foo is always good, those sugjestions). It seems like useful functionality at little/no cost. And a neat toy for now. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
