Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005 at 06:40:50PM -0400, David Schultz wrote: Using madvise doesn't require changes per app, since MADV_PROTECT is inherited across exec. You just have to write a wrapper, much in the spirit of nice(1), to execute a protected version of X. Hmm, that's new to me, but certainly very useful and would do the trick in the case of X. The only problem seems to be that it doesn't seem to be shareable across fork() and that is isn't easily tunable after a process is started. Btw.: Why is this in madvise() and not in rfork() ? Is it because of POSIX ? Any pointers are welcome... Marc pgpAcf5b8T5HU.pgp Description: PGP signature
re: kernel killing processes when out of swap
I've often wondered why swap doesn't have a root-only x% like the file system. Once the swap used reached 100% user processes would bomb but a root process could go ahead and use the secret x%. Another idea would be to have multiple swap files and implement a way to restrict one swap area to a specific GID (say wheel). While on the issue we could also reserve a range of process ID's for wheel group only. To always be able to log on and un-wedge a system would be most convenient. Later Mark Hittinger [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
kernel killing processes when out of swap
Just had a problem with a box where it looks like it ran out of swap due to a problem process, not a problem. The problem was that it seems the kernel on detecting this starts killing off seeming random processes, the first one being sshd hence making the machine inaccessible. So the question is: Does the kernel kill random processes when out of swap or does it kill any processes that require more memory when out of swap? Which leads to the question would it not be more sensible to kill off the largest process first as its more than likely that it is responsible for the problem? [quote] Apr 10 20:09:25 appledore kernel: pid 414 (sshd), uid 0, was killed: out of swap space [/quote] Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
[EMAIL PROTECTED] wrote: Just had a problem with a box where it looks like it ran out of swap due to a problem process, not a problem. The problem was that it seems the kernel on detecting this starts killing off seeming random processes, the first one being sshd hence making the machine inaccessible. So the question is: Does the kernel kill random processes when out of swap or does it kill any processes that require more memory when out of swap? Which leads to the question would it not be more sensible to kill off the largest process first as its more than likely that it is responsible for the problem? This procedure is not random, it indeed looks for the largest process and then kills it. It keeps repeating this procedure until the memory starvation problem is solved. You obviously are not running X on that machine, otherwise you would see that X would get killed before sshd. When you're out of swap, you're also out of luck if sshd is among the largest processes on your machine. Having a flag to tag processes as vital to prevent them from getting killed (or to give them lower next-to-be-killed priority so that all non-vital processes get killed first) when you run out of swap would be a useful feature, what do you guys think? ALeine ___ WebMail FREE http://mail.austrosearch.net ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005 at 05:24:15AM -0700, ALeine wrote: machine. Having a flag to tag processes as vital to prevent them from getting killed (or to give them lower next-to-be-killed priority so that all non-vital processes get killed first) when you run out of swap would be a useful feature, P_SYSTEM provides something like this feature, amongst others. It is used by init(8). However P_SYSTEM also says that process statistics should not be maintained, and that the signals that may be dispatched to such processes are restricted. It would be more appropriate if such processes use the mlockall(2) call for the time being. BMS ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
Thanks for the feedback seems very strange that sshd was the first thing the kernel killed off; so unless it was actually at fault ( would be very strange ) it would have been one of the smallest not largest processes. The box has runs several 200M+ process and more 100M+ where as sshd is usually 6M. So this leads me to the questions: 1. Any know issues ssh which could make it eat memory? 2. Is there possibly a bug with the large process detection? N.B. It seems more likely that #2 is case as the next processes to be killed where some small perl monitoring scripts we run only after that did it kill of one of the large 200M+ processes. Regards Steve - Original Message - From: ALeine [EMAIL PROTECTED] This procedure is not random, it indeed looks for the largest process and then kills it. It keeps repeating this procedure until the memory starvation problem is solved. You obviously are not running X on that machine, otherwise you would see that X would get killed before sshd. When you're out of swap, you're also out of luck if sshd is among the largest processes on your machine. Having a flag to tag processes as vital to prevent them from getting killed (or to give them lower next-to-be-killed priority so that all non-vital processes get killed first) when you run out of swap would be a useful feature, what do you guys think? This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005, Bruce M Simpson wrote: On Tue, Apr 12, 2005 at 05:24:15AM -0700, ALeine wrote: machine. Having a flag to tag processes as vital to prevent them from getting killed (or to give them lower next-to-be-killed priority so that all non-vital processes get killed first) when you run out of swap would be a useful feature, P_SYSTEM provides something like this feature, amongst others. It is used by init(8). However P_SYSTEM also says that process statistics should not be maintained, and that the signals that may be dispatched to such processes are restricted. A madvise(NULL, 0, MADV_PROTECT) is the best way to tell the kernel not to kill your process. You must be running as root to use it. Now that disk space is cheap, it would be even nicer if we had a non-overcommit mode for swap, as most System V-derived systems do. Then at least processes would get ENOMEM instead of SIGKILL in a space shortage. I don't think this would be too hard to add, except possibly for handling OBJT_DEFAULT objects. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
[patch] GBDE memory allocations bugs and mlockall(2) [Was Re: kernel killing processes when out of swap]
[EMAIL PROTECTED] wrote:
It would be more appropriate if such processes use the mlockall(2) call
for the time being.
Speaking of mlockall(2), almost two weeks ago when I posted here about
CGD (and GBDE) memory allocation bugs I sent the attached patch for
src/sbin/gbde/gbde.c to PHK, but since he never got back to me on that
I would like to ask you guys to review the patch and commit it as
appropriate, hopefully before the next release.
The patch addresses memory allocation bugs where memory is used without
checking first if allocation was successful. In one case this could lead
to a segment violation with the core dump containing the sensitive info
required to decrypt the GBDE volume. The patch also disables core dumping
by using setrlimit(2)) and prevents paging by using mlockall(2).
I have patched several similar memory allocation bugs in libgeom, so if
anyone would like to see these bugs fixed let me know and I'll post that
patch as well. Right now I get the feeling that due to memory overcommit
in FreeBSD some people might think that these checks are not absolutely
necessary since another larger process would be more likely to get killed
before malloc(3) would return NULL, but I believe that one should not rely
on that.
ALeine
___
WebMail FREE http://mail.austrosearch.net --- src/sbin/gbde/gbde.c.oldWed Mar 30 14:18:37 2005
+++ src/sbin/gbde/gbde.cWed Mar 30 15:11:15 2005
@@ -85,7 +85,10 @@
#include sys/stat.h
#include crypto/rijndael/rijndael.h
#include crypto/sha2/sha2.h
+#include sys/time.h
+#include sys/resource.h
#include sys/param.h
+#include sys/mman.h
#include sys/linker.h
#define GBDEMOD geom_bde
@@ -145,6 +148,19 @@
exit (1);
}
+static void
+disable_core_dumping(void)
+{
+ struct rlimit rl;
+ int error;
+
+ rl.rlim_cur = 0;
+ rl.rlim_max = 0;
+ error = setrlimit(RLIMIT_CORE, rl);
+ if (error)
+ err(1, setrlimit() failed to set core size to 0);
+}
+
void *
g_read_data(struct g_consumer *cp, off_t offset, off_t length, int *error)
{
@@ -355,7 +371,10 @@
u_char *sbuf;
off_t offset, offset2;
+ /* Allocate a sectorbuffer immediately and zero it */
sbuf = malloc(gl-sectorsize);
+ if (sbuf == NULL)
+ err(1, malloc);
memset(sbuf, 0, gl-sectorsize);
offset = (gl-lsector[key] ~(gl-sectorsize - 1));
offset2 = lseek(dfd, offset, SEEK_SET);
@@ -377,7 +396,10 @@
u_char *sbuf, *q;
off_t offset, offset2;
+ /* Allocate a sectorbuffer */
sbuf = malloc(gl-sectorsize);
+ if (sbuf == NULL)
+ err(1, malloc);
/*
* Find the byte-offset in the lock sector where we will put the lock
* data structure. We can put it any random place as long as the
@@ -422,9 +444,7 @@
errx(1, No -L option and no space in sector 0 for lockfile);
}
- /* Allocate a sectorbuffer and fill it with random junk */
- if (sbuf == NULL)
- err(1, malloc);
+ /* Fill sectorbuffer with random junk */
random_bits(sbuf, gl-sectorsize);
/* Fill random bits in the spare field */
@@ -722,12 +742,17 @@
if (argc 3)
usage(Too few arguments\n);
- if ((i = modfind(g_bde)) 0) {
- /* need to load the gbde module */
- if (kldload(GBDEMOD) 0 || modfind(g_bde) 0) {
- usage(GBDEMOD : Kernel module not available\n);
- }
- }
+ if ((i = modfind(g_bde)) 0) {
+ /* need to load the gbde module */
+ if (kldload(GBDEMOD) 0 || modfind(g_bde) 0) {
+ usage(GBDEMOD : Kernel module not available\n);
+ }
+ }
+
+ disable_core_dumping();
+ if (mlockall(MCL_FUTURE))
+ err(1, mlockall() failed to lock all memory);
+
doopen = 0;
if (!strcmp(argv[1], attach)) {
action = ACT_ATTACH;
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
Steven Hartland [EMAIL PROTECTED] writes: Thanks for the feedback seems very strange that sshd was the first thing the kernel killed off; so unless it was actually at fault ( would be very strange ) it would have been one of the smallest not largest processes. The box has runs several 200M+ process and more 100M+ where as sshd is usually 6M. So this leads me to the questions: 1. Any know issues ssh which could make it eat memory? 2. Is there possibly a bug with the large process detection? There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005 at 06:46:45PM +0200, Dag-Erling Smrgrav wrote: There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. Which sucks when a process like X tries to free and realloc things when possible and tries to be system friendly, but thus increases the chance to get shot down, while programs over-allocing memory and never freeing it get to survive. It's a sad world. :-P Anyway, when at our office we were running X on low-memory systems and had to reboot often because of X being killed, rendering the text console useless, we had a patch to prevent processes with specific names being killed. I could revive this and turn this into a sysctl if anyone's interested... Marc pgpXZ8bngu8xb.pgp Description: PGP signature
Re: kernel killing processes when out of swap
- Original Message - There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. That makes sense. Me trying to connect to see what was going on would hence cause sshd to be killed ( not good ). It would seem that we could do with doing something to prevent this. We have had a few suggestions so far. 1. Using madvise, requires code changes per app 2. Using a sysctl to name processes which shouldn't be killed. 3. Changing how the choice of process to kill is made. I don't like the idea of having to make code changes so would vote against #1. #3 I would imagine could be quite complex and error fraught so #2 seems like the best option so far. What do others think? Go with Marc's idea? Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005 at 06:46:45PM +0200, Dag-Erling Smrgrav wrote: There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. Are you suggesting that vm_pageout_pmap_collect() is no longer in use ? That has a FOREACH_PROC_IN_SYSTEM(p) ... find biggest unlocked ... kill-kill-kill... Marc pgpyexatlhIPU.pgp Description: PGP signature
Re: kernel killing processes when out of swap
At 6:46 PM +0200 4/12/05, Dag-Erling Smørgrav wrote: Steven Hartland [EMAIL PROTECTED] writes: Thanks for the feedback seems very strange that sshd was the first thing the kernel killed off; so unless it was actually at fault ( would be very strange ) it would have been one of the smallest not largest processes. The box has runs several 200M+ process and more 100M+ where as sshd is usually 6M. So this leads me to the questions: 1. Any know issues ssh which could make it eat memory? 2. Is there possibly a bug with the large process detection? There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. From time-to-time, we talk about implementing some form of SIGDANGER, similar to what AIX has. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
[EMAIL PROTECTED] wrote: On Tue, Apr 12, 2005 at 06:46:45PM +0200, Dag-Erling Smrgrav wrote: There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. Are you suggesting that vm_pageout_pmap_collect() is no longer in use ? That has a FOREACH_PROC_IN_SYSTEM(p) ... find biggest unlocked ... kill-kill-kill... The largest process gets killed in vm_pageout_scan() according to the latest sources: http://fxr.watson.org/fxr/source/vm/vm_pageout.c#L1168 1168 /* 1169 * If we are critically low on one of RAM or swap and low on 1170 * the other, kill the largest process. However, we avoid 1171 * doing this on the first pass in order to give ourselves a 1172 * chance to flush out dirty vnode-backed pages and to allow 1173 * active pages to be moved to the inactive queue and reclaimed. 1174 * 1175 * We keep the process bigproc locked once we find it to keep anyone 1176 * from messing with it; however, there is a possibility of 1177 * deadlock if process B is bigproc and one of it's child processes 1178 * attempts to propagate a signal to B while we are waiting for A's 1179 * lock while walking this list. To avoid this, we don't block on 1180 * the process lock but just skip a process if it is already locked. 1181 */ http://fxr.watson.org/fxr/source/vm/vm_pageout.c?v=RELENG4#L1112 1112 /* 1113 * If we are out of swap and were not able to reach our paging 1114 * target, kill the largest process. 1115 */ ALeine ___ WebMail FREE http://mail.austrosearch.net ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005 at 11:37:52AM -0700, ALeine wrote: Are you suggesting that vm_pageout_pmap_collect() is no longer in use ? That has a FOREACH_PROC_IN_SYSTEM(p) ... find biggest unlocked ... kill-kill-kill... The largest process gets killed in vm_pageout_scan() according to the latest sources: Erhm, my mistake I meant the same code, just copy-pasted the function name above vm_pageout_scan() by accident. :-/ My point was/is: it still seems to do big process detection / killing. The original patch preventing X to be killed was built into that same loop... Marc pgpidSVjlA2y6.pgp Description: PGP signature
Re: kernel killing processes when out of swap
On Tue, 2005-Apr-12 05:24:15 -0700, ALeine wrote: Having a flag to tag processes as vital to prevent them from getting killed (or to give them lower next-to-be-killed priority so that all non-vital processes get killed first) when you run out of swap would be a useful feature, what do you guys think? This has been discussed to death before - look in the archives for 'SIGDANGER' (probably pre-mailman). -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
[EMAIL PROTECTED] wrote: That makes sense. Me trying to connect to see what was going on would hence cause sshd to be killed ( not good ). It would seem that we could do with doing something to prevent this. We have had a few suggestions so far. 1. Using madvise, requires code changes per app 2. Using a sysctl to name processes which shouldn't be killed. 3. Changing how the choice of process to kill is made. I don't like the idea of having to make code changes so would vote against #1. #3 I would imagine could be quite complex and error fraught so #2 seems like the best option so far. What do others think? Go with Marc's idea? I would go with #3 and add a new process flag like P_KILLLAST and then update vm_pageout_scan() to honour that flag and make sure to kill processes without that flag first before moving to processes which have that flag set. It would require minimum changes and you would only need a way to set this flag on a process. No change in existing applications would be required because they would be oblivious to this feature. IMHO this would be preferable to having something like SIGDANGER where every program has to have a special signal handler in order to benefit from this feature. ALeine ___ WebMail FREE http://mail.austrosearch.net ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kernel killing processes when out of swap
On Tue, Apr 12, 2005, Steven Hartland wrote: - Original Message - There is no large process detection. The first process that tries to fault in a new page after the system runs out of swap gets killed. That makes sense. Me trying to connect to see what was going on would hence cause sshd to be killed ( not good ). It would seem that we could do with doing something to prevent this. We have had a few suggestions so far. 1. Using madvise, requires code changes per app Using madvise doesn't require changes per app, since MADV_PROTECT is inherited across exec. You just have to write a wrapper, much in the spirit of nice(1), to execute a protected version of X. I agree that it's possible to do better, but this has already been discussed several times before, and whenever it comes up, there's always lots of noise because lots of people who don't understand deadlock chime in. If someone really wants to fix this in a reliable way, they need to go off and implement the necessary accounting in the VM system so that we don't overcommit swap. Most of the other suggestions, such as SIGDANGER, are just heuristics that might save your butt if you're lucky. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
