Re: Speeding up exit(2)?
Wojciech Puchar(woj...@wojtek.tensor.gdynia.pl)@2009.03.15 19:14:57 +0100: would be really nice about advocacy of GOOD web browser that not only exits faster, but WORKS faster. i don't know any, except links but links doesn't have CSS support I know one: Google Chrome - but it's only for Windows. Not only was it designed to be faster, but also more secure. It uses multi-process model for fault isolation and less memory fragmentation, the one so familiar to *nix. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
I know one: Google Chrome - but it's only for Windows. Not only was it designed to be faster, but also more secure. You mean more securely getting all data about user for google. NO THANKS! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
No, that's not what I meant. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Speeding up exit(2)?
I've noticed that when a huge, partially or totally swapped out process exits, there is a lot of disk activity going on, before the process truly dies. This is not necessarily due to sync(2), because it also happens with CPU bound processes that write very little output. Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). Couldn't this be avoided and the paged out pages simply discarded without reading them back in? Or do those pages contain necessary data at this point (page directories etc.)? -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). is it your program and you are sure it's on exit? i'm sure it's not. it's because the program is writted the way it's doing a lot of things (probably unneeded) on exit. not exit(2) itself ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sun, Mar 15, 2009 at 10:27:53AM +0100, Wojciech Puchar wrote: Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). is it your program and you are sure it's on exit? Every memory hungry program is concerned; and yes: it happens exactly on exit. it's because the program is writted the way it's doing a lot of things (probably unneeded) on exit. Have a look at what happens during exit: /usr/src/sys/kern/kern_exit.c:exit1() especially at the call to vm_waitproc(): /usr/src/sys/vm/vm_glue.c:vm_waitproc which calls vmspace_exitfree(): /usr/src/sys/vm/vm_map.c:vmspace_exitfree() Now, vmspace_exit() and vmspace_exitfree() ultimately call: /usr/src/sys/vm/vm_map.c:vmspace_dofree() It then goes deep into the bowels of vm amd pmap, and that's the place where the pages are paged in again (I think). not exit(2) itself -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
is it your program and you are sure it's on exit?
Every memory hungry program is concerned; and yes: it happens exactly
on exit.
strange.
i just wrote a test program
#include stdio.h
int test[1024*1024*128];
main() {
int a;
for(a=0;a1024*1024*128;a++) test[a]=a;
puts(end);
}
it fills 512MB RAM and then ends. i have 256MB RAM in laptop
it swapped a lot, then wrote end and immediately exited.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sun, Mar 15, 2009 at 11:09:00AM +0100, Wojciech Puchar wrote:
is it your program and you are sure it's on exit?
Every memory hungry program is concerned; and yes: it happens exactly
on exit.
strange.
i just wrote a test program
#include stdio.h
int test[1024*1024*128];
main() {
int a;
for(a=0;a1024*1024*128;a++) test[a]=a;
puts(end);
}
it fills 512MB RAM and then ends. i have 256MB RAM in laptop
it swapped a lot, then wrote end and immediately exited.
Hmmm... yes, it's strange. With malloc-ed space, exit is also very
fast. On a 2 GB machine with amd64, exit is almost immediate:
- snip --
#include stdio.h
#include stdlib.h
#include unistd.h
#include strings.h
#define NRGIGS 4
#define BIGSIZE (1024*1024*1024)
#define SOMETIME 15
main() {
int a;
char *p;
for (a=0; aNRGIGS; a++) {
p = (char *)malloc(BIGSIZE);
bzero(p, BIGSIZE);
}
printf(about to end in %d seconds...\n, SOMETIME);
sleep(SOMETIME);
printf(end now.\n);
return 0;
}
- snip -
If I find a way to isolate the problem, I'll post it here.
Thanks,
-cpghost.
--
Cordula's Web. http://www.cordula.ws/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
it swapped a lot, then wrote end and immediately exited. Hmmm... yes, it's strange. With malloc-ed space, exit is also very fast. On a 2 GB machine with amd64, exit is almost immediate: try mallocing 2 million times 2 kilobytes and fill. maybe exit first free all malloc'ed space which is definitely nonsense. If I find a way to isolate the problem, I'll post it here. maybe your problematic program uses atexit and exit actually calls part of it? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
In the last episode (Mar 15), cpghost said: I've noticed that when a huge, partially or totally swapped out process exits, there is a lot of disk activity going on, before the process truly dies. This is not necessarily due to sync(2), because it also happens with CPU bound processes that write very little output. Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). Are you sure this is actually in _exit, and not in a cleanup function executed by the application as it exits? If there is a large linked list, for example, and the author has decided to actually free the list before exiting instead of just letting it disappear when the process exits, each swapped-out page will have to be brought back in as the list is traversed. C++ programs may have destructors doing this behind the scenes. Best way to figure out what's going on is to attach to the program with gdb while it's swapping, and print a stack trace. Also, since you mentioned a totally swapped out process exiting, are you terminating it externally with kill -9? It may be writing a core dump, which will force the kernel to pull back swapped-out pages to write them to the core file. Couldn't this be avoided and the paged out pages simply discarded without reading them back in? Or do those pages contain necessary data at this point (page directories etc.)? -- Dan Nelson dnel...@allantgroup.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sun, 15 Mar 2009 11:01:41 +0100 cpghost cpgh...@cordula.ws wrote: On Sun, Mar 15, 2009 at 10:27:53AM +0100, Wojciech Puchar wrote: Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). is it your program and you are sure it's on exit? Every memory hungry program is concerned; and yes: it happens exactly on exit. it's because the program is writted the way it's doing a lot of things (probably unneeded) on exit. Have a look at what happens during exit: /usr/src/sys/kern/kern_exit.c:exit1() especially at the call to vm_waitproc(): /usr/src/sys/vm/vm_glue.c:vm_waitproc which calls vmspace_exitfree(): /usr/src/sys/vm/vm_map.c:vmspace_exitfree() Now, vmspace_exit() and vmspace_exitfree() ultimately call: /usr/src/sys/vm/vm_map.c:vmspace_dofree() It then goes deep into the bowels of vm amd pmap, and that's the place where the pages are paged in again (I think). Sounds right. This is easy to demonstrate on a laptop with 160MB RAM, running a bunch of servers + X + KDE, then running Mozilla, then opening about 30 tabs of pages, many of which run vast and buggy javascript .. By this stage mozilla is about 150MB with about 60MB resident, and swap is pushing 200MB. *seriously* paging, just on flipping to another tab. Now close mozilla and watch top while it's shutting down. Go and pour yourself a cuppa, there's no hurry .. Apart from having to close each tab/window, freeing all its resources, bits of the executable itself need to be paged in to do various things, which may need to page out some more. What's amazing is that it can do that for several minutes, coming out unscathed when it finally quits! (extreme example, but a true story from a wild ebay session yesterday :) not exit(2) itself Well that just starts that big VM ball rolling, so to speak .. so it's a tad more complex than a program that fills memory (+ swap) then exits. cheers, Ian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sun, Mar 15, 2009 at 10:48:49AM -0500, Dan Nelson wrote: In the last episode (Mar 15), cpghost said: I've noticed that when a huge, partially or totally swapped out process exits, there is a lot of disk activity going on, before the process truly dies. This is not necessarily due to sync(2), because it also happens with CPU bound processes that write very little output. Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). Are you sure this is actually in _exit, and not in a cleanup function executed by the application as it exits? If there is a large linked list, for example, and the author has decided to actually free the list before exiting instead of just letting it disappear when the process exits, each swapped-out page will have to be brought back in as the list is traversed. C++ programs may have destructors doing this behind the scenes. Yes, that's quite possible. Meanwhile, I'm suspecting that free(3) is the culprit, and not the vm subsystem itself; though I was not yet able to construct a good example to be sure. And you're quite right: cleaning up paged-out linked lists or other dynamic data structures, either explicitly or via C++ destructors is also an obvious reason for swap activity. Didn't think of it first. Best way to figure out what's going on is to attach to the program with gdb while it's swapping, and print a stack trace. I'll try this. Very good idea. Also, since you mentioned a totally swapped out process exiting, are you terminating it externally with kill -9? It may be writing a core dump, which will force the kernel to pull back swapped-out pages to write them to the core file. Also an excellent point. I'm just killing them with Ctrl-C (SIGINT), which won't result in a core dump. But a core dump would also reawaken the pages, that's quite clear. Dan Nelson dnel...@allantgroup.com Thanks for all the hints. The fog is slowly lifting. ;-) -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Mon, Mar 16, 2009 at 03:24:51AM +1100, Ian Smith wrote: On Sun, 15 Mar 2009 11:01:41 +0100 cpghost cpgh...@cordula.ws wrote: On Sun, Mar 15, 2009 at 10:27:53AM +0100, Wojciech Puchar wrote: Not sure what's really going on there, but apparently, the process reads in pages from swap that have been paged out previously (according to top(1)). is it your program and you are sure it's on exit? Every memory hungry program is concerned; and yes: it happens exactly on exit. it's because the program is writted the way it's doing a lot of things (probably unneeded) on exit. Have a look at what happens during exit: /usr/src/sys/kern/kern_exit.c:exit1() especially at the call to vm_waitproc(): /usr/src/sys/vm/vm_glue.c:vm_waitproc which calls vmspace_exitfree(): /usr/src/sys/vm/vm_map.c:vmspace_exitfree() Now, vmspace_exit() and vmspace_exitfree() ultimately call: /usr/src/sys/vm/vm_map.c:vmspace_dofree() It then goes deep into the bowels of vm amd pmap, and that's the place where the pages are paged in again (I think). Sounds right. This is easy to demonstrate on a laptop with 160MB RAM, running a bunch of servers + X + KDE, then running Mozilla, then opening about 30 tabs of pages, many of which run vast and buggy javascript .. By this stage mozilla is about 150MB with about 60MB resident, and swap is pushing 200MB. *seriously* paging, just on flipping to another tab. Now close mozilla and watch top while it's shutting down. Go and pour yourself a cuppa, there's no hurry .. Apart from having to close each tab/window, freeing all its resources, bits of the executable itself need to be paged in to do various things, which may need to page out some more. What's amazing is that it can do that for several minutes, coming out unscathed when it finally quits! Yes, that's exactly what I'm seeing. And of course, after firefox finally shuts down, everything else is also (mostly) paged out, including most of X, so it takes some time to get a fully responsive system again. It's amazing how closing a process can actually make the system even less responsive. But as Dan has pointed out, firefox et al. are probably written in such a way that they reawaken all their dynamic data structures from swap while cleaning up. There's not much one can do from the OS side to prevent this from happening. (extreme example, but a true story from a wild ebay session yesterday :) not exit(2) itself Well that just starts that big VM ball rolling, so to speak .. so it's a tad more complex than a program that fills memory (+ swap) then exits. Yup, that's obviously more complicated: C++ dtors, atexit() handlers, etc. are called at this point. VM itself seems fast enough to clean up the vm space of any process without much swap I/O (but I'm not 100% sure yet). cheers, Ian Thanks, -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
Apart from having to close each tab/window, freeing all its resources, bits of the executable itself need to be paged in to do various things, which may need to page out some more. What's amazing is that it can do that for several minutes, coming out unscathed when it finally quits! (extreme example, but a true story from a wild ebay session yesterday :) perfect example of crappy software - unfortunately most todays web browsers are coded by people that don't spend even few minutes understanding how OS works. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
But as Dan has pointed out, firefox et al. are probably written in such a way that they reawaken all their dynamic data structures from swap while cleaning up. There's not much one can do from the OS side to prevent this from happening. indeed. there are no fix for crappy software, usually written in C++. Yup, that's obviously more complicated: C++ dtors, atexit() handlers, etc. are called at this point. VM itself seems fast enough to clean up the vm space of any process without much swap I/O (but I'm not 100% sure yet). yes it is ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
By this stage mozilla is about 150MB with about 60MB resident, and swap is pushing 200MB. *seriously* paging, just on flipping to another tab. Now close mozilla and watch top while it's shutting down. Go and pour yourself a cuppa, there's no hurry .. just tested with opera - the same. crappy software rulez ;) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
Wojciech Puchar wrote: By this stage mozilla is about 150MB with about 60MB resident, and swap is pushing 200MB. *seriously* paging, just on flipping to another tab. Now close mozilla and watch top while it's shutting down. Go and pour yourself a cuppa, there's no hurry .. just tested with opera - the same. crappy software rulez ;) Yes, clearly web browsers should be optimized for speed of exiting. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
Yes, clearly web browsers should be optimized for speed of exiting. Oh, yes, definitely. But, honestly, a marginally minor resource footprint would not be bad, either. But I've got a certain feeling that this thread'll go to advocacy sooner or later :) Regards, Peter. -- Nur bis 16.03.! DSL-Komplettanschluss inkl. WLAN-Modem für nur 17,95 ¿/mtl. + 1 Monat gratis!* http://dsl.gmx.de/?ac=OM.AD.PD003K11308T4569a ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
just tested with opera - the same. crappy software rulez ;) Yes, clearly web browsers should be optimized for speed of exiting. of other things - too ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
But, honestly, a marginally minor resource footprint would not be bad, either. But I've got a certain feeling that this thread'll go to advocacy sooner or later :) would be really nice about advocacy of GOOD web browser that not only exits faster, but WORKS faster. i don't know any, except links but links doesn't have CSS support ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
Wojciech Puchar wrote: just tested with opera - the same. crappy software rulez ;) Yes, clearly web browsers should be optimized for speed of exiting. of other things - too My point is that for all your high handed judgement, you have no idea what trade-offs might be in play here ;) Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sun, Mar 15, 2009 at 05:57:47PM +, Kris Kennaway wrote: Wojciech Puchar wrote: By this stage mozilla is about 150MB with about 60MB resident, and swap is pushing 200MB. *seriously* paging, just on flipping to another tab. Now close mozilla and watch top while it's shutting down. Go and pour yourself a cuppa, there's no hurry .. just tested with opera - the same. crappy software rulez ;) Yes, clearly web browsers should be optimized for speed of exiting. That's correct. Normally systematic freeing of resources is one of the best ways to prevent memory leaks in long running programs, so it can't be too bad to free/delete all malloc-ed/new-ed chunks upon exit. But here, the opposite may be better: there should be a faster way to exit, bypassing all manual and automatic cleaners. But that's up to every single program and not OS'es business. Thanks again for all the insight. Kris -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Speeding up exit(2)?
On Sunday 15 March 2009 11:20:58 am Kris Kennaway wrote: Wojciech Puchar wrote: just tested with opera - the same. crappy software rulez ;) Yes, clearly web browsers should be optimized for speed of exiting. of other things - too My point is that for all your high handed judgement, you have no idea what trade-offs might be in play here ;) I have always felt that good web programming took that into account. When I show an image, I use a mouse action that pops up a new window. I use another mouse action that allows you click on the image window that closes it and you are back to the original window with NO network activity. It is really fast :). Kent -- kent Stewart Richland, WA http://users.owt.com/kstewart/index.html ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
