Re: make -jN buildworld on 512MB ram
On Wed, Oct 31, 2012 at 12:58:18PM -0700, Alfred Perlstein wrote: It seems like the new compiler likes to get up to ~200+MB resident when building some basic things in our tree. Unfortunately this causes smaller machines (VMs) to take days because of swap thrashing. You could try reducing data seg size (ulimit -d) to something matching RAM size. GCC has two parameters to try to keep its memory usage reasonable: http://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/Optimize-Options.html 'ggc-min-expand' and 'ggc-min-heapsize'. ggc-min-expand ... If getrlimit is available, the notion of RAM is the smallest of actual RAM and RLIMIT_DATA ggc-min-heapsize The default is the smaller of RAM/8, RLIMIT_RSS, or a limit which tries to ensure that RLIMIT_DATA or RLIMIT_AS are not exceeded I don't know if Clang/LLVM has something similar or not. -- -- David (obr...@freebsd.org) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Fwd: Re: make -jN buildworld on 512MB ram
Ah, but make(1) can delay spawning any new processes when it knows its
children are paging.
Sounds good - but I doubt the complexity it adds would really be worth
it.
If the real problem is the C++ compiler or other large tool needing
sufficient resources that you have to ensure invocations are
sequenced/throttled, then the obvious solution would seem to be to
invoke it via a proxy that can do the the needed throttling on a global
(system, user, or whatever basis seems to make sense).
Think of make's invocation of ${CXX} as actually just submitting a job
to the CXX service, which might only be allowing one instance of C++ to
run at a time.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On Thu, 1 Nov 2012, Peter Jeremy wrote: On 2012-Oct-31 12:58:18 -0700, Alfred Perlstein bri...@mu.org wrote: It seems like the new compiler likes to get up to ~200+MB resident when building some basic things in our tree. The killer I found was the ctfmerge(1) on the kernel - which exceeds ~400MB on i386. Under low RAM, that fails _without_ reporting any errors back to make(1), resulting in a corrupt new kernel (it booted but had virtually no devices so it couldn't find root). Doesn't our make(1) have some stuff to mitigate this? I would expect it to be a bit smarter about detecting the number of swaps/pages/faults of its children and taking into account the machine's total ram before forking off new processes. The difficulty I see is that the make process can't tell anything about the memory requirements of the pipeline it is about to spawn. As a rule of thumb, C++ needs more memory than C but that depends on what is being compiled - I have a machine-generated C program that makes gcc bloat to ~12GB. Any ideas? I mean a really simple algorithm could be devised that would be better than what we appear to have (which is nothing). If you can afford to waste CPU, one approach would be for make(1) to setrlimit(2) child processes and if the child dies, it retries that child by itself - but that will generate unnecessary retries. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). Starts to sound similar to parts of linux cgroups. Which, incidentally, a friend of mine was recently complaining about not being able to run systemd in the linuxolator due to the lack of certain syscalls pertaining to cgroups... -Ben Kaduk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On Wed, Oct 31, 2012 at 12:58 PM, Alfred Perlstein bri...@mu.org wrote: It seems like the new compiler likes to get up to ~200+MB resident when building some basic things in our tree. Unfortunately this causes smaller machines (VMs) to take days because of swap thrashing. Doesn't our make(1) have some stuff to mitigate this? I would expect it to be a bit smarter about detecting the number of swaps/pages/faults of its children and taking into account the machine's total ram before forking off new processes. I know gmake has some algorithms, although last I checked they were very naive and didn't work well. Any ideas? I mean a really simple algorithm could be devised that would be better than what we appear to have (which is nothing). Even if an algorithm can't be come up with, why not something just to throttle the max number of c++/g++ processes thrown out. Maybe I'm missing a trick I can pull off with some make.conf knobs? Idk, summer of code idea? Anyone mentoring someone they want to have a look at this? FreeBSD make/bmake doesn't, but gmake sure does with --load [1]! I would ask sjg@ to be absolutely sure, but I don't see any getrlimit/setrlimit calls around the code, apart from RLIMIT_NOFILE. For now you have to tune your number of jobs appropriately.. HTH, -Garrett PS I feel your pain because I have a number of FreeBSD VMs with =2GB RAM. 1. http://www.gnu.org/software/make/manual/make.html#index-g_t_0040code_007b_002d_002dload_002daverage_007d-746 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On 2012-Oct-31 12:58:18 -0700, Alfred Perlstein bri...@mu.org wrote: It seems like the new compiler likes to get up to ~200+MB resident when building some basic things in our tree. The killer I found was the ctfmerge(1) on the kernel - which exceeds ~400MB on i386. Under low RAM, that fails _without_ reporting any errors back to make(1), resulting in a corrupt new kernel (it booted but had virtually no devices so it couldn't find root). Doesn't our make(1) have some stuff to mitigate this? I would expect it to be a bit smarter about detecting the number of swaps/pages/faults of its children and taking into account the machine's total ram before forking off new processes. The difficulty I see is that the make process can't tell anything about the memory requirements of the pipeline it is about to spawn. As a rule of thumb, C++ needs more memory than C but that depends on what is being compiled - I have a machine-generated C program that makes gcc bloat to ~12GB. Any ideas? I mean a really simple algorithm could be devised that would be better than what we appear to have (which is nothing). If you can afford to waste CPU, one approach would be for make(1) to setrlimit(2) child processes and if the child dies, it retries that child by itself - but that will generate unnecessary retries. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). -- Peter Jeremy pgpOxQEkEC3S2.pgp Description: PGP signature
Re: make -jN buildworld on 512MB ram
On 31 October 2012 13:41, Peter Jeremy pe...@rulingia.com wrote: Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). Well, does the compiler actually need that much memory? :) Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On 10/31/12 1:41 PM, Peter Jeremy wrote: On 2012-Oct-31 12:58:18 -0700, Alfred Perlstein bri...@mu.org wrote: It seems like the new compiler likes to get up to ~200+MB resident when building some basic things in our tree. The killer I found was the ctfmerge(1) on the kernel - which exceeds ~400MB on i386. Under low RAM, that fails _without_ reporting any errors back to make(1), resulting in a corrupt new kernel (it booted but had virtually no devices so it couldn't find root). Trolled by FreeBSD. :) Doesn't our make(1) have some stuff to mitigate this? I would expect it to be a bit smarter about detecting the number of swaps/pages/faults of its children and taking into account the machine's total ram before forking off new processes. The difficulty I see is that the make process can't tell anything about the memory requirements of the pipeline it is about to spawn. As a rule of thumb, C++ needs more memory than C but that depends on what is being compiled - I have a machine-generated C program that makes gcc bloat to ~12GB. Ah, but make(1) can delay spawning any new processes when it knows its children are paging. This is sort of like well you can't predict when an elevator will plunge to its doom. ...but you can stop loading hapless people onto it when it starts creaking... (paging/swapping). Any ideas? I mean a really simple algorithm could be devised that would be better than what we appear to have (which is nothing). If you can afford to waste CPU, one approach would be for make(1) to setrlimit(2) child processes and if the child dies, it retries that child by itself - but that will generate unnecessary retries. This doesn't really help. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). I think that could be done, this is actually a very interesting idea. Another idea is for make(1) to start to kill -STOP a child when it detects a lot of child paging until other independent children complete running, which is basically what I do manually when my build explodes until it gets past some C++ bits. *ugh* -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On Wed, Oct 31, 2012 at 1:44 PM, Adrian Chadd adr...@freebsd.org wrote: On 31 October 2012 13:41, Peter Jeremy pe...@rulingia.com wrote: Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). Well, does the compiler actually need that much memory? :) For calculating SSAs, directed graphs, and when dealing with larger optimization levels -- sadly, yes it can. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: make -jN buildworld on 512MB ram
On 2012-Oct-31 14:21:51 -0700, Alfred Perlstein bri...@mu.org wrote: Ah, but make(1) can delay spawning any new processes when it knows its children are paging. That could work in some cases and may be worth implementing. Where it won't work is when make(1) initially hits a parallelisable block of big programs after a series of short, small tasks: System is OK so the first big program is spawned. ~100msec later, the next small task finishes. System in still OK (because the first big task is still growing and hasn't achieved peak bloat[*]) so it spawns another big task. Repeat a few times and you have a collection of big processes starting to thrash the system. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). I think that could be done, this is actually a very interesting idea. Another idea is for make(1) to start to kill -STOP a child when it detects a lot of child paging until other independent children complete running, which is basically what I do manually when my build explodes until it gets past some C++ bits. This is roughly a userland variant of the scheduler change above. The downside is that make(1) can no longer just wait(2) for a process to exit and then decide what to do next. Instead, it needs to poll the system's paging activity and take action on one of its children. Some of the special cases it needs ta handle are: 1) The offending process isn't a direct child but a more distant descendent - this will be the typical case: make(1) starts gcc(1) which spawns cc1plus which bloats. 2) Multiple (potentially independent) make(1) processes all detect that the system is too busy and stop their children. Soon after, the system is free so they all SIGCONT their children. Repeat. (Note that any scheduler changes also need to cope with this). [*] Typical cc1/cc1plus behaviour is to steadily grow as the input is processed. At higher optimisation levels, parse trees are not freed at the end of a function to allow global inlining and optimisation. -- Peter Jeremy pgpQZGAx1DMTZ.pgp Description: PGP signature
Re: make -jN buildworld on 512MB ram
On 10/31/12 3:14 PM, Peter Jeremy wrote: On 2012-Oct-31 14:21:51 -0700, Alfred Perlstein bri...@mu.org wrote: Ah, but make(1) can delay spawning any new processes when it knows its children are paging. That could work in some cases and may be worth implementing. Where it won't work is when make(1) initially hits a parallelisable block of big programs after a series of short, small tasks: System is OK so the first big program is spawned. ~100msec later, the next small task finishes. System in still OK (because the first big task is still growing and hasn't achieved peak bloat[*]) so it spawns another big task. Repeat a few times and you have a collection of big processes starting to thrash the system. True, but the idea is to somewhat mitigate it, not solve it completely. So sure, you might thrash for a while, but I've seen buildworld thrash for HOURS not making any progress, so even if it thrashes for a bit that is a big win over endless thrashing. Another, more involved, approach would be for the scheduler to manage groups of processes - if a group of processes is causing memory pressure as a whole then the scheduler just stops scheduling some of them until the pressure reduces (effectively swap them out). (Yes, that's vague and lots of hand-waving that might not be realisable). I think that could be done, this is actually a very interesting idea. Another idea is for make(1) to start to kill -STOP a child when it detects a lot of child paging until other independent children complete running, which is basically what I do manually when my build explodes until it gets past some C++ bits. This is roughly a userland variant of the scheduler change above. The downside is that make(1) can no longer just wait(2) for a process to exit and then decide what to do next. Instead, it needs to poll the system's paging activity and take action on one of its children. Some of the special cases it needs ta handle are: 1) The offending process isn't a direct child but a more distant descendent - this will be the typical case: make(1) starts gcc(1) which spawns cc1plus which bloats. 2) Multiple (potentially independent) make(1) processes all detect that the system is too busy and stop their children. Soon after, the system is free so they all SIGCONT their children. Repeat. (Note that any scheduler changes also need to cope with this). [*] Typical cc1/cc1plus behaviour is to steadily grow as the input is processed. At higher optimisation levels, parse trees are not freed at the end of a function to allow global inlining and optimisation. Sure, these are obstacles, but I do not think they are insurmountable. 1 can be addressed by walking the process tree. 2 can be addressed by simply setting an environment flag that denotes the MASTER process, so that subchildren do not try to schedule as well. -Alfred ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
