Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Hi, Paul D. Smith elucidated on 29/11/06 02:27: [...] Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do). Is it undefined in actuality though? Has anyone checked if NULL does get returned from any implementations? (I'm surprised alloca() implementations didn't end up getting NULL returned in implementations over the years. The GLIBC Manual indicates a fatal signal is generated from its implementation: http://www.gnu.org/software/libc/manual/html_node/Disadvantages-of-Alloca.html My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Jon Grant wrote: My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon Choice of malloc implementation can have a huge effect on execution time. See this data http://highlandsun.com/hyc/#Malloc Some modern mallocs are good, but stack-based allocation is still better a lot of the time. Especially for temporary variables that are just going to be discarded after a few computations. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Wed, 2006-11-29 at 13:22 +, Jon Grant wrote: Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do). Is it undefined in actuality though? Has anyone checked if NULL does get returned from any implementations? From the standpoint of GNU make it doesn't matter all that much what some implementations do. In order to be portable it must assume the lowest common denominator. My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Well, yes, but changing from malloc() to calloc() actually has a significant difference: in the latter all the memory you've allocated needs to be zeroed out. That can have all kinds of implications even beyond the obvious need to walk all the bytes in each allocation, in terms of swapping, cache issues, etc. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Follow-up Comment #3, bug #18396 (project make): Regarding efficiency: First, the main point of my report is not alloca sucks but rather that setrlimit is an unexpected thing for make to do. But having opened up the can of worms, let me play with them a little (I don't really want to start a big argument, but I do want to defend my previous statements): In my experience, the dominant effect on program performance, after choice of algorithm, is data locality: frequently re-using the same or nearby memory locations. Instruction count is at best a second-order effect. Andrew Appel has a lovely paper where he proves that (of all things) garbage collection is faster than stack allocation, by exclusively focusing on instruction count: http://citeseer.ist.psu.edu/appel87garbage.html Of course, the analysis is fatally flawed by ignoring locality. My point regarding alloca is that if I have a lot of data, it cannot all be hot (high locality). If I put cold data on the stack, the stack as a whole has poorer locality, because the hot regions (stack frames) are interspersed with cold regions (the big chunks of data). So if I intend to allocate infrequently-accessed data, it is better to put it on the heap where it won't interfere with the locality of stack accesses. You say that only medium sized data ends up on the stack, which certainly sounds like a reasonable approach, but if there is enough of it to warrant calling setrlimit, then I have to suspect that some of it would be better placed elsewhere, from a strictly performance-oriented point of view. Regarding portability: Indeed, there is no portable way to detect running out of stack space, whether by alloca or not, though alloca obviously makes running out of stack space more likely. Of course, setrlimit is not all that portable either, so I was assuming that nonportable solutions were on the table. Toward that end, an appropriate signal handler (which will need to use nonportable tricks to distinguish stack overflow from other types of segfaults), or the GCC -fstack-check argument (when compiling with GCC, which is after all the common case) may be options. Moreover, I would argue that the architectural decision to make heavy reliance on alloca is what leads us into nonportable territory in the first place. It is not uncommon to find systems with ~10MB or less of stack space. The setrlimit call is a band-aid; in some circumstances it will help, but that call is not an antidote to running out of stack space. Of course, the big advantage of alloca is that it is easy to program with. I would not advocate suddenly rewriting GNU make to use malloc everywhere, since (as you say) that will likely introduce many more serious bugs than this one in the short term. (If it was written in C++ there would be better options, but that ship too has sailed.) So, at most, I would suggest doing some profiling to find the biggest consumers of alloca space and change just them (hopefully only a handful) to use [x]malloc instead. The primary motivation would be enhanced portability; the efficiency argument is just meant to allay fears that this will make everything slower. Regarding setrlimit itself: Yes, I would consider this bug (and I do consider it a bug...) to be fixed if 'make' would reliably return the resource limits to their original settings before exec'ing its children. ___ Reply to this item at: http://savannah.gnu.org/bugs/?18396 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
