Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Martin Dorey wrote: However, your point about programs invoked by make inheriting the setrlimit() is definitely something that seems problematic. Perhaps GNU make could change the stack limit back to what it was after it forks but before it execs its child. I wonder what happens if you change a limit to something too small for the current processes resources? It doesn't look specified by http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html. This test suggests that it works fine on my Linux box. I presume the limit check is normally done on a faulting access off the end of the stack, so setrlimit would need some to have some different and special code to check the current stack usage if it wanted to apply the check immediately. It seems a bit unlikely that anyone would have gone to that effort, thus stopping you from doing something useful in this sort of situation. In Linux (and most other Unix systems) the stack is allocated in page-sized chunks, and an extra page with no access permissions is mapped at the end of it. Accessing that page is what generates the SEGV that occurs when you overrun the stack. The mappings are only established when the stack is grown, and generally the stack never shrinks. So if you already have X pages of stack legitimately in use, lowering the rlimit isn't going to have an immediate effect. I.e., the stack growth occurred before the limit decreased. In general, Unix doesn't impose restrictions retroactively. (E.g., if you open a file and have a valid descriptor on it, and someone else chmods the file, removing your access permissions, you still have your original privileges through the open descriptor.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
> However, your point about programs invoked by make inheriting the > setrlimit() is definitely something that seems problematic. Perhaps GNU > make could change the stack limit back to what it was after it forks but > before it execs its child. I wonder what happens if you change a limit to > something too small for the current processes resources? It doesn't look specified by http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html. This test suggests that it works fine on my Linux box. I presume the limit check is normally done on a faulting access off the end of the stack, so setrlimit would need some to have some different and special code to check the current stack usage if it wanted to apply the check immediately. It seems a bit unlikely that anyone would have gone to that effort, thus stopping you from doing something useful in this sort of situation. #include #include #include #include #include #include #include void useLotsOfStack(size_t size); size_t getAndDisplayStackLimit() { rlimit resourceLimit; if (getrlimit(RLIMIT_STACK, &resourceLimit) < 0) { std::cerr << "getrlimit failed with " << errno << std::endl; } std::cout << "rlim_cur == " << resourceLimit.rlim_cur << std::endl; std::cout << "rlim_max == " << resourceLimit.rlim_max << std::endl; return resourceLimit.rlim_cur; } void setStackLimit(size_t size) { rlimit resourceLimit; resourceLimit.rlim_cur = size; if (setrlimit(RLIMIT_STACK, &resourceLimit) < 0) { std::cerr << "setrlimit failed with " << errno << std::endl; exit(1); } getAndDisplayStackLimit(); } int main() { size_t initialLimit = getAndDisplayStackLimit(); //useLotsOfStack(initialLimit * 10); setStackLimit(initialLimit * 100); useLotsOfStack(initialLimit * 10); setStackLimit(initialLimit); useLotsOfStack(initialLimit * 10); } void useLotsOfStack(size_t size) { memset(alloca(size), 0, size); } - Martin's Outlook, BlueArc Engineering ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Wed, 2006-11-29 at 13:22 +, Jon Grant wrote: > > Finally, there is no way to detect an out of stack error and exit gracefully > > with a warning as you suggest: the behavior of alloca() is undefined if you > > run out of stack space (it doesn't just return NULL as malloc() etc. do). > Is it undefined in actuality though? Has anyone checked if NULL does get > returned from any implementations? >From the standpoint of GNU make it doesn't matter all that much what some implementations do. In order to be portable it must assume the lowest common denominator. > My view would be that on modern computers switching to allocate from the > heap wouldn't make a big difference if it were changed. Modern heaps > have pools for small allocations to stop them fragmenting larger > allocations anyway. Someone would need to do a compressive test to know > for sure, these things often have knock on effects.. I've seen massive > slowdowns when someone switched malloc() to calloc() on MS-Windows! Well, yes, but changing from malloc() to calloc() actually has a significant difference: in the latter all the memory you've allocated needs to be zeroed out. That can have all kinds of implications even beyond the obvious need to walk all the bytes in each allocation, in terms of swapping, cache issues, etc. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Wed, 2006-11-29 at 05:49 -0800, Howard Chu wrote: > Some modern mallocs are good, but stack-based allocation is still better > a lot of the time. Especially for temporary variables that are just > going to be discarded after a few computations. Right. And remember it's not just the code managing the heap, but also the extra work involved in ensuring that the memory is freed again. This can mean reworking functions, changing code paths, etc. I'm not saying that the performance difference would be significant: without testing it's difficult to guess. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Jon Grant wrote: My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon Choice of malloc implementation can have a huge effect on execution time. See this data http://highlandsun.com/hyc/#Malloc Some modern mallocs are good, but stack-based allocation is still better a lot of the time. Especially for temporary variables that are just going to be discarded after a few computations. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Hi, Paul D. Smith elucidated on 29/11/06 02:27: [...] > Finally, there is no way to detect an out of stack error and exit gracefully > with a warning as you suggest: the behavior of alloca() is undefined if you > run out of stack space (it doesn't just return NULL as malloc() etc. do). Is it undefined in actuality though? Has anyone checked if NULL does get returned from any implementations? (I'm surprised alloca() implementations didn't end up getting NULL returned in implementations over the years. The GLIBC Manual indicates a fatal signal is generated from its implementation: http://www.gnu.org/software/libc/manual/html_node/Disadvantages-of-Alloca.html My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Tue, 2006-11-28 at 18:45 -0800, Martin Dorey wrote: > > Using heap, which requires a system call to get more memory > > (It doesn't affect the main point of Paul's reply but just for academic > interest) no it doesn't: > Even in less contrived applications, brk isn't called anything like as > often as malloc. Certainly that's true... I didn't mean to suggest every call to malloc() resulted in a system call. But using the heap DOES require a system call to get memory... obviously the C runtime will grab memory in much larger chunks and maintains a free pool, with algorithms to handle fragmentation and coalescing of free blocks, etc. There are whole dissertations written on the most efficient ways to manage heap :). But (as I'm sure we both agree) alloca() is unquestionably more efficient (assuming you have an appropriate amount of stack). ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
> Using heap, which requires a system call to get more memory (It doesn't affect the main point of Paul's reply but just for academic interest) no it doesn't: [EMAIL PROTECTED]:~/playpen$ cat ten-thousand-mallocs.c #include int main() { for (int ii = 0; ii != 10 * 1000; ++ ii) { free(malloc(1)); } } [EMAIL PROTECTED]:~/playpen$ g++ ten-thousand-mallocs.c [EMAIL PROTECTED]:~/playpen$ strace ./a.out 2>&1 | wc -l 152 [EMAIL PROTECTED]:~/playpen$ Even in less contrived applications, brk isn't called anything like as often as malloc. - Martin's Outlook, BlueArc Engineering ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make