RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
However, your point about programs invoked by make inheriting the
setrlimit() is definitely something that seems problematic. Perhaps
GNU
make could change the stack limit back to what it was after it forks
but
before it execs its child. I wonder what happens if you change a
limit to
something too small for the current processes resources?
It doesn't look specified by
http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html.
This test suggests that it works fine on my Linux box. I presume the
limit check is normally done on a faulting access off the end of the
stack, so setrlimit would need some to have some different and special
code to check the current stack usage if it wanted to apply the check
immediately. It seems a bit unlikely that anyone would have gone to
that effort, thus stopping you from doing something useful in this sort
of situation.
#include alloca.h
#include errno.h
#include iostream
#include stddef.h
#include string.h
#include sys/time.h
#include sys/resource.h
void useLotsOfStack(size_t size);
size_t getAndDisplayStackLimit() {
rlimit resourceLimit;
if (getrlimit(RLIMIT_STACK, resourceLimit) 0) {
std::cerr getrlimit failed with errno std::endl;
}
std::cout rlim_cur == resourceLimit.rlim_cur std::endl;
std::cout rlim_max == resourceLimit.rlim_max std::endl;
return resourceLimit.rlim_cur;
}
void setStackLimit(size_t size) {
rlimit resourceLimit;
resourceLimit.rlim_cur = size;
if (setrlimit(RLIMIT_STACK, resourceLimit) 0) {
std::cerr setrlimit failed with errno std::endl;
exit(1);
}
getAndDisplayStackLimit();
}
int main() {
size_t initialLimit = getAndDisplayStackLimit();
//useLotsOfStack(initialLimit * 10);
setStackLimit(initialLimit * 100);
useLotsOfStack(initialLimit * 10);
setStackLimit(initialLimit);
useLotsOfStack(initialLimit * 10);
}
void useLotsOfStack(size_t size) {
memset(alloca(size), 0, size);
}
-
Martin's Outlook, BlueArc Engineering
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Martin Dorey wrote: However, your point about programs invoked by make inheriting the setrlimit() is definitely something that seems problematic. Perhaps GNU make could change the stack limit back to what it was after it forks but before it execs its child. I wonder what happens if you change a limit to something too small for the current processes resources? It doesn't look specified by http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html. This test suggests that it works fine on my Linux box. I presume the limit check is normally done on a faulting access off the end of the stack, so setrlimit would need some to have some different and special code to check the current stack usage if it wanted to apply the check immediately. It seems a bit unlikely that anyone would have gone to that effort, thus stopping you from doing something useful in this sort of situation. In Linux (and most other Unix systems) the stack is allocated in page-sized chunks, and an extra page with no access permissions is mapped at the end of it. Accessing that page is what generates the SEGV that occurs when you overrun the stack. The mappings are only established when the stack is grown, and generally the stack never shrinks. So if you already have X pages of stack legitimately in use, lowering the rlimit isn't going to have an immediate effect. I.e., the stack growth occurred before the limit decreased. In general, Unix doesn't impose restrictions retroactively. (E.g., if you open a file and have a valid descriptor on it, and someone else chmods the file, removing your access permissions, you still have your original privileges through the open descriptor.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Hi, Paul D. Smith elucidated on 29/11/06 02:27: [...] Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do). Is it undefined in actuality though? Has anyone checked if NULL does get returned from any implementations? (I'm surprised alloca() implementations didn't end up getting NULL returned in implementations over the years. The GLIBC Manual indicates a fatal signal is generated from its implementation: http://www.gnu.org/software/libc/manual/html_node/Disadvantages-of-Alloca.html My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Jon Grant wrote: My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon Choice of malloc implementation can have a huge effect on execution time. See this data http://highlandsun.com/hyc/#Malloc Some modern mallocs are good, but stack-based allocation is still better a lot of the time. Especially for temporary variables that are just going to be discarded after a few computations. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Wed, 2006-11-29 at 13:22 +, Jon Grant wrote: Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do). Is it undefined in actuality though? Has anyone checked if NULL does get returned from any implementations? From the standpoint of GNU make it doesn't matter all that much what some implementations do. In order to be portable it must assume the lowest common denominator. My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Well, yes, but changing from malloc() to calloc() actually has a significant difference: in the latter all the memory you've allocated needs to be zeroed out. That can have all kinds of implications even beyond the obvious need to walk all the bytes in each allocation, in terms of swapping, cache issues, etc. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Using heap, which requires a system call to get more memory
(It doesn't affect the main point of Paul's reply but just for academic
interest) no it doesn't:
[EMAIL PROTECTED]:~/playpen$ cat ten-thousand-mallocs.c
#include stdlib.h
int main() {
for (int ii = 0; ii != 10 * 1000; ++ ii) {
free(malloc(1));
}
}
[EMAIL PROTECTED]:~/playpen$ g++ ten-thousand-mallocs.c
[EMAIL PROTECTED]:~/playpen$ strace ./a.out 21 | wc -l
152
[EMAIL PROTECTED]:~/playpen$
Even in less contrived applications, brk isn't called anything like as
often as malloc.
-
Martin's Outlook, BlueArc Engineering
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Tue, 2006-11-28 at 18:45 -0800, Martin Dorey wrote: Using heap, which requires a system call to get more memory (It doesn't affect the main point of Paul's reply but just for academic interest) no it doesn't: Even in less contrived applications, brk isn't called anything like as often as malloc. Certainly that's true... I didn't mean to suggest every call to malloc() resulted in a system call. But using the heap DOES require a system call to get memory... obviously the C runtime will grab memory in much larger chunks and maintains a free pool, with algorithms to handle fragmentation and coalescing of free blocks, etc. There are whole dissertations written on the most efficient ways to manage heap :). But (as I'm sure we both agree) alloca() is unquestionably more efficient (assuming you have an appropriate amount of stack). ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
