Re: MAX_ARG_PAGES has no effect?
* Andi Kleen <[EMAIL PROTECTED]> wrote: > On Thursday 01 September 2005 08:57, Ingo Molnar wrote: > > > the whole thing should be reworked, so that there is no artificial limit > > like MAX_ARG_PAGES. (it is after all just another piece of memory, in > > theory) > > Yes, a sysctl would probably lead to fragmentation problems and then > people would do ugly linked lists of buffers like poll. not really fragmentation problems (the unit of allocation of argument pages is already a single page, and we do an array of pages), the real problem is the DoS - right now the array pages are unswappable while an exec() is ongoing. > > If we do unconditional page-flipping then we fragment the argument > > space, if we do both page-flipping if things are unfragmented and > > well-aligned, and 'compact' the layout otherwise, we havent solved the > > problem and have introduced a significant extra layer of complexity to > > an already security-sensitive and fragile piece of code. > > Page flipping = COW like fork would do? i dont think we need COW. During execve() we are destroying the old context and are creating a completely new context, so in theory we could just 'flip over' the argument/environment pages (which are a parameter to sys_execve()) from the old mm into the newly created mm, without caring about the old mm. > Not sure how this would work - the arguments of execve can be anywhere > in the address space and would presumably be often be in a > inconvenient place like in the middle of the stack of the new > executable. yes, that's one of the issues. I've done some instrumentation some time ago and it seemed that the arguments are typically page-aligned, so the only issue would be to clear the partial page at the end of the arguments. But i still think the concept is volatile. > > The best method i found was to get rid of bprm->pages[] and to directly > > copy strings into the new mm via kmap (and to follow whatever RAM > > allocation policies/limits there are for the new mm), but that's quite > > ugly. > > That sounds better. yeah. It's also pretty laborous though. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
On Thursday 01 September 2005 08:57, Ingo Molnar wrote: > the whole thing should be reworked, so that there is no artificial limit > like MAX_ARG_PAGES. (it is after all just another piece of memory, in > theory) Yes, a sysctl would probably lead to fragmentation problems and then people would do ugly linked lists of buffers like poll. > If we do unconditional page-flipping then we fragment the argument > space, if we do both page-flipping if things are unfragmented and > well-aligned, and 'compact' the layout otherwise, we havent solved the > problem and have introduced a significant extra layer of complexity to > an already security-sensitive and fragile piece of code. Page flipping = COW like fork would do? Not sure how this would work - the arguments of execve can be anywhere in the address space and would presumably be often be in a inconvenient place like in the middle of the stack of the new executable. > The best method i found was to get rid of bprm->pages[] and to directly > copy strings into the new mm via kmap (and to follow whatever RAM > allocation policies/limits there are for the new mm), but that's quite > ugly. That sounds better. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
* Andi Kleen <[EMAIL PROTECTED]> wrote: > Ingo Molnar <[EMAIL PROTECTED]> writes: > > > > MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' > > output is hardcoded. (because the kernel does not provide the > > information dynamically) > > Perhaps it would be a good idea to make it a sysctl. Is there any > reason it should be hardcoded? I cannot think of any. > > Ok if someone lowers the sysctl then execve has to handle the case of > the args/environment possibly not fitting anymore, but that should be > easy. the whole thing should be reworked, so that there is no artificial limit like MAX_ARG_PAGES. (it is after all just another piece of memory, in theory) I have tried this a couple of times but failed - it's a hard problem. Linus had the idea years ago to page-flip the argument data into the new process's address space, but that doesnt work out in practice due to the way glibc has to extend the environment space. (glibc extends it by modifying the environment array, or relocating it if it has to be grown. execve() currently automatically 'linearizes' the environment by copying both the array and the old and new environment strings to a linear piece of memory.) If we do unconditional page-flipping then we fragment the argument space, if we do both page-flipping if things are unfragmented and well-aligned, and 'compact' the layout otherwise, we havent solved the problem and have introduced a significant extra layer of complexity to an already security-sensitive and fragile piece of code. The best method i found was to get rid of bprm->pages[] and to directly copy strings into the new mm via kmap (and to follow whatever RAM allocation policies/limits there are for the new mm), but that's quite ugly. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
* Andi Kleen [EMAIL PROTECTED] wrote: Ingo Molnar [EMAIL PROTECTED] writes: MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' output is hardcoded. (because the kernel does not provide the information dynamically) Perhaps it would be a good idea to make it a sysctl. Is there any reason it should be hardcoded? I cannot think of any. Ok if someone lowers the sysctl then execve has to handle the case of the args/environment possibly not fitting anymore, but that should be easy. the whole thing should be reworked, so that there is no artificial limit like MAX_ARG_PAGES. (it is after all just another piece of memory, in theory) I have tried this a couple of times but failed - it's a hard problem. Linus had the idea years ago to page-flip the argument data into the new process's address space, but that doesnt work out in practice due to the way glibc has to extend the environment space. (glibc extends it by modifying the environment array, or relocating it if it has to be grown. execve() currently automatically 'linearizes' the environment by copying both the array and the old and new environment strings to a linear piece of memory.) If we do unconditional page-flipping then we fragment the argument space, if we do both page-flipping if things are unfragmented and well-aligned, and 'compact' the layout otherwise, we havent solved the problem and have introduced a significant extra layer of complexity to an already security-sensitive and fragile piece of code. The best method i found was to get rid of bprm-pages[] and to directly copy strings into the new mm via kmap (and to follow whatever RAM allocation policies/limits there are for the new mm), but that's quite ugly. Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
On Thursday 01 September 2005 08:57, Ingo Molnar wrote: the whole thing should be reworked, so that there is no artificial limit like MAX_ARG_PAGES. (it is after all just another piece of memory, in theory) Yes, a sysctl would probably lead to fragmentation problems and then people would do ugly linked lists of buffers like poll. If we do unconditional page-flipping then we fragment the argument space, if we do both page-flipping if things are unfragmented and well-aligned, and 'compact' the layout otherwise, we havent solved the problem and have introduced a significant extra layer of complexity to an already security-sensitive and fragile piece of code. Page flipping = COW like fork would do? Not sure how this would work - the arguments of execve can be anywhere in the address space and would presumably be often be in a inconvenient place like in the middle of the stack of the new executable. The best method i found was to get rid of bprm-pages[] and to directly copy strings into the new mm via kmap (and to follow whatever RAM allocation policies/limits there are for the new mm), but that's quite ugly. That sounds better. -Andi - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
* Andi Kleen [EMAIL PROTECTED] wrote: On Thursday 01 September 2005 08:57, Ingo Molnar wrote: the whole thing should be reworked, so that there is no artificial limit like MAX_ARG_PAGES. (it is after all just another piece of memory, in theory) Yes, a sysctl would probably lead to fragmentation problems and then people would do ugly linked lists of buffers like poll. not really fragmentation problems (the unit of allocation of argument pages is already a single page, and we do an array of pages), the real problem is the DoS - right now the array pages are unswappable while an exec() is ongoing. If we do unconditional page-flipping then we fragment the argument space, if we do both page-flipping if things are unfragmented and well-aligned, and 'compact' the layout otherwise, we havent solved the problem and have introduced a significant extra layer of complexity to an already security-sensitive and fragile piece of code. Page flipping = COW like fork would do? i dont think we need COW. During execve() we are destroying the old context and are creating a completely new context, so in theory we could just 'flip over' the argument/environment pages (which are a parameter to sys_execve()) from the old mm into the newly created mm, without caring about the old mm. Not sure how this would work - the arguments of execve can be anywhere in the address space and would presumably be often be in a inconvenient place like in the middle of the stack of the new executable. yes, that's one of the issues. I've done some instrumentation some time ago and it seemed that the arguments are typically page-aligned, so the only issue would be to clear the partial page at the end of the arguments. But i still think the concept is volatile. The best method i found was to get rid of bprm-pages[] and to directly copy strings into the new mm via kmap (and to follow whatever RAM allocation policies/limits there are for the new mm), but that's quite ugly. That sounds better. yeah. It's also pretty laborous though. Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
Ingo Molnar <[EMAIL PROTECTED]> writes: > > MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' > output is hardcoded. (because the kernel does not provide the > information dynamically) Perhaps it would be a good idea to make it a sysctl. Is there any reason it should be hardcoded? I cannot think of any. Ok if someone lowers the sysctl then execve has to handle the case of the args/environment possibly not fitting anymore, but that should be easy. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
On Wed, Aug 31, 2005 at 02:11:44PM +0200, Ingo Molnar wrote: > > I recompiled and installed the kernel, but there's no change (getconf > > ARG_MAX still gives 131072.) What am I missing? > > MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' > output is hardcoded. (because the kernel does not provide the > information dynamically) Yeah, you get the value of ARG_MAX from that was compiled in when you compiled glibc. Jakub - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
* Nick Matteo <[EMAIL PROTECTED]> wrote: > The other day I was running a grep on a big directory tree and got a > "Argument list too long" error. Since I'd like to have this work > without messing with find and xargs each time, I went into > include/linux/binfmts.h and changed > > #define MAX_ARG_PAGES 32 > > to > > #define MAX_ARG_PAGES 64 > > I recompiled and installed the kernel, but there's no change (getconf > ARG_MAX still gives 131072.) What am I missing? MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' output is hardcoded. (because the kernel does not provide the information dynamically) Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
* Nick Matteo [EMAIL PROTECTED] wrote: The other day I was running a grep on a big directory tree and got a Argument list too long error. Since I'd like to have this work without messing with find and xargs each time, I went into include/linux/binfmts.h and changed #define MAX_ARG_PAGES 32 to #define MAX_ARG_PAGES 64 I recompiled and installed the kernel, but there's no change (getconf ARG_MAX still gives 131072.) What am I missing? MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' output is hardcoded. (because the kernel does not provide the information dynamically) Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
On Wed, Aug 31, 2005 at 02:11:44PM +0200, Ingo Molnar wrote: I recompiled and installed the kernel, but there's no change (getconf ARG_MAX still gives 131072.) What am I missing? MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' output is hardcoded. (because the kernel does not provide the information dynamically) Yeah, you get the value of ARG_MAX from linux/limits.h that was compiled in when you compiled glibc. Jakub - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: MAX_ARG_PAGES has no effect?
Ingo Molnar [EMAIL PROTECTED] writes: MAX_ARG_PAGES should work just fine. I think the 'getconf ARG_MAX' output is hardcoded. (because the kernel does not provide the information dynamically) Perhaps it would be a good idea to make it a sysctl. Is there any reason it should be hardcoded? I cannot think of any. Ok if someone lowers the sysctl then execve has to handle the case of the args/environment possibly not fitting anymore, but that should be easy. -Andi - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
MAX_ARG_PAGES has no effect?
The other day I was running a grep on a big directory tree and got a "Argument list too long" error. Since I'd like to have this work without messing with find and xargs each time, I went into include/linux/binfmts.h and changed #define MAX_ARG_PAGES 32 to #define MAX_ARG_PAGES 64 I recompiled and installed the kernel, but there's no change (getconf ARG_MAX still gives 131072.) What am I missing? I am running 2.6.13 on amd64. Thanks, Nick Matteo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
MAX_ARG_PAGES has no effect?
The other day I was running a grep on a big directory tree and got a Argument list too long error. Since I'd like to have this work without messing with find and xargs each time, I went into include/linux/binfmts.h and changed #define MAX_ARG_PAGES 32 to #define MAX_ARG_PAGES 64 I recompiled and installed the kernel, but there's no change (getconf ARG_MAX still gives 131072.) What am I missing? I am running 2.6.13 on amd64. Thanks, Nick Matteo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
