Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Tejun Heo
On Tue, Mar 03, 2015 at 12:31:19AM +1100, Aleksa Sarai wrote: > > If 16-bit PID's aren't a concern anymore, then why do we still default to > > treating it like a 16-bit signed int (the default for > > /proc/sys/kernel/pid_max is 32768)? > > I just want to emphasise that *even if* we changed to

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Tejun Heo
On Mon, Mar 02, 2015 at 08:13:23AM -0500, Austin S Hemmelgarn wrote: > If 16-bit PID's aren't a concern anymore, then why do we still default to > treating it like a 16-bit signed int (the default for > /proc/sys/kernel/pid_max is 32768)? Inertia. It has to start there for backward

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Aleksa Sarai
> If 16-bit PID's aren't a concern anymore, then why do we still default to > treating it like a 16-bit signed int (the default for > /proc/sys/kernel/pid_max is 32768)? I just want to emphasise that *even if* we changed to another default limit, the mere existence of a system-wide pid_max makes

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Austin S Hemmelgarn
On 2015-02-28 11:43, Tejun Heo wrote: Hello, Tim. On Sat, Feb 28, 2015 at 08:38:07AM -0800, Tim Hockin wrote: I know there is not much concern for legacy-system problems, but it is worth adding this case - there are systems that limit PIDs for other reasons, eg broken infrastructure that

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Tejun Heo
On Tue, Mar 03, 2015 at 12:31:19AM +1100, Aleksa Sarai wrote: If 16-bit PID's aren't a concern anymore, then why do we still default to treating it like a 16-bit signed int (the default for /proc/sys/kernel/pid_max is 32768)? I just want to emphasise that *even if* we changed to another

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Tejun Heo
On Mon, Mar 02, 2015 at 08:13:23AM -0500, Austin S Hemmelgarn wrote: If 16-bit PID's aren't a concern anymore, then why do we still default to treating it like a 16-bit signed int (the default for /proc/sys/kernel/pid_max is 32768)? Inertia. It has to start there for backward compatibility.

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Austin S Hemmelgarn
On 2015-02-28 11:43, Tejun Heo wrote: Hello, Tim. On Sat, Feb 28, 2015 at 08:38:07AM -0800, Tim Hockin wrote: I know there is not much concern for legacy-system problems, but it is worth adding this case - there are systems that limit PIDs for other reasons, eg broken infrastructure that

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-03-02 Thread Aleksa Sarai
If 16-bit PID's aren't a concern anymore, then why do we still default to treating it like a 16-bit signed int (the default for /proc/sys/kernel/pid_max is 32768)? I just want to emphasise that *even if* we changed to another default limit, the mere existence of a system-wide pid_max makes

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tim Hockin
On Feb 28, 2015 2:50 PM, "Tejun Heo" wrote: > > On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: > > Wow, so much anger. I'm not even sure how to respond, so I'll just > > say this and sign off. All I want is a better, friendlier, more > > useful system overall. We clearly have

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Johannes Weiner
On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: > On Sat, Feb 28, 2015 at 8:57 AM, Tejun Heo wrote: > > > > On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: > > > I am sorry that real-user problems are not perceived as substantial. This > > > was/is a real issue for us.

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: > Wow, so much anger. I'm not even sure how to respond, so I'll just > say this and sign off. All I want is a better, friendlier, more > useful system overall. We clearly have different ways of looking at > the problem. Can you

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tim Hockin
On Sat, Feb 28, 2015 at 8:57 AM, Tejun Heo wrote: > > On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: > > I am sorry that real-user problems are not perceived as substantial. This > > was/is a real issue for us. Being in limbo for years on end might not be a > > technical point, but

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: > I am sorry that real-user problems are not perceived as substantial. This > was/is a real issue for us. Being in limbo for years on end might not be a > technical point, but I do think it matters, and that was my point. It's a

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
Hello, Tim. On Sat, Feb 28, 2015 at 08:38:07AM -0800, Tim Hockin wrote: > I know there is not much concern for legacy-system problems, but it is > worth adding this case - there are systems that limit PIDs for other > reasons, eg broken infrastructure that assumes PIDs fit in a short int, >

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
Hello, Aleksa. On Sat, Feb 28, 2015 at 08:26:34PM +1100, Aleksa Sarai wrote: > I just want to quickly echo my support for this statement. Process IDs > aren't limited by kernel memory, they're a hard-set limit. Thus they are Process IDs become a hard global resource because we didn't switch to

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Aleksa Sarai
> I wouldn't think that preventing PID exhaustion would be all that much of a > niche case, it's fully possible for it to happen without using excessive > amounts of kernel memory (think about BIG server systems with terabytes of > memory running (arguably poorly written) forking servers that

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Aleksa Sarai
I wouldn't think that preventing PID exhaustion would be all that much of a niche case, it's fully possible for it to happen without using excessive amounts of kernel memory (think about BIG server systems with terabytes of memory running (arguably poorly written) forking servers that handle

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
Hello, Aleksa. On Sat, Feb 28, 2015 at 08:26:34PM +1100, Aleksa Sarai wrote: I just want to quickly echo my support for this statement. Process IDs aren't limited by kernel memory, they're a hard-set limit. Thus they are Process IDs become a hard global resource because we didn't switch to

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tim Hockin
On Sat, Feb 28, 2015 at 8:57 AM, Tejun Heo t...@kernel.org wrote: On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: I am sorry that real-user problems are not perceived as substantial. This was/is a real issue for us. Being in limbo for years on end might not be a technical

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: Wow, so much anger. I'm not even sure how to respond, so I'll just say this and sign off. All I want is a better, friendlier, more useful system overall. We clearly have different ways of looking at the problem. Can you

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Johannes Weiner
On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: On Sat, Feb 28, 2015 at 8:57 AM, Tejun Heo t...@kernel.org wrote: On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: I am sorry that real-user problems are not perceived as substantial. This was/is a real issue for

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
On Sat, Feb 28, 2015 at 08:48:12AM -0800, Tim Hockin wrote: I am sorry that real-user problems are not perceived as substantial. This was/is a real issue for us. Being in limbo for years on end might not be a technical point, but I do think it matters, and that was my point. It's a problem

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tejun Heo
Hello, Tim. On Sat, Feb 28, 2015 at 08:38:07AM -0800, Tim Hockin wrote: I know there is not much concern for legacy-system problems, but it is worth adding this case - there are systems that limit PIDs for other reasons, eg broken infrastructure that assumes PIDs fit in a short int,

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-28 Thread Tim Hockin
On Feb 28, 2015 2:50 PM, Tejun Heo t...@kernel.org wrote: On Sat, Feb 28, 2015 at 02:26:58PM -0800, Tim Hockin wrote: Wow, so much anger. I'm not even sure how to respond, so I'll just say this and sign off. All I want is a better, friendlier, more useful system overall. We clearly have

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 01:45:09PM -0800, Tim Hockin wrote: > Are you willing to put a drop-dead date on it? If we don't have > kmemcg working well enough to _actually_ bound PID usage and FD usage > by, say, June 1st, will you then accept a patch to this effect? If > the answer is no, then I

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 9:45 AM, Tejun Heo wrote: > On Fri, Feb 27, 2015 at 09:25:10AM -0800, Tim Hockin wrote: >> > In general, I'm pretty strongly against adding controllers for things >> > which aren't fundamental resources in the system. What's next? Open >> > files? Pipe buffer? Number

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, Austin. On Fri, Feb 27, 2015 at 01:49:53PM -0500, Austin S Hemmelgarn wrote: > As far as being trivial to achieve, I'm assuming you are referring to rlimit > and PAM's limits module, both of which have their own issues. Using > pam_limits.so to limit processes isn't trivial because it

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Austin S Hemmelgarn
On 2015-02-27 12:06, Tejun Heo wrote: Hello, On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: Kernel memory consumption isn't the only valid reason to want to limit the number of processes in a cgroup. Limiting the number of processes is very useful to ensure that a

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 12:45:03PM -0500, Tejun Heo wrote: > If your complaint is that this is taking too long, I hear you, and > there's a certain amount of validity in arguing that upstreaming a > temporary measure is the better trade-off, but the rationale for nproc > (or nfds, or virtual

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 09:25:10AM -0800, Tim Hockin wrote: > > In general, I'm pretty strongly against adding controllers for things > > which aren't fundamental resources in the system. What's next? Open > > files? Pipe buffer? Number of flocks? Number of session leaders or > > program

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 9:06 AM, Tejun Heo wrote: > Hello, > > On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: >> Kernel memory consumption isn't the only valid reason to want to limit the >> number of processes in a cgroup. Limiting the number of processes is very >> useful

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 09:12:45AM -0800, Tim Hockin wrote: > I was told that the plan was to use kmemcg - but I was told that YEARS > AGO. In the mean time we all either do our own thing or we do nothing > and suffer. Wasn't it like a year ago? Yeah, it's taking longer than everybody hoped but

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 8:42 AM, Austin S Hemmelgarn wrote: > On 2015-02-27 06:49, Tejun Heo wrote: >> >> Hello, >> >> On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: >>> >>> The current state of resource limitation for the number of open >>> processes (as well as the number of open

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: > Kernel memory consumption isn't the only valid reason to want to limit the > number of processes in a cgroup. Limiting the number of processes is very > useful to ensure that a program is working correctly (for

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Austin S Hemmelgarn
On 2015-02-27 06:49, Tejun Heo wrote: Hello, On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: The current state of resource limitation for the number of open processes (as well as the number of open file descriptors) requires you to use setrlimit(2), which means that you are

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Fri, Feb 27, 2015 at 02:46:13PM +0100, Richard Weinberger wrote: > just to make sure that I understand the big picture. > The plan is to limit kernel memory per cgroup such that fork bombs and > stuff cannot harm other groups of processes? Yes, the kmem part of memcg hasn't really been

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Richard Weinberger
Tejun, Am 27.02.2015 um 12:49 schrieb Tejun Heo: > This isn't a proper resource to control. kmemcg just grew proper > reclaim support and will be useable to control kernel side of memory > consumption. just to make sure that I understand the big picture. The plan is to limit kernel memory per

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: > The current state of resource limitation for the number of open > processes (as well as the number of open file descriptors) requires you > to use setrlimit(2), which means that you are limited to resource > limiting process

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Austin S Hemmelgarn
On 2015-02-27 06:49, Tejun Heo wrote: Hello, On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: The current state of resource limitation for the number of open processes (as well as the number of open file descriptors) requires you to use setrlimit(2), which means that you are

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 8:42 AM, Austin S Hemmelgarn ahferro...@gmail.com wrote: On 2015-02-27 06:49, Tejun Heo wrote: Hello, On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: The current state of resource limitation for the number of open processes (as well as the number of

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 09:12:45AM -0800, Tim Hockin wrote: I was told that the plan was to use kmemcg - but I was told that YEARS AGO. In the mean time we all either do our own thing or we do nothing and suffer. Wasn't it like a year ago? Yeah, it's taking longer than everybody hoped but

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 9:06 AM, Tejun Heo t...@kernel.org wrote: Hello, On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: Kernel memory consumption isn't the only valid reason to want to limit the number of processes in a cgroup. Limiting the number of processes is very

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: Kernel memory consumption isn't the only valid reason to want to limit the number of processes in a cgroup. Limiting the number of processes is very useful to ensure that a program is working correctly (for example,

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 09:25:10AM -0800, Tim Hockin wrote: In general, I'm pretty strongly against adding controllers for things which aren't fundamental resources in the system. What's next? Open files? Pipe buffer? Number of flocks? Number of session leaders or program groups?

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Fri, Feb 27, 2015 at 02:46:13PM +0100, Richard Weinberger wrote: just to make sure that I understand the big picture. The plan is to limit kernel memory per cgroup such that fork bombs and stuff cannot harm other groups of processes? Yes, the kmem part of memcg hasn't really been

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Richard Weinberger
Tejun, Am 27.02.2015 um 12:49 schrieb Tejun Heo: This isn't a proper resource to control. kmemcg just grew proper reclaim support and will be useable to control kernel side of memory consumption. just to make sure that I understand the big picture. The plan is to limit kernel memory per

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Austin S Hemmelgarn
On 2015-02-27 12:06, Tejun Heo wrote: Hello, On Fri, Feb 27, 2015 at 11:42:10AM -0500, Austin S Hemmelgarn wrote: Kernel memory consumption isn't the only valid reason to want to limit the number of processes in a cgroup. Limiting the number of processes is very useful to ensure that a

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 12:45:03PM -0500, Tejun Heo wrote: If your complaint is that this is taking too long, I hear you, and there's a certain amount of validity in arguing that upstreaming a temporary measure is the better trade-off, but the rationale for nproc (or nfds, or virtual memory,

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, Austin. On Fri, Feb 27, 2015 at 01:49:53PM -0500, Austin S Hemmelgarn wrote: As far as being trivial to achieve, I'm assuming you are referring to rlimit and PAM's limits module, both of which have their own issues. Using pam_limits.so to limit processes isn't trivial because it

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tim Hockin
On Fri, Feb 27, 2015 at 9:45 AM, Tejun Heo t...@kernel.org wrote: On Fri, Feb 27, 2015 at 09:25:10AM -0800, Tim Hockin wrote: In general, I'm pretty strongly against adding controllers for things which aren't fundamental resources in the system. What's next? Open files? Pipe buffer?

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
On Fri, Feb 27, 2015 at 01:45:09PM -0800, Tim Hockin wrote: Are you willing to put a drop-dead date on it? If we don't have kmemcg working well enough to _actually_ bound PID usage and FD usage by, say, June 1st, will you then accept a patch to this effect? If the answer is no, then I have

Re: [PATCH RFC 0/2] add nproc cgroup subsystem

2015-02-27 Thread Tejun Heo
Hello, On Mon, Feb 23, 2015 at 02:08:09PM +1100, Aleksa Sarai wrote: The current state of resource limitation for the number of open processes (as well as the number of open file descriptors) requires you to use setrlimit(2), which means that you are limited to resource limiting process trees