Re: [v8 0/4] cgroup-aware OOM killer

On Mon 02-10-17 13:24:25, Shakeel Butt wrote: > On Mon, Oct 2, 2017 at 12:56 PM, Michal Hocko wrote: > > On Mon 02-10-17 12:45:18, Shakeel Butt wrote: > >> > I am sorry to cut the rest of your proposal because it simply goes over > >> > the scope of the proposed solution while

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Oct 02, 2017 at 01:24:25PM -0700, Shakeel Butt wrote: > On Mon, Oct 2, 2017 at 12:56 PM, Michal Hocko wrote: > > On Mon 02-10-17 12:45:18, Shakeel Butt wrote: > >> > I am sorry to cut the rest of your proposal because it simply goes over > >> > the scope of the proposed

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Oct 2, 2017 at 12:56 PM, Michal Hocko wrote: > On Mon 02-10-17 12:45:18, Shakeel Butt wrote: >> > I am sorry to cut the rest of your proposal because it simply goes over >> > the scope of the proposed solution while the usecase you are mentioning >> > is still possible.

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 02-10-17 13:00:54, Tim Hockin wrote: > In the example above: > >root >/\ > A D > / \ >B C > > Does oom_group allow me to express "compare A and D; if A is chosen > compare B and C; kill the loser" ? As I understand the proposal (from > reading

Re: [v8 0/4] cgroup-aware OOM killer

In the example above: root /\ A D / \ B C Does oom_group allow me to express "compare A and D; if A is chosen compare B and C; kill the loser" ? As I understand the proposal (from reading thread, not patch) it does not. On Mon, Oct 2, 2017 at 12:56 PM,

Re: [v8 0/4] cgroup-aware OOM killer

> I am sorry to cut the rest of your proposal because it simply goes over > the scope of the proposed solution while the usecase you are mentioning > is still possible. If we want to compare intermediate nodes (which seems > to be the case) then we can always provide a knob to opt-in - be it your

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 02-10-17 12:00:43, Shakeel Butt wrote: > > Yes and nobody is disputing that, really. I guess the main disconnect > > here is that different people want to have more detailed control over > > the victim selection while the patchset tries to handle the most > > simplistic scenario when a no

Re: [v8 0/4] cgroup-aware OOM killer

> Yes and nobody is disputing that, really. I guess the main disconnect > here is that different people want to have more detailed control over > the victim selection while the patchset tries to handle the most > simplistic scenario when a no userspace control over the selection is > required. And

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 02-10-17 13:47:12, Roman Gushchin wrote: > On Mon, Oct 02, 2017 at 02:24:34PM +0200, Michal Hocko wrote: [...] > > I believe the latest version (v9) looks sensible from the semantic point > > of view and we should focus on making it into a mergeable shape. > > The only thing is that after

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Oct 02, 2017 at 02:24:34PM +0200, Michal Hocko wrote: > On Sun 01-10-17 16:29:48, Shakeel Butt wrote: > > > > > > Going back to Michal's example, say the user configured the following: > > > > > >root > > > /\ > > > A D > > > / \ > > >B C > > > > > > A

Re: [v8 0/4] cgroup-aware OOM killer

On Sun 01-10-17 16:29:48, Shakeel Butt wrote: > > > > Going back to Michal's example, say the user configured the following: > > > >root > > /\ > > A D > > / \ > >B C > > > > A global OOM event happens and we find this: > > - A > D > > - B, C, D are oomgroups

Re: [v8 0/4] cgroup-aware OOM killer

Shakeel Butt wrote: > I think Tim has given very clear explanation why comparing A & D makes > perfect sense. However I think the above example, a single user system > where a user has designed and created the whole hierarchy and then > attaches different jobs/applications to different nodes in

Re: [v8 0/4] cgroup-aware OOM killer

> > Going back to Michal's example, say the user configured the following: > >root > /\ > A D > / \ >B C > > A global OOM event happens and we find this: > - A > D > - B, C, D are oomgroups > > What the user is telling us is that B, C, and D are compound

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 27, 2017 at 9:23 AM, Roman Gushchin wrote: > On Wed, Sep 27, 2017 at 08:35:50AM -0700, Tim Hockin wrote: >> On Wed, Sep 27, 2017 at 12:43 AM, Michal Hocko wrote: >> > On Tue 26-09-17 20:37:37, Tim Hockin wrote: >> > [...] >> >> I feel like David has

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 27, 2017 at 08:35:50AM -0700, Tim Hockin wrote: > On Wed, Sep 27, 2017 at 12:43 AM, Michal Hocko wrote: > > On Tue 26-09-17 20:37:37, Tim Hockin wrote: > > [...] > >> I feel like David has offered examples here, and many of us at Google > >> have offered examples as

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 27, 2017 at 12:43 AM, Michal Hocko wrote: > On Tue 26-09-17 20:37:37, Tim Hockin wrote: > [...] >> I feel like David has offered examples here, and many of us at Google >> have offered examples as long ago as 2013 (if I recall) of cases where >> the proposed

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 27, 2017 at 09:43:19AM +0200, Michal Hocko wrote: > On Tue 26-09-17 20:37:37, Tim Hockin wrote: > [...] > > I feel like David has offered examples here, and many of us at Google > > have offered examples as long ago as 2013 (if I recall) of cases where > > the proposed heuristic is

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 27, 2017 at 09:37:44AM +0200, Michal Hocko wrote: > On Tue 26-09-17 14:04:41, David Rientjes wrote: > > On Tue, 26 Sep 2017, Michal Hocko wrote: > > > > > > No, I agree that we shouldn't compare sibling memory cgroups based on > > > > different criteria depending on whether group_oom

Re: [v8 0/4] cgroup-aware OOM killer

On Tue 26-09-17 20:37:37, Tim Hockin wrote: [...] > I feel like David has offered examples here, and many of us at Google > have offered examples as long ago as 2013 (if I recall) of cases where > the proposed heuristic is EXACTLY WRONG. I do not think we have discussed anything resembling the

Re: [v8 0/4] cgroup-aware OOM killer

On Tue 26-09-17 14:04:41, David Rientjes wrote: > On Tue, 26 Sep 2017, Michal Hocko wrote: > > > > No, I agree that we shouldn't compare sibling memory cgroups based on > > > different criteria depending on whether group_oom is set or not. > > > > > > I think it would be better to compare

Re: [v8 0/4] cgroup-aware OOM killer

I'm excited to see this being discussed again - it's been years since the last attempt. I've tried to stay out of the conversation, but I feel obligated say something and then go back to lurking. On Tue, Sep 26, 2017 at 10:26 AM, Johannes Weiner wrote: > On Tue, Sep 26, 2017

Re: [v8 0/4] cgroup-aware OOM killer

On Tue, 26 Sep 2017, Michal Hocko wrote: > > No, I agree that we shouldn't compare sibling memory cgroups based on > > different criteria depending on whether group_oom is set or not. > > > > I think it would be better to compare siblings based on the same criteria > > independent of group_oom

Re: [v8 0/4] cgroup-aware OOM killer

On Tue, Sep 26, 2017 at 03:30:40PM +0200, Michal Hocko wrote: > On Tue 26-09-17 13:13:00, Roman Gushchin wrote: > > On Tue, Sep 26, 2017 at 01:21:34PM +0200, Michal Hocko wrote: > > > On Tue 26-09-17 11:59:25, Roman Gushchin wrote: > > > > On Mon, Sep 25, 2017 at 10:25:21PM +0200, Michal Hocko

Re: [v8 0/4] cgroup-aware OOM killer

On Tue 26-09-17 13:13:00, Roman Gushchin wrote: > On Tue, Sep 26, 2017 at 01:21:34PM +0200, Michal Hocko wrote: > > On Tue 26-09-17 11:59:25, Roman Gushchin wrote: > > > On Mon, Sep 25, 2017 at 10:25:21PM +0200, Michal Hocko wrote: > > > > On Mon 25-09-17 19:15:33, Roman Gushchin wrote: > > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Tue, Sep 26, 2017 at 01:21:34PM +0200, Michal Hocko wrote: > On Tue 26-09-17 11:59:25, Roman Gushchin wrote: > > On Mon, Sep 25, 2017 at 10:25:21PM +0200, Michal Hocko wrote: > > > On Mon 25-09-17 19:15:33, Roman Gushchin wrote: > > > [...] > > > > I'm not against this model, as I've said

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Sep 25, 2017 at 10:25:21PM +0200, Michal Hocko wrote: > On Mon 25-09-17 19:15:33, Roman Gushchin wrote: > [...] > > I'm not against this model, as I've said before. It feels logical, > > and will work fine in most cases. > > > > In this case we can drop any mount/boot options, because it

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 25-09-17 15:21:03, David Rientjes wrote: > On Mon, 25 Sep 2017, Johannes Weiner wrote: > > > > True but we want to have the semantic reasonably understandable. And it > > > is quite hard to explain that the oom killer hasn't selected the largest > > > memcg just because it happened to be

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, 25 Sep 2017, Johannes Weiner wrote: > > True but we want to have the semantic reasonably understandable. And it > > is quite hard to explain that the oom killer hasn't selected the largest > > memcg just because it happened to be in a deeper hierarchy which has > > been configured to

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 25-09-17 19:15:33, Roman Gushchin wrote: [...] > I'm not against this model, as I've said before. It feels logical, > and will work fine in most cases. > > In this case we can drop any mount/boot options, because it preserves > the existing behavior in the default configuration. A big

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Sep 25, 2017 at 02:24:00PM +0200, Michal Hocko wrote: > I would really appreciate some feedback from Tejun, Johannes here. > > On Wed 20-09-17 14:53:41, Roman Gushchin wrote: > > On Mon, Sep 18, 2017 at 08:14:05AM +0200, Michal Hocko wrote: > > > On Fri 15-09-17 08:23:01, Roman Gushchin

Re: [v8 0/4] cgroup-aware OOM killer

I would really appreciate some feedback from Tejun, Johannes here. On Wed 20-09-17 14:53:41, Roman Gushchin wrote: > On Mon, Sep 18, 2017 at 08:14:05AM +0200, Michal Hocko wrote: > > On Fri 15-09-17 08:23:01, Roman Gushchin wrote: > > > On Fri, Sep 15, 2017 at 12:58:26PM +0200, Michal Hocko

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, 22 Sep 2017, Tejun Heo wrote: > > If you have this low priority maintenance job charging memory to the high > > priority hierarchy, you're already misconfigured unless you adjust > > /proc/pid/oom_score_adj because it will oom kill any larger process than > > itself in today's kernels

Re: [v8 0/4] cgroup-aware OOM killer

Hello, On Fri, Sep 22, 2017 at 01:39:55PM -0700, David Rientjes wrote: > Current heuristic based on processes is coupled with per-process > /proc/pid/oom_score_adj. The proposed > heuristic has no ability to be influenced by userspace, and it needs one. > The proposed heuristic based on

Re: [v8 0/4] cgroup-aware OOM killer

On Thu, 21 Sep 2017, Johannes Weiner wrote: > > The issue is that if you opt-in to the new feature, then you are forced to > > change /proc/pid/oom_score_adj of all processes attached to a cgroup that > > you do not want oom killed based on size to be oom disabled. > > You're assuming that

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, 22 Sep 2017, Tejun Heo wrote: > > It doesn't have anything to do with my particular usecase, but rather the > > ability of userspace to influence the decisions of the kernel. Previous > > to this patchset, when selection is done based on process size, userspace > > has full control

Re: [v8 0/4] cgroup-aware OOM killer

Hello, David. On Thu, Sep 21, 2017 at 02:17:25PM -0700, David Rientjes wrote: > It doesn't have anything to do with my particular usecase, but rather the > ability of userspace to influence the decisions of the kernel. Previous > to this patchset, when selection is done based on process size,

Re: [v8 0/4] cgroup-aware OOM killer

On Thu, Sep 21, 2017 at 02:17:25PM -0700, David Rientjes wrote: > On Thu, 21 Sep 2017, Johannes Weiner wrote: > > > That's a ridiculous nak. > > > > The fact that this patch series doesn't solve your particular problem > > is not a technical argument to *reject* somebody else's work to solve > >

Re: [v8 0/4] cgroup-aware OOM killer

On Thu, 21 Sep 2017, Johannes Weiner wrote: > That's a ridiculous nak. > > The fact that this patch series doesn't solve your particular problem > is not a technical argument to *reject* somebody else's work to solve > a different problem. It's not a regression when behavior is completely >

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Sep 11, 2017 at 01:44:39PM -0700, David Rientjes wrote: > On Mon, 11 Sep 2017, Roman Gushchin wrote: > > > This patchset makes the OOM killer cgroup-aware. > > > > v8: > > - Do not kill tasks with OOM_SCORE_ADJ -1000 > > - Make the whole thing opt-in with cgroup mount option control

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, 18 Sep 2017, Roman Gushchin wrote: > > As said in other email. We can make priorities hierarchical (in the same > > sense as hard limit or others) so that children cannot override their > > parent. > > You mean they can set the knob to any value, but parent's value is enforced, > if it's

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, 20 Sep 2017, Roman Gushchin wrote: > > It's actually much more complex because in our environment we'd need an > > "activity manager" with CAP_SYS_RESOURCE to control oom priorities of user > > subcontainers when today it need only be concerned with top-level memory > > cgroups. Users

Re: [v8 0/4] cgroup-aware OOM killer

On Tue, Sep 19, 2017 at 01:54:48PM -0700, David Rientjes wrote: > On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > > > But then you just enforce a structural restriction on your > > > > > configuration > > > > > because > > > > > root > > > > > / \ > > > > >AD > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Sep 18, 2017 at 08:14:05AM +0200, Michal Hocko wrote: > On Fri 15-09-17 08:23:01, Roman Gushchin wrote: > > On Fri, Sep 15, 2017 at 12:58:26PM +0200, Michal Hocko wrote: > > > On Thu 14-09-17 09:05:48, Roman Gushchin wrote: > > > > On Thu, Sep 14, 2017 at 03:40:14PM +0200, Michal Hocko

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > But then you just enforce a structural restriction on your configuration > > > > because > > > > root > > > > / \ > > > >AD > > > > /\ > > > > B C > > > > > > > > is a different thing than > > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, Sep 18, 2017 at 08:20:45AM +0200, Michal Hocko wrote: > On Fri 15-09-17 14:08:07, Roman Gushchin wrote: > > On Fri, Sep 15, 2017 at 12:55:55PM -0700, David Rientjes wrote: > > > On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > > > > > But then you just enforce a structural restriction

Re: [v8 0/4] cgroup-aware OOM killer

On Fri 15-09-17 14:08:07, Roman Gushchin wrote: > On Fri, Sep 15, 2017 at 12:55:55PM -0700, David Rientjes wrote: > > On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > > > But then you just enforce a structural restriction on your configuration > > > > because > > > > root > > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Fri 15-09-17 12:55:55, David Rientjes wrote: > On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > But then you just enforce a structural restriction on your configuration > > > because > > > root > > > / \ > > >AD > > > /\ > > > B C > > > > > > is a

Re: [v8 0/4] cgroup-aware OOM killer

On Fri 15-09-17 08:23:01, Roman Gushchin wrote: > On Fri, Sep 15, 2017 at 12:58:26PM +0200, Michal Hocko wrote: > > On Thu 14-09-17 09:05:48, Roman Gushchin wrote: > > > On Thu, Sep 14, 2017 at 03:40:14PM +0200, Michal Hocko wrote: > > > > On Wed 13-09-17 14:56:07, Roman Gushchin wrote: > > > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, Sep 15, 2017 at 12:55:55PM -0700, David Rientjes wrote: > On Fri, 15 Sep 2017, Roman Gushchin wrote: > > > > But then you just enforce a structural restriction on your configuration > > > because > > > root > > > / \ > > >AD > > > /\ > > > B C > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, 15 Sep 2017, Roman Gushchin wrote: > > But then you just enforce a structural restriction on your configuration > > because > > root > > / \ > >AD > > /\ > > B C > > > > is a different thing than > > root > > / | \ > >B C D >

Re: [v8 0/4] cgroup-aware OOM killer

On Fri, Sep 15, 2017 at 12:58:26PM +0200, Michal Hocko wrote: > On Thu 14-09-17 09:05:48, Roman Gushchin wrote: > > On Thu, Sep 14, 2017 at 03:40:14PM +0200, Michal Hocko wrote: > > > On Wed 13-09-17 14:56:07, Roman Gushchin wrote: > > > > On Wed, Sep 13, 2017 at 02:29:14PM +0200, Michal Hocko

Re: [v8 0/4] cgroup-aware OOM killer

On Thu 14-09-17 09:05:48, Roman Gushchin wrote: > On Thu, Sep 14, 2017 at 03:40:14PM +0200, Michal Hocko wrote: > > On Wed 13-09-17 14:56:07, Roman Gushchin wrote: > > > On Wed, Sep 13, 2017 at 02:29:14PM +0200, Michal Hocko wrote: > > [...] > > > > I strongly believe that comparing only leaf

Re: [v8 0/4] cgroup-aware OOM killer

On Thu, 14 Sep 2017, Michal Hocko wrote: > > It is certainly possible to add oom priorities on top before it is merged, > > but I don't see why it isn't part of the patchset. > > Because the semantic of the priority for non-leaf memcgs is not fully > clear and I would rather have the core of

Re: [v8 0/4] cgroup-aware OOM killer

On Thu, Sep 14, 2017 at 03:40:14PM +0200, Michal Hocko wrote: > On Wed 13-09-17 14:56:07, Roman Gushchin wrote: > > On Wed, Sep 13, 2017 at 02:29:14PM +0200, Michal Hocko wrote: > [...] > > > I strongly believe that comparing only leaf memcgs > > > is more straightforward and it doesn't lead to

Re: [v8 0/4] cgroup-aware OOM killer

On Wed 13-09-17 14:56:07, Roman Gushchin wrote: > On Wed, Sep 13, 2017 at 02:29:14PM +0200, Michal Hocko wrote: [...] > > I strongly believe that comparing only leaf memcgs > > is more straightforward and it doesn't lead to unexpected results as > > mentioned before (kill a small memcg which is a

Re: [v8 0/4] cgroup-aware OOM killer

On Wed 13-09-17 13:46:08, David Rientjes wrote: > On Wed, 13 Sep 2017, Michal Hocko wrote: > > > > > This patchset makes the OOM killer cgroup-aware. > > > > > > > > v8: > > > > - Do not kill tasks with OOM_SCORE_ADJ -1000 > > > > - Make the whole thing opt-in with cgroup mount option

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, Sep 13, 2017 at 02:29:14PM +0200, Michal Hocko wrote: > On Mon 11-09-17 13:44:39, David Rientjes wrote: > > On Mon, 11 Sep 2017, Roman Gushchin wrote: > > > > > This patchset makes the OOM killer cgroup-aware. > > > > > > v8: > > > - Do not kill tasks with OOM_SCORE_ADJ -1000 > > > -

Re: [v8 0/4] cgroup-aware OOM killer

On Wed, 13 Sep 2017, Michal Hocko wrote: > > > This patchset makes the OOM killer cgroup-aware. > > > > > > v8: > > > - Do not kill tasks with OOM_SCORE_ADJ -1000 > > > - Make the whole thing opt-in with cgroup mount option control > > > - Drop oom_priority for further discussions > > > >

Re: [v8 0/4] cgroup-aware OOM killer

On Mon 11-09-17 13:44:39, David Rientjes wrote: > On Mon, 11 Sep 2017, Roman Gushchin wrote: > > > This patchset makes the OOM killer cgroup-aware. > > > > v8: > > - Do not kill tasks with OOM_SCORE_ADJ -1000 > > - Make the whole thing opt-in with cgroup mount option control > > - Drop

Re: [v8 0/4] cgroup-aware OOM killer

On Mon, 11 Sep 2017, Roman Gushchin wrote: > This patchset makes the OOM killer cgroup-aware. > > v8: > - Do not kill tasks with OOM_SCORE_ADJ -1000 > - Make the whole thing opt-in with cgroup mount option control > - Drop oom_priority for further discussions Nack, we specifically require

[v8 0/4] cgroup-aware OOM killer

This patchset makes the OOM killer cgroup-aware. v8: - Do not kill tasks with OOM_SCORE_ADJ -1000 - Make the whole thing opt-in with cgroup mount option control - Drop oom_priority for further discussions - Kill the whole cgroup if oom_group is set and it's memory.max is reached -