Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-02-01 Thread Johannes Weiner
On Fri, Feb 01, 2019 at 11:27:41AM +0100, Michal Hocko wrote: > On Thu 31-01-19 11:22:48, Johannes Weiner wrote: > > On Thu, Jan 31, 2019 at 09:58:08AM +0100, Michal Hocko wrote: > > > On Wed 30-01-19 16:31:31, Johannes Weiner wrote: > > > > On Wed, Jan 30, 2019 at 09:05:59PM +0100, Michal Hocko

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-02-01 Thread Michal Hocko
On Thu 31-01-19 11:22:48, Johannes Weiner wrote: > On Thu, Jan 31, 2019 at 09:58:08AM +0100, Michal Hocko wrote: > > On Wed 30-01-19 16:31:31, Johannes Weiner wrote: > > > On Wed, Jan 30, 2019 at 09:05:59PM +0100, Michal Hocko wrote: > > [...] > > > > I thought I have already mentioned an example.

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-31 Thread Johannes Weiner
On Thu, Jan 31, 2019 at 09:58:08AM +0100, Michal Hocko wrote: > On Wed 30-01-19 16:31:31, Johannes Weiner wrote: > > On Wed, Jan 30, 2019 at 09:05:59PM +0100, Michal Hocko wrote: > [...] > > > I thought I have already mentioned an example. Say you have an observer > > > on the top of a delegated

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-31 Thread Michal Hocko
On Wed 30-01-19 16:31:31, Johannes Weiner wrote: > On Wed, Jan 30, 2019 at 09:05:59PM +0100, Michal Hocko wrote: [...] > > I thought I have already mentioned an example. Say you have an observer > > on the top of a delegated cgroup hierarchy and you setup limits (e.g. hard > > limit) on the root

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Johannes Weiner
On Wed, Jan 30, 2019 at 09:05:59PM +0100, Michal Hocko wrote: > On Wed 30-01-19 14:23:45, Johannes Weiner wrote: > > On Mon, Jan 28, 2019 at 01:51:51PM +0100, Michal Hocko wrote: > > > On Fri 25-01-19 10:28:08, Tejun Heo wrote: > > > > On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote:

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Michal Hocko
On Wed 30-01-19 14:23:45, Johannes Weiner wrote: > On Mon, Jan 28, 2019 at 01:51:51PM +0100, Michal Hocko wrote: > > On Fri 25-01-19 10:28:08, Tejun Heo wrote: > > > On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote: > > > > Please note that I understand that this might be confusing

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Shakeel Butt
On Wed, Jan 30, 2019 at 11:30 AM Johannes Weiner wrote: > > On Wed, Jan 30, 2019 at 02:27:12PM -0500, Johannes Weiner wrote: > > On Wed, Jan 30, 2019 at 11:11:44AM -0800, Shakeel Butt wrote: > > > Hi Tejun, > > > > > > On Wed, Jan 30, 2019 at 9:07 AM Tejun Heo wrote: > > > > > > > > Hello,

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Johannes Weiner
On Wed, Jan 30, 2019 at 02:27:12PM -0500, Johannes Weiner wrote: > On Wed, Jan 30, 2019 at 11:11:44AM -0800, Shakeel Butt wrote: > > Hi Tejun, > > > > On Wed, Jan 30, 2019 at 9:07 AM Tejun Heo wrote: > > > > > > Hello, Michal. > > > > > > On Wed, Jan 30, 2019 at 05:50:58PM +0100, Michal Hocko

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Johannes Weiner
On Wed, Jan 30, 2019 at 11:11:44AM -0800, Shakeel Butt wrote: > Hi Tejun, > > On Wed, Jan 30, 2019 at 9:07 AM Tejun Heo wrote: > > > > Hello, Michal. > > > > On Wed, Jan 30, 2019 at 05:50:58PM +0100, Michal Hocko wrote: > > > > Yeah, cgroup.events and .stat files as some of the local stats would

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Johannes Weiner
On Mon, Jan 28, 2019 at 01:51:51PM +0100, Michal Hocko wrote: > On Fri 25-01-19 10:28:08, Tejun Heo wrote: > > On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote: > > > Please note that I understand that this might be confusing with the rest > > > of the cgroup APIs but considering that

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Shakeel Butt
Hi Tejun, On Wed, Jan 30, 2019 at 9:07 AM Tejun Heo wrote: > > Hello, Michal. > > On Wed, Jan 30, 2019 at 05:50:58PM +0100, Michal Hocko wrote: > > > Yeah, cgroup.events and .stat files as some of the local stats would > > > be useful too, so if we don't flip memory.events we'll end up with sth

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Michal Hocko
On Wed 30-01-19 09:52:22, Tejun Heo wrote: > On Wed, Jan 30, 2019 at 06:41:17PM +0100, Michal Hocko wrote: > > But we are discussing the file name effectively. I do not see a long > > term maintenance burden. Confusing? Probably yes but that is were the > > Cost on user side. > > > documentation

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Tejun Heo
On Wed, Jan 30, 2019 at 06:41:17PM +0100, Michal Hocko wrote: > But we are discussing the file name effectively. I do not see a long > term maintenance burden. Confusing? Probably yes but that is were the Cost on user side. > documentation would be helpful. which is an a lot worse option with

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Michal Hocko
On Wed 30-01-19 09:06:58, Tejun Heo wrote: > Hello, Michal. > > On Wed, Jan 30, 2019 at 05:50:58PM +0100, Michal Hocko wrote: > > > Yeah, cgroup.events and .stat files as some of the local stats would > > > be useful too, so if we don't flip memory.events we'll end up with sth > > > like

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Tejun Heo
Hello, Michal. On Wed, Jan 30, 2019 at 05:50:58PM +0100, Michal Hocko wrote: > > Yeah, cgroup.events and .stat files as some of the local stats would > > be useful too, so if we don't flip memory.events we'll end up with sth > > like cgroup.events.local, memory.events.tree and memory.stats.local,

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-30 Thread Michal Hocko
On Tue 29-01-19 06:52:40, Tejun Heo wrote: > Hello, > > On Tue, Jan 29, 2019 at 03:43:06PM +0100, Michal Hocko wrote: > > All memcg events are represented non-hierarchical AFAICS > > memcg_memory_event() simply accounts at the level when it happens. Or do > > I miss something? Or are you talking

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-29 Thread Tejun Heo
Hello, On Tue, Jan 29, 2019 at 03:43:06PM +0100, Michal Hocko wrote: > All memcg events are represented non-hierarchical AFAICS > memcg_memory_event() simply accounts at the level when it happens. Or do > I miss something? Or are you talking about .events files for other > controllers? Yeah,

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-29 Thread Michal Hocko
On Mon 28-01-19 09:49:05, Tejun Heo wrote: > Hello, Michal. > > On Mon, Jan 28, 2019 at 06:05:26PM +0100, Michal Hocko wrote: > > Yeah, that is quite clear. But it also assumes that the hierarchy is > > pretty stable but cgroups might go away at any time. I am not saying > > that the aggregated

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, Michal. On Mon, Jan 28, 2019 at 06:05:26PM +0100, Michal Hocko wrote: > Yeah, that is quite clear. But it also assumes that the hierarchy is > pretty stable but cgroups might go away at any time. I am not saying > that the aggregated events are not useful I am just saying that it is >

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Shakeel Butt
Hi Tejun, On Mon, Jan 28, 2019 at 8:05 AM Tejun Heo wrote: > > Hello, Shakeel. > > On Mon, Jan 28, 2019 at 07:59:33AM -0800, Shakeel Butt wrote: > > Why not make this configurable at the delegation boundary? As you > > mentioned, there are jobs who want centralized workload manager to > > watch

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, On Mon, Jan 28, 2019 at 08:08:26AM -0800, Shakeel Butt wrote: > Do you envision a separate interface/file for recursive and local > counters? That would make notifications simpler but that is an > additional interface. I need to think more about it but my first throught is that a separate

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Michal Hocko
On Mon 28-01-19 07:41:50, Tejun Heo wrote: > Hello, Michal. > > On Mon, Jan 28, 2019 at 04:18:59PM +0100, Michal Hocko wrote: > > How do you make an atomic snapshot of the hierarchy state? Or you do > > not need it because event counters are monotonic and you are willing to > > sacrifice some

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, Shakeel. On Mon, Jan 28, 2019 at 07:59:33AM -0800, Shakeel Butt wrote: > Why not make this configurable at the delegation boundary? As you > mentioned, there are jobs who want centralized workload manager to > watch over their subtrees while there can be jobs which want to > monitor their

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Shakeel Butt
Hi Tejun, On Fri, Jan 25, 2019 at 10:28 AM Tejun Heo wrote: > > Hello, Michal. > > On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote: > > > What if a user wants to monitor any ooms in the subtree tho, which is > > > a valid use case? > > > > How is that information useful without know

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, Michal. On Mon, Jan 28, 2019 at 04:18:59PM +0100, Michal Hocko wrote: > How do you make an atomic snapshot of the hierarchy state? Or you do > not need it because event counters are monotonic and you are willing to > sacrifice some lost or misinterpreted events? For example, you receive >

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Michal Hocko
On Mon 28-01-19 06:54:07, Tejun Heo wrote: > Hello, > > On Mon, Jan 28, 2019 at 03:52:10PM +0100, Michal Hocko wrote: > > > All .events files generate aggregated stateful notifications. For > > > anyone to do anything, they'd have to remember the previous state to > > > identify what actually

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, On Mon, Jan 28, 2019 at 03:52:10PM +0100, Michal Hocko wrote: > > All .events files generate aggregated stateful notifications. For > > anyone to do anything, they'd have to remember the previous state to > > identify what actually happened. Being hierarchical, it'd of course > > need to

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Michal Hocko
On Mon 28-01-19 06:28:16, Tejun Heo wrote: > Hello, Michal. > > On Mon, Jan 28, 2019 at 01:51:51PM +0100, Michal Hocko wrote: > > > For example, a workload manager watching over a subtree for a job with > > > nested memory limits set by the job itself. It wants to take action > > > (reporting

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
On Wed, Jan 23, 2019 at 05:31:44PM -0500, Chris Down wrote: > memory.stat and other files already consider subtrees in their output, > and we should too in order to not present an inconsistent interface. > > The current situation is fairly confusing, because people interacting > with cgroups

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Tejun Heo
Hello, Michal. On Mon, Jan 28, 2019 at 01:51:51PM +0100, Michal Hocko wrote: > > For example, a workload manager watching over a subtree for a job with > > nested memory limits set by the job itself. It wants to take action > > (reporting and possibly other remediative actions) when something

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-28 Thread Michal Hocko
On Fri 25-01-19 10:28:08, Tejun Heo wrote: > Hello, Michal. > > On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote: > > > What if a user wants to monitor any ooms in the subtree tho, which is > > > a valid use case? > > > > How is that information useful without know which memcg the

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-25 Thread Tejun Heo
Hello, Michal. On Fri, Jan 25, 2019 at 06:37:13PM +0100, Michal Hocko wrote: > > What if a user wants to monitor any ooms in the subtree tho, which is > > a valid use case? > > How is that information useful without know which memcg the oom applies > to? For example, a workload manager watching

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-25 Thread Michal Hocko
On Fri 25-01-19 08:51:52, Tejun Heo wrote: [...] > > I do see your point about consistency. But it is also important to > > consider the usability of this interface. As already mentioned, catching > > an oom event at a level where the oom doesn't happen and having hard > > time to identify that

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-25 Thread Tejun Heo
Hello, Michal. On Fri, Jan 25, 2019 at 09:42:13AM +0100, Michal Hocko wrote: > > If you read my sentence again, I'm not talking about the kernel but > > the surrounding infrastructure that consumes this data. The risk is > > not dependent on the age of the interface age, but on its adoption. > >

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-25 Thread Michal Hocko
On Thu 24-01-19 13:23:28, Johannes Weiner wrote: > On Thu, Jan 24, 2019 at 06:01:17PM +0100, Michal Hocko wrote: > > On Thu 24-01-19 11:00:10, Johannes Weiner wrote: > > [...] > > > We cannot fully eliminate a risk for regression, but it strikes me as > > > highly unlikely, given the extremely

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Johannes Weiner
On Thu, Jan 24, 2019 at 06:01:17PM +0100, Michal Hocko wrote: > On Thu 24-01-19 11:00:10, Johannes Weiner wrote: > [...] > > We cannot fully eliminate a risk for regression, but it strikes me as > > highly unlikely, given the extremely young age of cgroup2-based system > > management and

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Michal Hocko
On Thu 24-01-19 11:00:10, Johannes Weiner wrote: [...] > We cannot fully eliminate a risk for regression, but it strikes me as > highly unlikely, given the extremely young age of cgroup2-based system > management and surrounding tooling. I am not really sure what you consider young but this

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Johannes Weiner
On Thu, Jan 24, 2019 at 09:22:52AM +0100, Michal Hocko wrote: > On Wed 23-01-19 17:31:44, Chris Down wrote: > > Acked-by: Johannes Weiner > > btw. I do not see this patch posted anywhere yet it already comes with > an ack. Have I just missed a previous version? I reviewed it offline before

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Michal Hocko
On Thu 24-01-19 07:21:22, Tejun Heo wrote: > Hello, Michal. > > On Thu, Jan 24, 2019 at 09:22:52AM +0100, Michal Hocko wrote: > > I do not think we can do that for two reasons. It breaks the existing > > semantic userspace might depend on and more importantly this is not a > > correct behavior

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Tejun Heo
Hello, Michal. On Thu, Jan 24, 2019 at 09:22:52AM +0100, Michal Hocko wrote: > I do not think we can do that for two reasons. It breaks the existing > semantic userspace might depend on and more importantly this is not a > correct behavior IMO. This is a valid concern but I'll come back to this

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-24 Thread Michal Hocko
On Wed 23-01-19 17:31:44, Chris Down wrote: > memory.stat and other files already consider subtrees in their output, > and we should too in order to not present an inconsistent interface. > > The current situation is fairly confusing, because people interacting > with cgroups expect hierarchical

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-23 Thread Chris Down
Roman Gushchin writes: On Wed, Jan 23, 2019 at 05:31:44PM -0500, Chris Down wrote: memory.stat and other files already consider subtrees in their output, and we should too in order to not present an inconsistent interface. The current situation is fairly confusing, because people interacting

Re: [PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-23 Thread Roman Gushchin
On Wed, Jan 23, 2019 at 05:31:44PM -0500, Chris Down wrote: > memory.stat and other files already consider subtrees in their output, > and we should too in order to not present an inconsistent interface. > > The current situation is fairly confusing, because people interacting > with cgroups

[PATCH 2/2] mm: Consider subtrees in memory.events

2019-01-23 Thread Chris Down
memory.stat and other files already consider subtrees in their output, and we should too in order to not present an inconsistent interface. The current situation is fairly confusing, because people interacting with cgroups expect hierarchical behaviour in the vein of memory.stat, cgroup.events,