[RFC] Per file OOM-badness / RSS once more

Hello everyone, To summarize the issue I'm trying to address here: Processes can allocate resources through a file descriptor without being held responsible for it. I'm not explaining all the details again. See here for a more deeply description of the problem:

Re: [RFC] Per file OOM badness

On 2018-04-04 11:36 AM, Lucas Stach wrote: > Am Mittwoch, den 04.04.2018, 11:09 +0200 schrieb Michel Dänzer: >> On 2018-03-26 04:36 PM, Lucas Stach wrote: >>> Am Dienstag, den 30.01.2018, 11:28 +0100 schrieb Michal Hocko: On Tue 30-01-18 10:29:10, Michel Dänzer wrote: > On 2018-01-24

Re: [RFC] Per file OOM badness

Am Mittwoch, den 04.04.2018, 11:09 +0200 schrieb Michel Dänzer: > On 2018-03-26 04:36 PM, Lucas Stach wrote: > > Am Dienstag, den 30.01.2018, 11:28 +0100 schrieb Michal Hocko: > > > On Tue 30-01-18 10:29:10, Michel Dänzer wrote: > > > > On 2018-01-24 12:50 PM, Michal Hocko wrote: > > > > > On Wed

Re: [RFC] Per file OOM badness

On 2018-03-26 04:36 PM, Lucas Stach wrote: > Am Dienstag, den 30.01.2018, 11:28 +0100 schrieb Michal Hocko: >> On Tue 30-01-18 10:29:10, Michel Dänzer wrote: >>> On 2018-01-24 12:50 PM, Michal Hocko wrote: On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > On 2018-01-24 12:01 PM, Michal

Re: [RFC] Per file OOM badness

Hi all, Am Dienstag, den 30.01.2018, 11:28 +0100 schrieb Michal Hocko: > On Tue 30-01-18 10:29:10, Michel Dänzer wrote: > > On 2018-01-24 12:50 PM, Michal Hocko wrote: > > > On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > > > > On 2018-01-24 12:01 PM, Michal Hocko wrote: > > > > > On Wed

Re: [RFC] Per file OOM badness

On 2018-01-30 12:56 PM, Christian König wrote: > Am 30.01.2018 um 12:42 schrieb Michel Dänzer: >> On 2018-01-30 12:36 PM, Nicolai Hähnle wrote: >>> On 30.01.2018 12:34, Michel Dänzer wrote: On 2018-01-30 12:28 PM, Christian König wrote: > Am 30.01.2018 um 12:02 schrieb Michel Dänzer:

Re: [RFC] Per file OOM badness

Am 30.01.2018 um 12:42 schrieb Michel Dänzer: On 2018-01-30 12:36 PM, Nicolai Hähnle wrote: On 30.01.2018 12:34, Michel Dänzer wrote: On 2018-01-30 12:28 PM, Christian König wrote: Am 30.01.2018 um 12:02 schrieb Michel Dänzer: On 2018-01-30 11:40 AM, Christian König wrote: Am 30.01.2018 um

Re: [RFC] Per file OOM badness

On 2018-01-30 12:36 PM, Nicolai Hähnle wrote: > On 30.01.2018 12:34, Michel Dänzer wrote: >> On 2018-01-30 12:28 PM, Christian König wrote: >>> Am 30.01.2018 um 12:02 schrieb Michel Dänzer: On 2018-01-30 11:40 AM, Christian König wrote: > Am 30.01.2018 um 10:43 schrieb Michel Dänzer:

Re: [RFC] Per file OOM badness

On 30.01.2018 11:48, Michel Dänzer wrote: On 2018-01-30 11:42 AM, Daniel Vetter wrote: On Tue, Jan 30, 2018 at 10:43:10AM +0100, Michel Dänzer wrote: On 2018-01-30 10:31 AM, Daniel Vetter wrote: I guess a good first order approximation would be if we simply charge any newly allocated buffers

Re: [RFC] Per file OOM badness

On 2018-01-30 12:28 PM, Christian König wrote: > Am 30.01.2018 um 12:02 schrieb Michel Dänzer: >> On 2018-01-30 11:40 AM, Christian König wrote: >>> Am 30.01.2018 um 10:43 schrieb Michel Dänzer: [SNIP] > Would it be ok to hang onto potentially arbitrary mmget references > essentially

Re: [RFC] Per file OOM badness

Am 30.01.2018 um 12:02 schrieb Michel Dänzer: On 2018-01-30 11:40 AM, Christian König wrote: Am 30.01.2018 um 10:43 schrieb Michel Dänzer: [SNIP] Would it be ok to hang onto potentially arbitrary mmget references essentially forever? If that's ok I think we can do your process based account

Re: [RFC] Per file OOM badness

On 2018-01-30 11:40 AM, Christian König wrote: > Am 30.01.2018 um 10:43 schrieb Michel Dänzer: >> [SNIP] >>> Would it be ok to hang onto potentially arbitrary mmget references >>> essentially forever? If that's ok I think we can do your process based >>> account (minus a few minor inaccuracies for

Re: [RFC] Per file OOM badness

On 2018-01-30 11:42 AM, Daniel Vetter wrote: > On Tue, Jan 30, 2018 at 10:43:10AM +0100, Michel Dänzer wrote: >> On 2018-01-30 10:31 AM, Daniel Vetter wrote: >> >>> I guess a good first order approximation would be if we simply charge any >>> newly allocated buffers to the process that created

Re: [RFC] Per file OOM badness

On Tue, Jan 30, 2018 at 10:43:10AM +0100, Michel Dänzer wrote: > On 2018-01-30 10:31 AM, Daniel Vetter wrote: > > On Wed, Jan 24, 2018 at 01:11:09PM +0100, Christian König wrote: > >> Am 24.01.2018 um 12:50 schrieb Michal Hocko: > >>> On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > On

Re: [RFC] Per file OOM badness

Am 30.01.2018 um 10:43 schrieb Michel Dänzer: [SNIP] Would it be ok to hang onto potentially arbitrary mmget references essentially forever? If that's ok I think we can do your process based account (minus a few minor inaccuracies for shared stuff perhaps, but no one cares about that).

Re: [RFC] Per file OOM badness

On Tue 30-01-18 10:29:10, Michel Dänzer wrote: > On 2018-01-24 12:50 PM, Michal Hocko wrote: > > On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > >> On 2018-01-24 12:01 PM, Michal Hocko wrote: > >>> On Wed 24-01-18 11:27:15, Michel Dänzer wrote: > > [...] > 2. If the OOM killer kills a

Re: [RFC] Per file OOM badness

On Wed, Jan 24, 2018 at 01:11:09PM +0100, Christian König wrote: > Am 24.01.2018 um 12:50 schrieb Michal Hocko: > > On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > > > On 2018-01-24 12:01 PM, Michal Hocko wrote: > > > > On Wed 24-01-18 11:27:15, Michel Dänzer wrote: > > [...] > > > > > 2. If the

Re: [RFC] Per file OOM badness

On 2018-01-24 12:50 PM, Michal Hocko wrote: > On Wed 24-01-18 12:23:10, Michel Dänzer wrote: >> On 2018-01-24 12:01 PM, Michal Hocko wrote: >>> On Wed 24-01-18 11:27:15, Michel Dänzer wrote: > [...] 2. If the OOM killer kills a process which is sharing BOs with another process, this

Re: [RFC] Per file OOM badness

On Wed 24-01-18 11:27:15, Michel Dänzer wrote: > On 2018-01-24 10:28 AM, Michal Hocko wrote: [...] > > So how exactly then helps to kill one of those processes? The memory > > stays pinned behind or do I still misunderstand? > > Fundamentally, the memory is only released once all references to

Re: [RFC] Per file OOM badness

On 2018-01-24 12:50 PM, Michal Hocko wrote: > On Wed 24-01-18 12:23:10, Michel Dänzer wrote: >> On 2018-01-24 12:01 PM, Michal Hocko wrote: >>> On Wed 24-01-18 11:27:15, Michel Dänzer wrote: > [...] 2. If the OOM killer kills a process which is sharing BOs with another process, this

Re: [RFC] Per file OOM badness

Am 24.01.2018 um 12:50 schrieb Michal Hocko: On Wed 24-01-18 12:23:10, Michel Dänzer wrote: On 2018-01-24 12:01 PM, Michal Hocko wrote: On Wed 24-01-18 11:27:15, Michel Dänzer wrote: [...] 2. If the OOM killer kills a process which is sharing BOs with another process, this should result in

Re: [RFC] Per file OOM badness

On Wed 24-01-18 12:23:10, Michel Dänzer wrote: > On 2018-01-24 12:01 PM, Michal Hocko wrote: > > On Wed 24-01-18 11:27:15, Michel Dänzer wrote: [...] > >> 2. If the OOM killer kills a process which is sharing BOs with another > >> process, this should result in the other process dropping its

Re: [RFC] Per file OOM badness

On 2018-01-24 12:01 PM, Michal Hocko wrote: > On Wed 24-01-18 11:27:15, Michel Dänzer wrote: >> On 2018-01-24 10:28 AM, Michal Hocko wrote: > [...] >>> So how exactly then helps to kill one of those processes? The memory >>> stays pinned behind or do I still misunderstand? >> >> Fundamentally, the

Re: [RFC] Per file OOM badness

On 2018-01-24 10:28 AM, Michal Hocko wrote: > On Tue 23-01-18 17:39:19, Michel Dänzer wrote: >> On 2018-01-23 04:36 PM, Michal Hocko wrote: >>> On Tue 23-01-18 15:27:00, Roman Gushchin wrote: On Thu, Jan 18, 2018 at 06:00:06PM +0100, Michal Hocko wrote: > On Thu 18-01-18 11:47:48, Andrey

Re: [RFC] Per file OOM badness

On Tue 23-01-18 17:39:19, Michel Dänzer wrote: > On 2018-01-23 04:36 PM, Michal Hocko wrote: > > On Tue 23-01-18 15:27:00, Roman Gushchin wrote: > >> On Thu, Jan 18, 2018 at 06:00:06PM +0100, Michal Hocko wrote: > >>> On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > Hi, this series is a

Re: [RFC] Per file OOM badness

On 2018-01-23 04:36 PM, Michal Hocko wrote: > On Tue 23-01-18 15:27:00, Roman Gushchin wrote: >> On Thu, Jan 18, 2018 at 06:00:06PM +0100, Michal Hocko wrote: >>> On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: Hi, this series is a revised version of an RFC sent by Christian König a

Re: [RFC] Per file OOM badness

On Thu, Jan 18, 2018 at 06:00:06PM +0100, Michal Hocko wrote: > On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > > Hi, this series is a revised version of an RFC sent by Christian König > > a few years ago. The original RFC can be found at > >

Re: [RFC] Per file OOM badness

On Tue 23-01-18 15:27:00, Roman Gushchin wrote: > On Thu, Jan 18, 2018 at 06:00:06PM +0100, Michal Hocko wrote: > > On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > > > Hi, this series is a revised version of an RFC sent by Christian König > > > a few years ago. The original RFC can be found

Re: [RFC] Per file OOM badness

On Fri 19-01-18 17:54:36, Christian König wrote: > Am 19.01.2018 um 13:20 schrieb Michal Hocko: > > On Fri 19-01-18 13:13:51, Michal Hocko wrote: > > > On Fri 19-01-18 12:37:51, Christian König wrote: > > > [...] > > > > The per file descriptor badness is/was just the much easier approach to > > >

Re: [RFC] Per file OOM badness

On Thu, 18 Jan 2018 11:47:48 -0500 Andrey Grodzovsky wrote: > Hi, this series is a revised version of an RFC sent by Christian König > a few years ago. The original RFC can be found at > https://lists.freedesktop.org/archives/dri-devel/2015-September/089778.html > >

Re: [RFC] Per file OOM badness

On 01/22/2018 06:23 PM, Andrew Morton wrote: On Thu, 18 Jan 2018 11:47:48 -0500 Andrey Grodzovsky wrote: Hi, this series is a revised version of an RFC sent by Christian König a few years ago. The original RFC can be found at

Re: [RFC] Per file OOM badness

Michel Dänzer writes: > On 2018-01-19 11:02 AM, Michel Dänzer wrote: >> On 2018-01-19 10:58 AM, Christian König wrote: >>> Am 19.01.2018 um 10:32 schrieb Michel Dänzer: On 2018-01-19 09:39 AM, Christian König wrote: > Am 19.01.2018 um 09:20 schrieb Michal Hocko:

Re: [RFC] Per file OOM badness

Am 19.01.2018 um 13:20 schrieb Michal Hocko: On Fri 19-01-18 13:13:51, Michal Hocko wrote: On Fri 19-01-18 12:37:51, Christian König wrote: [...] The per file descriptor badness is/was just the much easier approach to solve the issue, because the drivers already knew which client is currently

Re: [RFC] Per file OOM badness

On 2018-01-19 12:37 PM, Christian König wrote: > Am 19.01.2018 um 11:40 schrieb Michal Hocko: >> On Fri 19-01-18 09:39:03, Christian König wrote: >>> Am 19.01.2018 um 09:20 schrieb Michal Hocko: >> [...] OK, in that case I would propose a different approach. We already have rss_stat. So

Re: [RFC] Per file OOM badness

On Fri 19-01-18 13:13:51, Michal Hocko wrote: > On Fri 19-01-18 12:37:51, Christian König wrote: > [...] > > The per file descriptor badness is/was just the much easier approach to > > solve the issue, because the drivers already knew which client is currently > > using which buffer objects. > >

Re: [RFC] Per file OOM badness

On Fri 19-01-18 09:39:03, Christian König wrote: > Am 19.01.2018 um 09:20 schrieb Michal Hocko: [...] > > OK, in that case I would propose a different approach. We already > > have rss_stat. So why do not we simply add a new counter there > > MM_KERNELPAGES and consider those in oom_badness? The

Re: [RFC] Per file OOM badness

On Fri 19-01-18 12:37:51, Christian König wrote: [...] > The per file descriptor badness is/was just the much easier approach to > solve the issue, because the drivers already knew which client is currently > using which buffer objects. > > I of course agree that file descriptors can be shared

Re: [RFC] Per file OOM badness

On 2018-01-19 11:02 AM, Michel Dänzer wrote: > On 2018-01-19 10:58 AM, Christian König wrote: >> Am 19.01.2018 um 10:32 schrieb Michel Dänzer: >>> On 2018-01-19 09:39 AM, Christian König wrote: Am 19.01.2018 um 09:20 schrieb Michal Hocko: > OK, in that case I would propose a different

Re: [RFC] Per file OOM badness

Am 19.01.2018 um 11:40 schrieb Michal Hocko: On Fri 19-01-18 09:39:03, Christian König wrote: Am 19.01.2018 um 09:20 schrieb Michal Hocko: [...] OK, in that case I would propose a different approach. We already have rss_stat. So why do not we simply add a new counter there MM_KERNELPAGES and

Re: [RFC] Per file OOM badness

On 2018-01-19 10:58 AM, Christian König wrote: > Am 19.01.2018 um 10:32 schrieb Michel Dänzer: >> On 2018-01-19 09:39 AM, Christian König wrote: >>> Am 19.01.2018 um 09:20 schrieb Michal Hocko: On Thu 18-01-18 12:01:32, Eric Anholt wrote: > Michal Hocko writes: >

Re: [RFC] Per file OOM badness

On 2018年01月19日 16:25, Michal Hocko wrote: [removed the broken quoting - please try to use an email client which doesn't mess up the qouted text] On Fri 19-01-18 06:01:26, He, Roger wrote: [...] I think you are misunderstanding here. Actually for now, the memory in TTM Pools already has

Re: [RFC] Per file OOM badness

Am 19.01.2018 um 10:32 schrieb Michel Dänzer: On 2018-01-19 09:39 AM, Christian König wrote: Am 19.01.2018 um 09:20 schrieb Michal Hocko: On Thu 18-01-18 12:01:32, Eric Anholt wrote: Michal Hocko writes: On Thu 18-01-18 18:00:06, Michal Hocko wrote: On Thu 18-01-18

Re: [RFC] Per file OOM badness

On 2018-01-19 09:39 AM, Christian König wrote: > Am 19.01.2018 um 09:20 schrieb Michal Hocko: >> On Thu 18-01-18 12:01:32, Eric Anholt wrote: >>> Michal Hocko writes: >>> On Thu 18-01-18 18:00:06, Michal Hocko wrote: > On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote:

Re: [RFC] Per file OOM badness

On Thu 18-01-18 12:01:32, Eric Anholt wrote: > Michal Hocko writes: > > > On Thu 18-01-18 18:00:06, Michal Hocko wrote: > >> On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > >> > Hi, this series is a revised version of an RFC sent by Christian König > >> > a few years ago.

Re: [RFC] Per file OOM badness

[removed the broken quoting - please try to use an email client which doesn't mess up the qouted text] On Fri 19-01-18 06:01:26, He, Roger wrote: [...] > I think you are misunderstanding here. > Actually for now, the memory in TTM Pools already has mm_shrink which is > implemented in

Re: [RFC] Per file OOM badness

Am 19.01.2018 um 09:20 schrieb Michal Hocko: On Thu 18-01-18 12:01:32, Eric Anholt wrote: Michal Hocko writes: On Thu 18-01-18 18:00:06, Michal Hocko wrote: On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: Hi, this series is a revised version of an RFC sent by

Re: [RFC] Per file OOM badness

Am 18.01.2018 um 21:01 schrieb Eric Anholt: Michal Hocko writes: [SNIP] But files are not killable, they can be shared... In other words this doesn't help the oom killer to make an educated guess at all. Maybe some more context would help the discussion? Thanks for doing

Re: [RFC] Per file OOM badness

Friday, January 19, 2018 12:48 AM To: linux-ker...@vger.kernel.org; linux...@kvack.org; dri-de...@lists.freedesktop.org; amd-gfx@lists.freedesktop.org Cc: Koenig, Christian <christian.koe...@amd.com> Subject: [RFC] Per file OOM badness Hi, this series is a revised version of an RFC sent

RE: [RFC] Per file OOM badness

g; dri-de...@lists.freedesktop.org; Koenig, Christian <christian.koe...@amd.com> Subject: Re: [RFC] Per file OOM badness On Thu 18-01-18 18:00:06, Michal Hocko wrote: > On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > > Hi, this series is a revised version of an RFC sent by Chris

RE: [RFC] Per file OOM badness

.@amd.com> Subject: [RFC] Per file OOM badness Hi, this series is a revised version of an RFC sent by Christian König a few years ago. The original RFC can be found at https://lists.freedesktop.org/archives/dri-devel/2015-September/089778.html This is the same idea and I've just adressed his

Re: [RFC] Per file OOM badness

On Thu 18-01-18 18:00:06, Michal Hocko wrote: > On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: > > Hi, this series is a revised version of an RFC sent by Christian König > > a few years ago. The original RFC can be found at > >

Re: [RFC] Per file OOM badness

Michal Hocko writes: > On Thu 18-01-18 18:00:06, Michal Hocko wrote: >> On Thu 18-01-18 11:47:48, Andrey Grodzovsky wrote: >> > Hi, this series is a revised version of an RFC sent by Christian König >> > a few years ago. The original RFC can be found at >> >

[RFC] Per file OOM badness

Hi, this series is a revised version of an RFC sent by Christian König a few years ago. The original RFC can be found at https://lists.freedesktop.org/archives/dri-devel/2015-September/089778.html This is the same idea and I've just adressed his concern from the original RFC and switched to a