Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-08 Thread Johannes Weiner
On Thu, Oct 08, 2020 at 08:55:57AM -0700, Shakeel Butt wrote: > On Thu, Oct 8, 2020 at 7:55 AM Johannes Weiner wrote: > > > > On Tue, Oct 06, 2020 at 09:55:43AM -0700, Shakeel Butt wrote: > > > On Thu, Oct 1, 2020 at 7:33 AM Johannes Weiner wrote: > > > > > > > [snip] > > > > > >So instead

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-08 Thread Shakeel Butt
On Thu, Oct 8, 2020 at 7:55 AM Johannes Weiner wrote: > > On Tue, Oct 06, 2020 at 09:55:43AM -0700, Shakeel Butt wrote: > > On Thu, Oct 1, 2020 at 7:33 AM Johannes Weiner wrote: > > > > > [snip] > > > > >So instead of asking users for a target size whose suitability > > > > >heavily

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-08 Thread Johannes Weiner
On Mon, Oct 05, 2020 at 02:59:10PM -0700, Shakeel Butt wrote: > Hi Johannes, > > On Thu, Oct 1, 2020 at 8:12 AM Johannes Weiner wrote: > > > > Hello Shakeel, > > > > On Wed, Sep 30, 2020 at 08:26:26AM -0700, Shakeel Butt wrote: > > > On Mon, Sep 28, 2020 at 2:03 PM Johannes Weiner > > > wrote:

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-08 Thread Johannes Weiner
On Tue, Oct 06, 2020 at 09:55:43AM -0700, Shakeel Butt wrote: > On Thu, Oct 1, 2020 at 7:33 AM Johannes Weiner wrote: > > > [snip] > > > >So instead of asking users for a target size whose suitability > > > >heavily depends on the kernel's LRU implementation, the readahead > > > >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-06 Thread Shakeel Butt
On Thu, Oct 1, 2020 at 7:33 AM Johannes Weiner wrote: > [snip] > > >So instead of asking users for a target size whose suitability > > >heavily depends on the kernel's LRU implementation, the readahead > > >code, the IO device's capability and general load, why not directly > > >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-05 Thread Shakeel Butt
Hi Johannes, On Thu, Oct 1, 2020 at 8:12 AM Johannes Weiner wrote: > > Hello Shakeel, > > On Wed, Sep 30, 2020 at 08:26:26AM -0700, Shakeel Butt wrote: > > On Mon, Sep 28, 2020 at 2:03 PM Johannes Weiner wrote: > > > Workloads may not > > > allocate anything for hours, and then suddenly

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-01 Thread Johannes Weiner
Hello Shakeel, On Wed, Sep 30, 2020 at 08:26:26AM -0700, Shakeel Butt wrote: > On Mon, Sep 28, 2020 at 2:03 PM Johannes Weiner wrote: > > Workloads may not > > allocate anything for hours, and then suddenly allocate gigabytes > > within seconds. A sudden onset of streaming reads through the > >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-10-01 Thread Johannes Weiner
On Wed, Sep 30, 2020 at 08:45:17AM -0700, Shakeel Butt wrote: > On Tue, Sep 29, 2020 at 2:55 PM Johannes Weiner wrote: > > > > On Tue, Sep 29, 2020 at 05:04:44PM +0200, Michal Hocko wrote: > > > On Mon 28-09-20 17:02:16, Johannes Weiner wrote: > > > [...] > > > > My take is that a proactive

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-30 Thread Shakeel Butt
On Tue, Sep 29, 2020 at 2:55 PM Johannes Weiner wrote: > > On Tue, Sep 29, 2020 at 05:04:44PM +0200, Michal Hocko wrote: > > On Mon 28-09-20 17:02:16, Johannes Weiner wrote: > > [...] > > > My take is that a proactive reclaim feature, whose goal is never to > > > thrash or punish but to keep the

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-30 Thread Shakeel Butt
Hi Johannes, On Mon, Sep 28, 2020 at 2:03 PM Johannes Weiner wrote: > > Hello, > > I apologize for the late reply. The proposed interface has been an > ongoing topic and area of experimentation within Facebook as well, > which makes it a bit difficult to respond with certainty here. > > I agree

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-29 Thread Johannes Weiner
On Tue, Sep 29, 2020 at 05:04:44PM +0200, Michal Hocko wrote: > On Mon 28-09-20 17:02:16, Johannes Weiner wrote: > [...] > > My take is that a proactive reclaim feature, whose goal is never to > > thrash or punish but to keep the LRUs warm and the workingset trimmed, > > would ideally have: > > >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-29 Thread Michal Hocko
On Mon 28-09-20 17:02:16, Johannes Weiner wrote: [...] > My take is that a proactive reclaim feature, whose goal is never to > thrash or punish but to keep the LRUs warm and the workingset trimmed, > would ideally have: > > - a pressure or size target specified by userspace but with >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-28 Thread Johannes Weiner
Hello, I apologize for the late reply. The proposed interface has been an ongoing topic and area of experimentation within Facebook as well, which makes it a bit difficult to respond with certainty here. I agree with both your usecases. They apply to us as well. We currently make two small

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Shakeel Butt
On Tue, Sep 22, 2020 at 12:09 PM Michal Hocko wrote: > > On Tue 22-09-20 11:10:17, Shakeel Butt wrote: > > On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: > [...] > > > Last but not least the memcg > > > background reclaim is something that should be possible without a new > > > interface. >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Yang Shi
On Tue, Sep 22, 2020 at 12:09 PM Michal Hocko wrote: > > On Tue 22-09-20 11:10:17, Shakeel Butt wrote: > > On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: > [...] > > > Last but not least the memcg > > > background reclaim is something that should be possible without a new > > > interface. >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Michal Hocko
On Tue 22-09-20 11:10:17, Shakeel Butt wrote: > On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: [...] > > Last but not least the memcg > > background reclaim is something that should be possible without a new > > interface. > > So, it comes down to adding more functionality/semantics to >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Shakeel Butt
On Tue, Sep 22, 2020 at 11:31 AM Michal Hocko wrote: > > On Tue 22-09-20 11:10:17, Shakeel Butt wrote: > > On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: > [...] > > > So far I have learned that you are primarily working around an > > > implementation detail in the zswap which is doing the

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Michal Hocko
On Tue 22-09-20 11:10:17, Shakeel Butt wrote: > On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: [...] > > So far I have learned that you are primarily working around an > > implementation detail in the zswap which is doing the swapout path > > directly in the pageout path. > > Wait how did

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Shakeel Butt
On Tue, Sep 22, 2020 at 9:55 AM Michal Hocko wrote: > > On Tue 22-09-20 08:54:25, Shakeel Butt wrote: > > On Tue, Sep 22, 2020 at 4:49 AM Michal Hocko wrote: > > > > > > On Mon 21-09-20 10:50:14, Shakeel Butt wrote: > [...] > > > > Let me add one more point. Even if the high limit reclaim is

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Michal Hocko
On Tue 22-09-20 08:54:25, Shakeel Butt wrote: > On Tue, Sep 22, 2020 at 4:49 AM Michal Hocko wrote: > > > > On Mon 21-09-20 10:50:14, Shakeel Butt wrote: [...] > > > Let me add one more point. Even if the high limit reclaim is swift, it > > > can still take 100s of usecs. Most of our jobs are

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Shakeel Butt
On Tue, Sep 22, 2020 at 4:49 AM Michal Hocko wrote: > > On Mon 21-09-20 10:50:14, Shakeel Butt wrote: > > On Mon, Sep 21, 2020 at 9:30 AM Michal Hocko wrote: > > > > > > On Wed 09-09-20 14:57:52, Shakeel Butt wrote: > > > > Introduce an memcg interface to trigger memory reclaim on a memory > >

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-22 Thread Michal Hocko
On Mon 21-09-20 10:50:14, Shakeel Butt wrote: > On Mon, Sep 21, 2020 at 9:30 AM Michal Hocko wrote: > > > > On Wed 09-09-20 14:57:52, Shakeel Butt wrote: > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > > > Use cases: > > > -- > > > > > > 1)

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-21 Thread Shakeel Butt
On Mon, Sep 21, 2020 at 9:30 AM Michal Hocko wrote: > > On Wed 09-09-20 14:57:52, Shakeel Butt wrote: > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > Use cases: > > -- > > > > 1) Per-memcg uswapd: > > > > Usually applications consists of combination

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-21 Thread Michal Hocko
On Wed 09-09-20 14:57:52, Shakeel Butt wrote: > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > Use cases: > -- > > 1) Per-memcg uswapd: > > Usually applications consists of combination of latency sensitive and > latency tolerant tasks. For example, tasks

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-10 Thread SeongJae Park
> On Wed, Sep 9, 2020 at 11:37 PM SeongJae Park wrote: > > > > On 2020-09-09T14:57:52-07:00 Shakeel Butt wrote: > > > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > > > Use cases: > > > -- > > > > > > 1) Per-memcg uswapd: > > > > > > Usually

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-10 Thread Shakeel Butt
On Wed, Sep 9, 2020 at 11:37 PM SeongJae Park wrote: > > On 2020-09-09T14:57:52-07:00 Shakeel Butt wrote: > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > Use cases: > > -- > > > > 1) Per-memcg uswapd: > > > > Usually applications consists of

Re: [PATCH] memcg: introduce per-memcg reclaim interface

2020-09-10 Thread SeongJae Park
On 2020-09-09T14:57:52-07:00 Shakeel Butt wrote: > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > Use cases: > -- > > 1) Per-memcg uswapd: > > Usually applications consists of combination of latency sensitive and > latency tolerant tasks. For example,

[PATCH] memcg: introduce per-memcg reclaim interface

2020-09-09 Thread Shakeel Butt
Introduce an memcg interface to trigger memory reclaim on a memory cgroup. Use cases: -- 1) Per-memcg uswapd: Usually applications consists of combination of latency sensitive and latency tolerant tasks. For example, tasks serving user requests vs tasks doing data backup for a database