Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Tue, 26 Mar 2024 16:03:09 -0700 SeongJae Park wrote: > On Mon, 25 Mar 2024 15:53:03 -0700 SeongJae Park wrote: > > > On Mon, 25 Mar 2024 21:01:04 +0900 Honggyu Kim wrote: > [...] > > > On Fri, 22 Mar 2024 09:32:23 -0700 SeongJae Park wrote: > > > > On Fri, 22 Mar 2024 18:02:23 +0900 Honggyu Kim > > > > wrote: > [...] > > > > > > I would like to hear how you think about this. > > > > So, to summarize my humble opinion, > > > > 1. I like the idea of having two actions. But I'd like to use names other > > than > >'promote' and 'demote'. > > 2. I still prefer having a filter for the page granularity access re-check. > > > [...] > > > I will join the DAMON Beer/Coffee/Tea Chat tomorrow as scheduled so I > > > can talk more about this issue. > > > > Looking forward to chatting with you :) > > We met and discussed about this topic in the chat series yesterday. Sharing > the summary for keeping the open discussion. > > Honggyu thankfully accepted my humble suggestions on the last reply. Honggyu > will post the third version of this patchset soon. The patchset will > implement > two new DAMOS actions, namely MIGRATE_HOT and MIGRATE_COLD. Those will > migrate > the DAMOS target regions to a user-specified NUMA node, but will have > different > prioritization score function. As name implies, they will prioritize more hot > regions and cold regions, respectively. It's a late answer but thanks very much for the summary. It was really helpful discussion in the chat series. I'm leaving a message so that anyone can follow for the update. The v3 is posted at https://lore.kernel.org/damon/20240405060858.2818-1-honggyu@sk.com. > Honggyu, please feel free to fix if there is anything wrong or missed. I don't have anything to fix from your summary. > And thanks to Honggyu again for patiently keeping this productive discussion > and their awesome work. I appreciate your persistent support and kind reviews. Thanks, Honggyu > > Thanks, > SJ > > [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Mon, 25 Mar 2024 15:53:03 -0700 SeongJae Park wrote: > On Mon, 25 Mar 2024 21:01:04 +0900 Honggyu Kim wrote: [...] > > On Fri, 22 Mar 2024 09:32:23 -0700 SeongJae Park wrote: > > > On Fri, 22 Mar 2024 18:02:23 +0900 Honggyu Kim wrote: [...] > > > > I would like to hear how you think about this. > > So, to summarize my humble opinion, > > 1. I like the idea of having two actions. But I'd like to use names other > than >'promote' and 'demote'. > 2. I still prefer having a filter for the page granularity access re-check. > [...] > > I will join the DAMON Beer/Coffee/Tea Chat tomorrow as scheduled so I > > can talk more about this issue. > > Looking forward to chatting with you :) We met and discussed about this topic in the chat series yesterday. Sharing the summary for keeping the open discussion. Honggyu thankfully accepted my humble suggestions on the last reply. Honggyu will post the third version of this patchset soon. The patchset will implement two new DAMOS actions, namely MIGRATE_HOT and MIGRATE_COLD. Those will migrate the DAMOS target regions to a user-specified NUMA node, but will have different prioritization score function. As name implies, they will prioritize more hot regions and cold regions, respectively. Honggyu, please feel free to fix if there is anything wrong or missed. And thanks to Honggyu again for patiently keeping this productive discussion and their awesome work. Thanks, SJ [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Mon, 25 Mar 2024 21:01:04 +0900 Honggyu Kim wrote:
> Hi SeongJae,
>
> On Fri, 22 Mar 2024 09:32:23 -0700 SeongJae Park wrote:
> > On Fri, 22 Mar 2024 18:02:23 +0900 Honggyu Kim wrote:
[...]
> > > > Honggyu joined DAMON Beer/Coffee/Tea Chat[1] yesterday, and we
> > > > discussed about
> > > > this patchset in high level. Sharing the summary here for open
> > > > discussion. As
> > > > also discussed on the first version of this patchset[2], we want to
> > > > make single
> > > > action for general page migration with minimum changes, but would like
> > > > to keep
> > > > page level access re-check. We also agreed the previously proposed
> > > > DAMOS
> > > > filter-based approach could make sense for the purpose.
> > >
> > > Thanks very much for the summary. I have been trying to merge promote
> > > and demote actions into a single migrate action, but I found an issue
> > > regarding damon_pa_scheme_score. It currently calls damon_cold_score()
> > > for demote action and damon_hot_score() for promote action, but what
> > > should we call when we use a single migrate action?
> >
> > Good point! This is what I didn't think about when suggesting that. Thank
> > you
> > for letting me know this gap! I think there could be two approach, off the
> > top
> > of my head.
> >
> > The first one would be extending the interface so that the user can select
> > the
> > score function. This would let flexible usage, but I'm bit concerned if
> > this
> > could make things unnecessarily complex, and would really useful in many
> > general use case.
>
> I also think this looks complicated and may not be useful for general
> users.
>
> > The second approach would be letting DAMON infer the intention. In this
> > case,
> > I think we could know the intention is the demotion if the scheme has a youg
> > pages exclusion filter. Then, we could use the cold_score(). And vice
> > versa.
> > To cover a case that there is no filter at all, I think we could have one
> > assumption. My humble intuition says the new action (migrate) may be used
> > more
> > for promotion use case. So, in damon_pa_scheme_score(), if the action of
> > the
> > given scheme is the new one (say, MIGRATE), the function will further check
> > if
> > the scheme has a filter for excluding young pages. If so, the function will
> > use cold_score(). Otherwise, the function will use hot_score().
>
> Thanks for suggesting many ideas but I'm afraid that I feel this doesn't
> look good. Thinking it again, I think we can think about keep using
> DAMOS_PROMOTE and DAMOS_DEMOTE,
In other words, keep having dedicated DAMOS action for intuitive prioritization
score function, or, coupling the prioritization with each action, right? I
think this makes sense, and fit well with the documentation.
The prioritization mechanism should be different for each action. For
example,
rarely accessed (colder) memory regions would be prioritized for page-out
scheme action. In contrast, the colder regions would be deprioritized for
huge
page collapse scheme action. Hence, the prioritization mechanisms for each
action are implemented in each DAMON operations set, together with the
actions.
In other words, each DAMOS action should allow users intuitively understand
what types of regions will be prioritized. We already have such couples of
DAMOS actions such as DAMOS_[NO]HUGEPAGE and DAMOS_LRU_[DE]PRIO. So adding a
couple of action for this case sounds reasonable to me. And I think this is
better and simpler than having the inferrence based behavior.
That said, I concern if 'PROMOTE' and 'DEMOTE' still sound bit ambiguous to
people who don't know 'demote_folio_list()' and its friends. Meanwhile, the
name might sound too detail about what it does to people who know the
functions, so make it bit unflexible. They might also get confused since we
don't have 'promote_folio_list()'.
To my humble understanding, what you really want to do is migrating pages to
specific address range (or node) prioritizing the pages based on the hotness.
What about, say, MIGRATE_{HOT,COLD}?
> but I can make them directly call
> damon_folio_young() for access check instead of using young filter.
>
> And we can internally handle the complicated combination such as demote
> action sets "young" filter with "matching" true and promote action sets
> "young" filter with "matching" false. IMHO, this will make the usage
> simpler.
I think whether to exclude young/non-young (maybe idle is better than
non-young?) pages from the action is better to be decoupled for following
reasons.
Firstly, we want to check the page granularity youngness mainly because we
found DAMON's monitoring result is not accurate enough for this use case. Or,
we could say that's because you cannot wait until DAMON's monitoring result
becomes accurate enough. For more detail, you could increase minimum age of
your scheme's target access pattern. I show you set
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Fri, 22 Mar 2024 09:32:23 -0700 SeongJae Park wrote: > On Fri, 22 Mar 2024 18:02:23 +0900 Honggyu Kim wrote: > > > Hi SeongJae, > > > > On Tue, 27 Feb 2024 15:51:20 -0800 SeongJae Park wrote: > > > On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > > > > > > > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > > > > posted at [1]. > > > > > > > > It says there is no implementation of the demote/promote DAMOS action > > > > are made. This RFC is about its implementation for physical address > > > > space. > > > > > > > > > [...] > > > Thank you for running the tests again with the new version of the patches > > > and > > > sharing the results! > > > > It's a bit late answer, but the result was from the previous evaluation. > > I ran it again with RFC v2, but didn't see much difference so just > > pasted the same result here. > > No problem, thank you for clarifying :) > > [...] > > > > Honggyu Kim (3): > > > > mm/damon: refactor DAMOS_PAGEOUT with migration_mode > > > > mm: make alloc_demote_folio externally invokable for migration > > > > mm/damon: introduce DAMOS_DEMOTE action for demotion > > > > > > > > Hyeongtak Ji (4): > > > > mm/memory-tiers: add next_promotion_node to find promotion target > > > > mm/damon: introduce DAMOS_PROMOTE action for promotion > > > > mm/damon/sysfs-schemes: add target_nid on sysfs-schemes > > > > mm/damon/sysfs-schemes: apply target_nid for promote and demote > > > > actions > > > > > > Honggyu joined DAMON Beer/Coffee/Tea Chat[1] yesterday, and we discussed > > > about > > > this patchset in high level. Sharing the summary here for open > > > discussion. As > > > also discussed on the first version of this patchset[2], we want to make > > > single > > > action for general page migration with minimum changes, but would like to > > > keep > > > page level access re-check. We also agreed the previously proposed DAMOS > > > filter-based approach could make sense for the purpose. > > > > Thanks very much for the summary. I have been trying to merge promote > > and demote actions into a single migrate action, but I found an issue > > regarding damon_pa_scheme_score. It currently calls damon_cold_score() > > for demote action and damon_hot_score() for promote action, but what > > should we call when we use a single migrate action? > > Good point! This is what I didn't think about when suggesting that. Thank > you > for letting me know this gap! I think there could be two approach, off the > top > of my head. > > The first one would be extending the interface so that the user can select the > score function. This would let flexible usage, but I'm bit concerned if this > could make things unnecessarily complex, and would really useful in many > general use case. I also think this looks complicated and may not be useful for general users. > The second approach would be letting DAMON infer the intention. In this case, > I think we could know the intention is the demotion if the scheme has a youg > pages exclusion filter. Then, we could use the cold_score(). And vice versa. > To cover a case that there is no filter at all, I think we could have one > assumption. My humble intuition says the new action (migrate) may be used > more > for promotion use case. So, in damon_pa_scheme_score(), if the action of the > given scheme is the new one (say, MIGRATE), the function will further check if > the scheme has a filter for excluding young pages. If so, the function will > use cold_score(). Otherwise, the function will use hot_score(). Thanks for suggesting many ideas but I'm afraid that I feel this doesn't look good. Thinking it again, I think we can think about keep using DAMOS_PROMOTE and DAMOS_DEMOTE, but I can make them directly call damon_folio_young() for access check instead of using young filter. And we can internally handle the complicated combination such as demote action sets "young" filter with "matching" true and promote action sets "young" filter with "matching" false. IMHO, this will make the usage simpler. I would like to hear how you think about this. > So I'd more prefer the second approach. I think it would be not too late to > consider the first approach after waiting for it turns out more actions have > such ambiguity and need more general interface for explicitly set the score > function. I will join the DAMON Beer/Coffee/Tea Chat tomorrow as scheduled so I can talk more about this issue. Thanks, Honggyu > > Thanks, > SJ > > [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Fri, 22 Mar 2024 17:27:34 +0900 Honggyu Kim wrote: [...] > OK. It could be a matter of preference and the current filter is already > in the mainline so I won't insist more. Thank you for accepting my humble suggestion. [...] > > I'd prefer improving the documents or > > user-space tool and keep the kernel code simple. > > OK. I will see if there is a way to improve damo tool for this instead > of making changes on the kernel side. Looking forward! [...] > Yeah, I made this thread too much about filter naming discussion rather > than tiered memory support. No problem at all. Thank you for keeping this productive discussion. [...] > Thanks again for your feedback. That's my pleasure :) Thanks, SJ [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Fri, 22 Mar 2024 18:02:23 +0900 Honggyu Kim wrote: > Hi SeongJae, > > On Tue, 27 Feb 2024 15:51:20 -0800 SeongJae Park wrote: > > On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > > > > > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > > > posted at [1]. > > > > > > It says there is no implementation of the demote/promote DAMOS action > > > are made. This RFC is about its implementation for physical address > > > space. > > > > > > [...] > > Thank you for running the tests again with the new version of the patches > > and > > sharing the results! > > It's a bit late answer, but the result was from the previous evaluation. > I ran it again with RFC v2, but didn't see much difference so just > pasted the same result here. No problem, thank you for clarifying :) [...] > > > Honggyu Kim (3): > > > mm/damon: refactor DAMOS_PAGEOUT with migration_mode > > > mm: make alloc_demote_folio externally invokable for migration > > > mm/damon: introduce DAMOS_DEMOTE action for demotion > > > > > > Hyeongtak Ji (4): > > > mm/memory-tiers: add next_promotion_node to find promotion target > > > mm/damon: introduce DAMOS_PROMOTE action for promotion > > > mm/damon/sysfs-schemes: add target_nid on sysfs-schemes > > > mm/damon/sysfs-schemes: apply target_nid for promote and demote > > > actions > > > > Honggyu joined DAMON Beer/Coffee/Tea Chat[1] yesterday, and we discussed > > about > > this patchset in high level. Sharing the summary here for open discussion. > > As > > also discussed on the first version of this patchset[2], we want to make > > single > > action for general page migration with minimum changes, but would like to > > keep > > page level access re-check. We also agreed the previously proposed DAMOS > > filter-based approach could make sense for the purpose. > > Thanks very much for the summary. I have been trying to merge promote > and demote actions into a single migrate action, but I found an issue > regarding damon_pa_scheme_score. It currently calls damon_cold_score() > for demote action and damon_hot_score() for promote action, but what > should we call when we use a single migrate action? Good point! This is what I didn't think about when suggesting that. Thank you for letting me know this gap! I think there could be two approach, off the top of my head. The first one would be extending the interface so that the user can select the score function. This would let flexible usage, but I'm bit concerned if this could make things unnecessarily complex, and would really useful in many general use case. The second approach would be letting DAMON infer the intention. In this case, I think we could know the intention is the demotion if the scheme has a youg pages exclusion filter. Then, we could use the cold_score(). And vice versa. To cover a case that there is no filter at all, I think we could have one assumption. My humble intuition says the new action (migrate) may be used more for promotion use case. So, in damon_pa_scheme_score(), if the action of the given scheme is the new one (say, MIGRATE), the function will further check if the scheme has a filter for excluding young pages. If so, the function will use cold_score(). Otherwise, the function will use hot_score(). So I'd more prefer the second approach. I think it would be not too late to consider the first approach after waiting for it turns out more actions have such ambiguity and need more general interface for explicitly set the score function. Thanks, SJ [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Tue, 27 Feb 2024 15:51:20 -0800 SeongJae Park wrote: > On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > > > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > > posted at [1]. > > > > It says there is no implementation of the demote/promote DAMOS action > > are made. This RFC is about its implementation for physical address > > space. > > > > > > Introduction > > > > > > With the advent of CXL/PCIe attached DRAM, which will be called simply > > as CXL memory in this cover letter, some systems are becoming more > > heterogeneous having memory systems with different latency and bandwidth > > characteristics. They are usually handled as different NUMA nodes in > > separate memory tiers and CXL memory is used as slow tiers because of > > its protocol overhead compared to local DRAM. > > > > In this kind of systems, we need to be careful placing memory pages on > > proper NUMA nodes based on the memory access frequency. Otherwise, some > > frequently accessed pages might reside on slow tiers and it makes > > performance degradation unexpectedly. Moreover, the memory access > > patterns can be changed at runtime. > > > > To handle this problem, we need a way to monitor the memory access > > patterns and migrate pages based on their access temperature. The > > DAMON(Data Access MONitor) framework and its DAMOS(DAMON-based Operation > > Schemes) can be useful features for monitoring and migrating pages. > > DAMOS provides multiple actions based on DAMON monitoring results and it > > can be used for proactive reclaim, which means swapping cold pages out > > with DAMOS_PAGEOUT action, but it doesn't support migration actions such > > as demotion and promotion between tiered memory nodes. > > > > This series supports two new DAMOS actions; DAMOS_DEMOTE for demotion > > from fast tiers and DAMOS_PROMOTE for promotion from slow tiers. This > > prevents hot pages from being stuck on slow tiers, which makes > > performance degradation and cold pages can be proactively demoted to > > slow tiers so that the system can increase the chance to allocate more > > hot pages to fast tiers. > > > > The DAMON provides various tuning knobs but we found that the proactive > > demotion for cold pages is especially useful when the system is running > > out of memory on its fast tier nodes. > > > > Our evaluation result shows that it reduces the performance slowdown > > compared to the default memory policy from 15~17% to 4~5% when the > > system runs under high memory pressure on its fast tier DRAM nodes. > > > > > > DAMON configuration > > === > > > > The specific DAMON configuration doesn't have to be in the scope of this > > patch series, but some rough idea is better to be shared to explain the > > evaluation result. > > > > The DAMON provides many knobs for fine tuning but its configuration file > > is generated by HMSDK[2]. It includes gen_config.py script that > > generates a json file with the full config of DAMON knobs and it creates > > multiple kdamonds for each NUMA node when the DAMON is enabled so that > > it can run hot/cold based migration for tiered memory. > > I was feeling a bit confused from here since DAMON doesn't receive parameters > via a file. To my understanding, the 'configuration file' means the input > file > for DAMON user-space tool, damo, not DAMON. Just a trivial thing, but making > it clear if possible could help readers in my opinion. > > > > > > > Evaluation Workload > > === > > > > The performance evaluation is done with redis[3], which is a widely used > > in-memory database and the memory access patterns are generated via > > YCSB[4]. We have measured two different workloads with zipfian and > > latest distributions but their configs are slightly modified to make > > memory usage higher and execution time longer for better evaluation. > > > > The idea of evaluation using these demote and promote actions covers > > system-wide memory management rather than partitioning hot/cold pages of > > a single workload. The default memory allocation policy creates pages > > to the fast tier DRAM node first, then allocates newly created pages to > > the slow tier CXL node when the DRAM node has insufficient free space. > > Once the page allocation is done then those pages never move between > > NUMA nodes. It's not true when using numa balancing, but it is not the > > scope of this DAMON based 2-tier memory management support. > > > > If the working set of redis can be fit fully into the DRAM node, then > > the redis will access the fast DRAM only. Since the performance of DRAM > > only is faster than partially accessing CXL memory in slow tiers, this > > environment is not useful to evaluate this patch series. > > > > To make pages of redis be distributed across fast DRAM node and slow > > CXL node to evaluate our demote and promote actions, we pre-allocate > > some cold memory
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Wed, 20 Mar 2024 09:56:19 -0700 SeongJae Park wrote: > Hi Honggyu, > > On Wed, 20 Mar 2024 16:07:48 +0900 Honggyu Kim wrote: > > > Hi SeongJae, > > > > On Mon, 18 Mar 2024 12:07:21 -0700 SeongJae Park wrote: > > > On Mon, 18 Mar 2024 22:27:45 +0900 Honggyu Kim wrote: > > > > > > > Hi SeongJae, > > > > > > > > On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park > > > > wrote: > > > > > Hi Honggyu, > > > > > > > > > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim > > > > > wrote: > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > Thanks for the confirmation. I have a few comments on young filter > > > > > > so > > > > > > please read the inline comments again. > > > > > > > > > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park > > > > > > wrote: > > > > > > > Hi Honggyu, > > > > > > > > > > > > > > > > -Original Message- > > > > > > > > > From: SeongJae Park > > > > > > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > > > > > > To: Honggyu Kim > > > > > > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > > > > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > > > > > > management for CXL memory > > > > > > > > > > > > > > > > > > Hi Honggyu, > > > > > > > > > > > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > > > > > > > > > I've tested it again and found that "young" filter has to > > > > > > > > > > be set > > > > > > > > > > differently as follows. > > > > > > > > > > - demote action: set "young" filter with "matching" true > > > > > > > > > > - promote action: set "young" filter with "matching" false > > > > > > > > Thinking it again, I feel like "matching" true or false looks quite > > > > vague to me as a general user. > > > > > > > > Instead, I would like to have more meaningful names for "matching" as > > > > follows. > > > > > > > > - matching "true" can be either (filter) "out" or "skip". > > > > - matching "false" can be either (filter) "in" or "apply". > > > > > > I agree the naming could be done much better. And thank you for the nice > > > suggestions. I have a few concerns, though. > > > > I don't think my suggestion is best. I just would like to have more > > discussion about it. > > I also understand my naming sense is far from good :) I'm grateful to have > this constructive discussion! Yeah, naming is always difficult. Thanks anyway :) > > > > > Firstly, increasing the number of behavioral concepts. DAMOS filter > > > feature > > > has only single behavior: excluding some types of memory from DAMOS action > > > target. The "matching" is to provide a flexible way for further > > > specifying the > > > target to exclude in a bit detail. Without it, we would need non-variant > > > for > > > each filter type. Comapred to the current terms, the new terms feel like > > > implying there are two types of behaviors. I think one behavior is > > > easier to > > > understand than two behaviors, and better match what internal code is > > > doing. > > > > > > Secondly, ambiguity in "in" and "apply". To me, the terms sound like > > > _adding_ > > > something more than _excluding_. > > > > I understood that young filter "matching" "false" means apply action > > only to young pages. Do I misunderstood something here? If not, > > Technically speaking, having a DAMOS filter with 'matching' parameter as >
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi Honggyu, On Wed, 20 Mar 2024 16:07:48 +0900 Honggyu Kim wrote: > Hi SeongJae, > > On Mon, 18 Mar 2024 12:07:21 -0700 SeongJae Park wrote: > > On Mon, 18 Mar 2024 22:27:45 +0900 Honggyu Kim wrote: > > > > > Hi SeongJae, > > > > > > On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > > > > Hi Honggyu, > > > > > > > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim > > > > wrote: > > > > > > > > > Hi SeongJae, > > > > > > > > > > Thanks for the confirmation. I have a few comments on young filter so > > > > > please read the inline comments again. > > > > > > > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park > > > > > wrote: > > > > > > Hi Honggyu, > > > > > > > > > > > > > > -----Original Message- > > > > > > > > From: SeongJae Park > > > > > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > > > > > To: Honggyu Kim > > > > > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > > > > > management for CXL memory > > > > > > > > > > > > > > > > Hi Honggyu, > > > > > > > > > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > > > > > > > I've tested it again and found that "young" filter has to be > > > > > > > > > set > > > > > > > > > differently as follows. > > > > > > > > > - demote action: set "young" filter with "matching" true > > > > > > > > > - promote action: set "young" filter with "matching" false > > > > > > Thinking it again, I feel like "matching" true or false looks quite > > > vague to me as a general user. > > > > > > Instead, I would like to have more meaningful names for "matching" as > > > follows. > > > > > > - matching "true" can be either (filter) "out" or "skip". > > > - matching "false" can be either (filter) "in" or "apply". > > > > I agree the naming could be done much better. And thank you for the nice > > suggestions. I have a few concerns, though. > > I don't think my suggestion is best. I just would like to have more > discussion about it. I also understand my naming sense is far from good :) I'm grateful to have this constructive discussion! > > > Firstly, increasing the number of behavioral concepts. DAMOS filter feature > > has only single behavior: excluding some types of memory from DAMOS action > > target. The "matching" is to provide a flexible way for further specifying > > the > > target to exclude in a bit detail. Without it, we would need non-variant > > for > > each filter type. Comapred to the current terms, the new terms feel like > > implying there are two types of behaviors. I think one behavior is easier > > to > > understand than two behaviors, and better match what internal code is doing. > > > > Secondly, ambiguity in "in" and "apply". To me, the terms sound like > > _adding_ > > something more than _excluding_. > > I understood that young filter "matching" "false" means apply action > only to young pages. Do I misunderstood something here? If not, Technically speaking, having a DAMOS filter with 'matching' parameter as 'false' for 'young pages' type means you want DAMOS to "exclude pages that not young from the scheme's action target". That's the only thing it truly does, and what it tries to guarantee. Whether the action will be applied to young pages or not depends on more factors including additional filters and DAMOS parameter. IOW, that's not what the simple setting promises. Of course, I know you are assuming there is only the single filter. Hence, effectively you're correct. And the sentence may be a better wording for end users. However, it tooke me a bit time to understand your assumption and concluding whether
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Mon, 18 Mar 2024 12:07:21 -0700 SeongJae Park wrote: > On Mon, 18 Mar 2024 22:27:45 +0900 Honggyu Kim wrote: > > > Hi SeongJae, > > > > On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > > > Hi Honggyu, > > > > > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > > > > > > > Hi SeongJae, > > > > > > > > Thanks for the confirmation. I have a few comments on young filter so > > > > please read the inline comments again. > > > > > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park > > > > wrote: > > > > > Hi Honggyu, > > > > > > > > > > > > -Original Message- > > > > > > > From: SeongJae Park > > > > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > > > > To: Honggyu Kim > > > > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > > > > management for CXL memory > > > > > > > > > > > > > > Hi Honggyu, > > > > > > > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > > > > wrote: > > > > > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > > > > > I've tested it again and found that "young" filter has to be set > > > > > > > > differently as follows. > > > > > > > > - demote action: set "young" filter with "matching" true > > > > > > > > - promote action: set "young" filter with "matching" false > > > > Thinking it again, I feel like "matching" true or false looks quite > > vague to me as a general user. > > > > Instead, I would like to have more meaningful names for "matching" as > > follows. > > > > - matching "true" can be either (filter) "out" or "skip". > > - matching "false" can be either (filter) "in" or "apply". > > I agree the naming could be done much better. And thank you for the nice > suggestions. I have a few concerns, though. I don't think my suggestion is best. I just would like to have more discussion about it. > Firstly, increasing the number of behavioral concepts. DAMOS filter feature > has only single behavior: excluding some types of memory from DAMOS action > target. The "matching" is to provide a flexible way for further specifying > the > target to exclude in a bit detail. Without it, we would need non-variant for > each filter type. Comapred to the current terms, the new terms feel like > implying there are two types of behaviors. I think one behavior is easier to > understand than two behaviors, and better match what internal code is doing. > > Secondly, ambiguity in "in" and "apply". To me, the terms sound like _adding_ > something more than _excluding_. I understood that young filter "matching" "false" means apply action only to young pages. Do I misunderstood something here? If not, "apply" means _adding_ or _including_ only the matched pages for action. It looks like you thought about _excluding_ non matched pages here. > I think that might confuse people in some > cases. Actually, I have used the term "filter-out" and "filter-in" on > this and several threads. When saying about "filter-in" scenario, I had to > emphasize the fact that it is not adding something but excluding others. Excluding others also means including matched pages. I think we better focus on what to do only for the matched pages. > I now think that was not a good approach. > > Finally, "apply" sounds a bit deterministic. I think it could be a bit > confusing in some cases such as when using multiple filters in a combined way. > For example, if we have two filters for 1) "apply" a memcg and 2) skip anon > pages, the given DAMOS action will not be applied to anon pages of the memcg. > I think this might be a bit confusing. No objection on this. If so, I think "in" sounds better than "apply". > > > > Internally, the type of "matching" can be boolean, but it'd be better > > for general users have another ways to set it such as "out"/"in" or > > "s
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Mon, 18 Mar 2024 22:27:45 +0900 Honggyu Kim wrote: > Hi SeongJae, > > On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > > Hi Honggyu, > > > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > > > > > Hi SeongJae, > > > > > > Thanks for the confirmation. I have a few comments on young filter so > > > please read the inline comments again. > > > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > > > > Hi Honggyu, > > > > > > > > > > -Original Message- > > > > > > From: SeongJae Park > > > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > > > To: Honggyu Kim > > > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > > > management for CXL memory > > > > > > > > > > > > Hi Honggyu, > > > > > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > > > wrote: > > > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > > > I've tested it again and found that "young" filter has to be set > > > > > > > differently as follows. > > > > > > > - demote action: set "young" filter with "matching" true > > > > > > > - promote action: set "young" filter with "matching" false > > Thinking it again, I feel like "matching" true or false looks quite > vague to me as a general user. > > Instead, I would like to have more meaningful names for "matching" as > follows. > > - matching "true" can be either (filter) "out" or "skip". > - matching "false" can be either (filter) "in" or "apply". I agree the naming could be done much better. And thank you for the nice suggestions. I have a few concerns, though. Firstly, increasing the number of behavioral concepts. DAMOS filter feature has only single behavior: excluding some types of memory from DAMOS action target. The "matching" is to provide a flexible way for further specifying the target to exclude in a bit detail. Without it, we would need non-variant for each filter type. Comapred to the current terms, the new terms feel like implying there are two types of behaviors. I think one behavior is easier to understand than two behaviors, and better match what internal code is doing. Secondly, ambiguity in "in" and "apply". To me, the terms sound like _adding_ something more than _excluding_. I think that might confuse people in some cases. Actually, I have used the term "filter-out" and "filter-in" on this and several threads. When saying about "filter-in" scenario, I had to emphasize the fact that it is not adding something but excluding others. I now think that was not a good approach. Finally, "apply" sounds a bit deterministic. I think it could be a bit confusing in some cases such as when using multiple filters in a combined way. For example, if we have two filters for 1) "apply" a memcg and 2) skip anon pages, the given DAMOS action will not be applied to anon pages of the memcg. I think this might be a bit confusing. > > Internally, the type of "matching" can be boolean, but it'd be better > for general users have another ways to set it such as "out"/"in" or > "skip"/"apply" via sysfs interface. I prefer "skip" and "apply" looks > more intuitive, but I don't have strong objection on "out" and "in" as > well. Unfortunately, DAMON sysfs interface is an ABI that we want to keep stable. Of course we could make some changes on it if really required. But I'm unsure if the problem of current naming and benefit of the sugegsted change are big enough to outweighs the stability risk and additional efforts. Also, DAMON sysfs interface is arguably not for _very_ general users. DAMON user-space tool is the one for _more_ general users. To quote DAMON usage document, - *DAMON user space tool.* `This <https://github.com/awslabs/damo>`_ is for privileged people such as system administrators who want a just-working human-friendly interface. [...] - *sysfs interface.* :ref:`This ` is for privileged user space programmers who want more optimized use of DAMON. [...] If the concept is that confused, I think we could improve the docum
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Sun, 17 Mar 2024 12:13:57 -0700 SeongJae Park wrote: > On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > > > Hi Honggyu, > > > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > > > > > Hi SeongJae, > > > > > > Thanks for the confirmation. I have a few comments on young filter so > > > please read the inline comments again. > > > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > > > > Hi Honggyu, > [...] > > > Thanks. I see that it works fine, but I would like to have more > > > discussion about "young" filter. What I think about filter is that if I > > > apply "young" filter "true" for demotion, then the action applies only > > > for "young" pages, but the current implementation works opposite. > > > > > > I understand the function name of internal implementation is > > > "damos_pa_filter_out" so the basic action is filtering out, but the > > > cgroup filter works in the opposite way for now. > > > > Does memcg filter works in the opposite way? I don't think so because > > __damos_pa_filter_out() sets 'matches' as 'true' only if the the given > > folio is > > contained in the given memcg. 'young' filter also simply sets 'matches' as > > 'true' only if the given folio is young. > > > > If it works in the opposite way, it's a bug that need to be fixed. Please > > let > > me know if I'm missing something. > > I just read the DAMOS filters part of the documentation for DAMON sysfs > interface again. I believe it is explaining the meaning of 'matching' as I > intended to, as below: > > You can write ``Y`` or ``N`` to ``matching`` file to filter out pages > that does > or does not match to the type, respectively. Then, the scheme's action > will > not be applied to the pages that specified to be filtered out. > > But, I found the following example for memcg filter is wrong, as below: > > For example, below restricts a DAMOS action to be applied to only > non-anonymous > pages of all memory cgroups except ``/having_care_already``.:: > > # echo 2 > nr_filters > # # filter out anonymous pages > echo anon > 0/type > echo Y > 0/matching > # # further filter out all cgroups except one at > '/having_care_already' > echo memcg > 1/type > echo /having_care_already > 1/memcg_path > echo N > 1/matching > > Specifically, the last line of the commands should write 'Y' instead of 'N' to > do what explained. Without the fix, the action will be applied to only > non-anonymous pages of 'having_care_already' memcg. This is definitely wrong. > I will fix this soon. I'm unsure if this is what made you to believe memcg > DAMOS filter is working in the opposite way, though. I got confused not because of this. I just think it again that this user interface is better to be more intuitive as I mentioned in the previous thread. Thanks, Honggyu > > Thanks, > SJ > > [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > Hi Honggyu, > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > > > Hi SeongJae, > > > > Thanks for the confirmation. I have a few comments on young filter so > > please read the inline comments again. > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > > > Hi Honggyu, > > > > > > > > -Original Message- > > > > > From: SeongJae Park > > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > > To: Honggyu Kim > > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > > management for CXL memory > > > > > > > > > > Hi Honggyu, > > > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > > wrote: > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > I've tested it again and found that "young" filter has to be set > > > > > > differently as follows. > > > > > > - demote action: set "young" filter with "matching" true > > > > > > - promote action: set "young" filter with "matching" false Thinking it again, I feel like "matching" true or false looks quite vague to me as a general user. Instead, I would like to have more meaningful names for "matching" as follows. - matching "true" can be either (filter) "out" or "skip". - matching "false" can be either (filter) "in" or "apply". Internally, the type of "matching" can be boolean, but it'd be better for general users have another ways to set it such as "out"/"in" or "skip"/"apply" via sysfs interface. I prefer "skip" and "apply" looks more intuitive, but I don't have strong objection on "out" and "in" as well. I also feel the filter name "young" is more for developers not for general users. I think this can be changed to "accessed" filter instead. The demote and promote filters can be set as follows using these. - demote action: set "accessed" filter with "matching" to "skip" - promote action: set "accessed" filter with "matching" to "apply" I also think that you can feel this is more complicated so I would like to hear how you think about this. > > > > > > > > > > DAMOS filter is basically for filtering "out" memory regions that > > > > > matches to > > > > > the condition. Right. In other tools, I see filters are more used as filtering "in" rather than filtering "out". I feel this makes me more confused. > > > > > Hence in your setup, young pages are not filtered out from > > > > > demote action target. > > > > > > > > I thought young filter true means "young pages ARE filtered out" for > > > > demotion. > > > > > > You're correct. > > > > Ack. > > > > > > > > > > > That is, you're demoting pages that "not" young. > > > > > > > > Your explanation here looks opposite to the previous statement. > > > > > > Again, you're correct. My intention was "non-young pages are not ..." but > > > maybe I was out of my mind and mistakenly removed "non-" part. Sorry for > > > the > > > confusion. > > > > No problem. I also think it's quite confusing. > > > > > > > > > > > And vice versa, so you're applying promote to non-non-young (young) > > > > > pages. > > > > > > > > Yes, I understand "promote non-non-young pages" means "promote young > > > > pages". > > > > This might be understood as "young pages are NOT filtered out" for > > > > promotion > > > > but it doesn't mean that "old pages are filtered out" instead. > > > > And we just rely hot detection only on DAMOS logics such as access > > > > frequency > > > > and age. Am I correct? > > > > > > You're correct. > > > > Ack. But if it doesn't mean that "old pages are filtered out" instead, > > It does mean that. Here, f
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Sun, 17 Mar 2024 08:31:44 -0700 SeongJae Park wrote: > Hi Honggyu, > > On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > > > Hi SeongJae, > > > > Thanks for the confirmation. I have a few comments on young filter so > > please read the inline comments again. > > > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > > > Hi Honggyu, [...] > > Thanks. I see that it works fine, but I would like to have more > > discussion about "young" filter. What I think about filter is that if I > > apply "young" filter "true" for demotion, then the action applies only > > for "young" pages, but the current implementation works opposite. > > > > I understand the function name of internal implementation is > > "damos_pa_filter_out" so the basic action is filtering out, but the > > cgroup filter works in the opposite way for now. > > Does memcg filter works in the opposite way? I don't think so because > __damos_pa_filter_out() sets 'matches' as 'true' only if the the given folio > is > contained in the given memcg. 'young' filter also simply sets 'matches' as > 'true' only if the given folio is young. > > If it works in the opposite way, it's a bug that need to be fixed. Please let > me know if I'm missing something. I just read the DAMOS filters part of the documentation for DAMON sysfs interface again. I believe it is explaining the meaning of 'matching' as I intended to, as below: You can write ``Y`` or ``N`` to ``matching`` file to filter out pages that does or does not match to the type, respectively. Then, the scheme's action will not be applied to the pages that specified to be filtered out. But, I found the following example for memcg filter is wrong, as below: For example, below restricts a DAMOS action to be applied to only non-anonymous pages of all memory cgroups except ``/having_care_already``.:: # echo 2 > nr_filters # # filter out anonymous pages echo anon > 0/type echo Y > 0/matching # # further filter out all cgroups except one at '/having_care_already' echo memcg > 1/type echo /having_care_already > 1/memcg_path echo N > 1/matching Specifically, the last line of the commands should write 'Y' instead of 'N' to do what explained. Without the fix, the action will be applied to only non-anonymous pages of 'having_care_already' memcg. This is definitely wrong. I will fix this soon. I'm unsure if this is what made you to believe memcg DAMOS filter is working in the opposite way, though. Thanks, SJ [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi Honggyu, On Sun, 17 Mar 2024 17:36:29 +0900 Honggyu Kim wrote: > Hi SeongJae, > > Thanks for the confirmation. I have a few comments on young filter so > please read the inline comments again. > > On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > > Hi Honggyu, > > > > > > -Original Message- > > > > From: SeongJae Park > > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > > To: Honggyu Kim > > > > Cc: SeongJae Park ; kernel_team > > > > > > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory > > > > management for CXL memory > > > > > > > > Hi Honggyu, > > > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > > wrote: > > > > > > > > > Hi SeongJae, > > > > > > > > > > I've tested it again and found that "young" filter has to be set > > > > > differently as follows. > > > > > - demote action: set "young" filter with "matching" true > > > > > - promote action: set "young" filter with "matching" false > > > > > > > > DAMOS filter is basically for filtering "out" memory regions that > > > > matches to > > > > the condition. Hence in your setup, young pages are not filtered out > > > > from > > > > demote action target. > > > > > > I thought young filter true means "young pages ARE filtered out" for > > > demotion. > > > > You're correct. > > Ack. > > > > > > > > That is, you're demoting pages that "not" young. > > > > > > Your explanation here looks opposite to the previous statement. > > > > Again, you're correct. My intention was "non-young pages are not ..." but > > maybe I was out of my mind and mistakenly removed "non-" part. Sorry for > > the > > confusion. > > No problem. I also think it's quite confusing. > > > > > > > > And vice versa, so you're applying promote to non-non-young (young) > > > > pages. > > > > > > Yes, I understand "promote non-non-young pages" means "promote young > > > pages". > > > This might be understood as "young pages are NOT filtered out" for > > > promotion > > > but it doesn't mean that "old pages are filtered out" instead. > > > And we just rely hot detection only on DAMOS logics such as access > > > frequency > > > and age. Am I correct? > > > > You're correct. > > Ack. But if it doesn't mean that "old pages are filtered out" instead, It does mean that. Here, filtering is exclusive. Hence, "filter-in a type of pages" means "filter-out pages of other types". At least that's the intention. To quote the documentation (https://docs.kernel.org/mm/damon/design.html#filters), Each filter specifies the type of target memory, and whether it should exclude the memory of the type (filter-out), or all except the memory of the type (filter-in). > then do we really need this filter for promotion? If not, maybe should > we create another "old" filter for promotion? As of now, the promotion > is mostly done inaccurately, but the accurate migration is done at > demotion level. Is this based on your theory? Or, a real behavior that you're seeing from your setup? If this is a real behavior, I think that should be a bug that need to be fixed. > To avoid this issue, I feel we should promotion only "young" pages after > filtering "old" pages out. > > > > > > > > I understand this is somewhat complex, but what we have for now. > > > > > > Thanks for the explanation. I guess you mean my filter setup is correct. > > > Is it correct? > > > > Again, you're correct. Your filter setup is what I expected to :) > > Thanks. I see that it works fine, but I would like to have more > discussion about "young" filter. What I think about filter is that if I > apply "young" filter "true" for demotion, then the action applies only > for "young" pages, but the current implementation works opposite. > > I understand the function name of internal implementation is > "damos_pa_filter_out" so the basic action is filtering out, but the > cgroup filter works in the opposite way for now. Does memcg filter works in the opposit
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, Thanks for the confirmation. I have a few comments on young filter so please read the inline comments again. On Wed, 12 Mar 2024 08:53:00 -0800 SeongJae Park wrote: > Hi Honggyu, > > > > -Original Message- > > > From: SeongJae Park > > > Sent: Tuesday, March 12, 2024 3:33 AM > > > To: Honggyu Kim > > > Cc: SeongJae Park ; kernel_team > > > Subject: RE: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management > > > for CXL memory > > > > > > Hi Honggyu, > > > > > > On Mon, 11 Mar 2024 12:51:12 + "honggyu@sk.com" > > > wrote: > > > > > > > Hi SeongJae, > > > > > > > > I've tested it again and found that "young" filter has to be set > > > > differently as follows. > > > > - demote action: set "young" filter with "matching" true > > > > - promote action: set "young" filter with "matching" false > > > > > > DAMOS filter is basically for filtering "out" memory regions that matches > > > to > > > the condition. Hence in your setup, young pages are not filtered out from > > > demote action target. > > > > I thought young filter true means "young pages ARE filtered out" for > > demotion. > > You're correct. Ack. > > > > > That is, you're demoting pages that "not" young. > > > > Your explanation here looks opposite to the previous statement. > > Again, you're correct. My intention was "non-young pages are not ..." but > maybe I was out of my mind and mistakenly removed "non-" part. Sorry for the > confusion. No problem. I also think it's quite confusing. > > > > > And vice versa, so you're applying promote to non-non-young (young) pages. > > > > Yes, I understand "promote non-non-young pages" means "promote young pages". > > This might be understood as "young pages are NOT filtered out" for promotion > > but it doesn't mean that "old pages are filtered out" instead. > > And we just rely hot detection only on DAMOS logics such as access frequency > > and age. Am I correct? > > You're correct. Ack. But if it doesn't mean that "old pages are filtered out" instead, then do we really need this filter for promotion? If not, maybe should we create another "old" filter for promotion? As of now, the promotion is mostly done inaccurately, but the accurate migration is done at demotion level. To avoid this issue, I feel we should promotion only "young" pages after filtering "old" pages out. > > > > > I understand this is somewhat complex, but what we have for now. > > > > Thanks for the explanation. I guess you mean my filter setup is correct. > > Is it correct? > > Again, you're correct. Your filter setup is what I expected to :) Thanks. I see that it works fine, but I would like to have more discussion about "young" filter. What I think about filter is that if I apply "young" filter "true" for demotion, then the action applies only for "young" pages, but the current implementation works opposite. I understand the function name of internal implementation is "damos_pa_filter_out" so the basic action is filtering out, but the cgroup filter works in the opposite way for now. I would like to hear how you think about this. > > > > > > Then, I see that "hot_cold" migrates hot/cold memory correctly. > > > > > > Thank you so much for sharing this great news! My tests also show no bad > > > signal so far. > > > > > > > > > > > Could you please upload the "damon_folio_mkold" patch to LKML? > > > > Then I will rebase our changes based on it and run the redis test again. > > > > > > I will do that soon. > > > > Thanks a lot for sharing the RFC v2 for DAMOS young filter. > > https://lore.kernel.org/damon/20240311204545.47097-1...@kernel.org/ > > > > I will rebase our work based on it and share the result. > > Cool, looking forward to it! Hopefully we will make it be merged into the > mainline by v6.10! I hope so. Thanks for your help! Honggyu
Re: Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hi SeongJae, I couldn't send email to LKML properly due to internal system issues, but it's better now so I will be able to communicate better. On Wed, 6 Mar 2024 19:05:50 -0800 SeongJae Park wrote: > > Hello, > > On Tue, 27 Feb 2024 15:51:20 -0800 SeongJae Park wrote: > > > On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > > > > > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > > > posted at [1]. > > > > > > It says there is no implementation of the demote/promote DAMOS action > > > are made. This RFC is about its implementation for physical address > > > space. > [...] > > Honggyu joined DAMON Beer/Coffee/Tea Chat[1] yesterday, and we discussed > > about > > this patchset in high level. Sharing the summary here for open discussion. > > As > > also discussed on the first version of this patchset[2], we want to make > > single > > action for general page migration with minimum changes, but would like to > > keep > > page level access re-check. We also agreed the previously proposed DAMOS > > filter-based approach could make sense for the purpose. > > > > Because I was anyway planning making such DAMOS filter for not only > > promotion/demotion but other types of DAMOS action, I will start developing > > the > > page level access re-check results based DAMOS filter. Once the > > implementation > > of the prototype is done, I will share the early implementation. Then, > > Honggyu > > will adjust their implementation based on the filter, and run their tests > > again > > and share the results. > > I just posted an RFC patchset for the page level access re-check DAMOS filter: > https://lore.kernel.org/r/20240307030013.47041-1...@kernel.org > > I hope it to help you better understanding and testing the idea. Thanks very much for your work! I will test it based on your changes. Thanks, Honggyu > > > > > [1] https://lore.kernel.org/damon/20220810225102.124459-1...@kernel.org/ > > [2] https://lore.kernel.org/damon/20240118171756.80356-1...@kernel.org > > > Thanks, > SJ > > [...] > > >
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
Hello, On Tue, 27 Feb 2024 15:51:20 -0800 SeongJae Park wrote: > On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > > > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > > posted at [1]. > > > > It says there is no implementation of the demote/promote DAMOS action > > are made. This RFC is about its implementation for physical address > > space. [...] > Honggyu joined DAMON Beer/Coffee/Tea Chat[1] yesterday, and we discussed about > this patchset in high level. Sharing the summary here for open discussion. > As > also discussed on the first version of this patchset[2], we want to make > single > action for general page migration with minimum changes, but would like to keep > page level access re-check. We also agreed the previously proposed DAMOS > filter-based approach could make sense for the purpose. > > Because I was anyway planning making such DAMOS filter for not only > promotion/demotion but other types of DAMOS action, I will start developing > the > page level access re-check results based DAMOS filter. Once the > implementation > of the prototype is done, I will share the early implementation. Then, > Honggyu > will adjust their implementation based on the filter, and run their tests > again > and share the results. I just posted an RFC patchset for the page level access re-check DAMOS filter: https://lore.kernel.org/r/20240307030013.47041-1...@kernel.org I hope it to help you better understanding and testing the idea. > > [1] https://lore.kernel.org/damon/20220810225102.124459-1...@kernel.org/ > [2] https://lore.kernel.org/damon/20240118171756.80356-1...@kernel.org Thanks, SJ [...]
Re: [RFC PATCH v2 0/7] DAMON based 2-tier memory management for CXL memory
On Mon, 26 Feb 2024 23:05:46 +0900 Honggyu Kim wrote: > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > posted at [1]. > > It says there is no implementation of the demote/promote DAMOS action > are made. This RFC is about its implementation for physical address > space. > > > Introduction > > > With the advent of CXL/PCIe attached DRAM, which will be called simply > as CXL memory in this cover letter, some systems are becoming more > heterogeneous having memory systems with different latency and bandwidth > characteristics. They are usually handled as different NUMA nodes in > separate memory tiers and CXL memory is used as slow tiers because of > its protocol overhead compared to local DRAM. > > In this kind of systems, we need to be careful placing memory pages on > proper NUMA nodes based on the memory access frequency. Otherwise, some > frequently accessed pages might reside on slow tiers and it makes > performance degradation unexpectedly. Moreover, the memory access > patterns can be changed at runtime. > > To handle this problem, we need a way to monitor the memory access > patterns and migrate pages based on their access temperature. The > DAMON(Data Access MONitor) framework and its DAMOS(DAMON-based Operation > Schemes) can be useful features for monitoring and migrating pages. > DAMOS provides multiple actions based on DAMON monitoring results and it > can be used for proactive reclaim, which means swapping cold pages out > with DAMOS_PAGEOUT action, but it doesn't support migration actions such > as demotion and promotion between tiered memory nodes. > > This series supports two new DAMOS actions; DAMOS_DEMOTE for demotion > from fast tiers and DAMOS_PROMOTE for promotion from slow tiers. This > prevents hot pages from being stuck on slow tiers, which makes > performance degradation and cold pages can be proactively demoted to > slow tiers so that the system can increase the chance to allocate more > hot pages to fast tiers. > > The DAMON provides various tuning knobs but we found that the proactive > demotion for cold pages is especially useful when the system is running > out of memory on its fast tier nodes. > > Our evaluation result shows that it reduces the performance slowdown > compared to the default memory policy from 15~17% to 4~5% when the > system runs under high memory pressure on its fast tier DRAM nodes. > > > DAMON configuration > === > > The specific DAMON configuration doesn't have to be in the scope of this > patch series, but some rough idea is better to be shared to explain the > evaluation result. > > The DAMON provides many knobs for fine tuning but its configuration file > is generated by HMSDK[2]. It includes gen_config.py script that > generates a json file with the full config of DAMON knobs and it creates > multiple kdamonds for each NUMA node when the DAMON is enabled so that > it can run hot/cold based migration for tiered memory. I was feeling a bit confused from here since DAMON doesn't receive parameters via a file. To my understanding, the 'configuration file' means the input file for DAMON user-space tool, damo, not DAMON. Just a trivial thing, but making it clear if possible could help readers in my opinion. > > > Evaluation Workload > === > > The performance evaluation is done with redis[3], which is a widely used > in-memory database and the memory access patterns are generated via > YCSB[4]. We have measured two different workloads with zipfian and > latest distributions but their configs are slightly modified to make > memory usage higher and execution time longer for better evaluation. > > The idea of evaluation using these demote and promote actions covers > system-wide memory management rather than partitioning hot/cold pages of > a single workload. The default memory allocation policy creates pages > to the fast tier DRAM node first, then allocates newly created pages to > the slow tier CXL node when the DRAM node has insufficient free space. > Once the page allocation is done then those pages never move between > NUMA nodes. It's not true when using numa balancing, but it is not the > scope of this DAMON based 2-tier memory management support. > > If the working set of redis can be fit fully into the DRAM node, then > the redis will access the fast DRAM only. Since the performance of DRAM > only is faster than partially accessing CXL memory in slow tiers, this > environment is not useful to evaluate this patch series. > > To make pages of redis be distributed across fast DRAM node and slow > CXL node to evaluate our demote and promote actions, we pre-allocate > some cold memory externally using mmap and memset before launching > redis-server. We assumed that there are enough amount of cold memory in > datacenters as TMO[5] and TPP[6] papers mentioned. > > The evaluation sequence is as follows. > > 1. Turn on DAMON with DAMOS_DEMOTE action
