Re: [PATCH v23 01/15] mm: Introduce Data Access MONitor (DAMON)
On Wed, 23 Dec 2020 14:49:57 -0800 Shakeel Butt wrote: > On Wed, Dec 23, 2020 at 8:34 AM SeongJae Park wrote: > [snip] > > > Overall the patch looks good to me. Two concerns I have are if we > > > should damon_callback here or with the real user and the regions part > > > of primitive abstraction. For the first one, I don't have any strong > > > opinion but for the second one I do. > > > > I'd like to keep 'damon_callback' part here, to let API users know how the > > monitoring result will be available to them. > > > > For the 'regions' part, I will rename relevant things as below in the next > > version, to reduce any confusion. > > > > init_target_regions() -> init() > > update_target_regions() -> update() > > regions_update_interval -> update_interval > > last_regions_update -> last_update > > > > > > > > More specifically the question is if sampling and adaptive region > > > adjustment are general enough to be part of core monitoring context? > > > Can you give an example of a different primitive/use-case where these > > > would be beneficial. > > > > I think all adress spaces having some spatial locality and monitoring > > requests > > that need to have upper-bound overhead and best-effort accuracy could get > > benefit from it. The primitives targetting 'virtual address spaces' and the > > 'physical address space' clearly showed the benefit. > > I am still not much convinced on the 'physical address space' use-case > or the way you are presenting it. I understand the concern. I also once thought the mechanism might not work well for the physical address space because we cannot expect much spatial locality in the space. However, it turned out that there is some (temporal) spatial locality that enough to make DAMON work reasonably well. The word, 'reasonably well' might be controversial. With the mechanism, DAMON provides only 'best-effort' accuracy, rather than 100% accuracy. Our goal is to make the information accurate enough only for DRAM-centric optimizations. I'd like to also note that there are knobs that you can use to make minimum quality higher (nr_min_regions) while setting the upperbound of the monitoring overhead (nr_max_regions). What I can say for now is that we ran DAMON for physical address space of our production systems (shared detail in the 'Real-workd User Story' section of coverletter[1]) and the result was reasonable enough to convince the owner of the systems. [1] https://lore.kernel.org/linux-mm/20201215115448.25633-1-sjp...@amazon.com/ > Anyways I think we start with what you have and if in future there is a > use-case where regions adjustment does not make sense, we can change it then. 100% agreed, and thank you for understanding my argument. Thanks, SeongJae Park
Re: [PATCH v23 01/15] mm: Introduce Data Access MONitor (DAMON)
On Wed, Dec 23, 2020 at 8:34 AM SeongJae Park wrote: [snip] > > Overall the patch looks good to me. Two concerns I have are if we > > should damon_callback here or with the real user and the regions part > > of primitive abstraction. For the first one, I don't have any strong > > opinion but for the second one I do. > > I'd like to keep 'damon_callback' part here, to let API users know how the > monitoring result will be available to them. > > For the 'regions' part, I will rename relevant things as below in the next > version, to reduce any confusion. > > init_target_regions() -> init() > update_target_regions() -> update() > regions_update_interval -> update_interval > last_regions_update -> last_update > > > > > More specifically the question is if sampling and adaptive region > > adjustment are general enough to be part of core monitoring context? > > Can you give an example of a different primitive/use-case where these > > would be beneficial. > > I think all adress spaces having some spatial locality and monitoring requests > that need to have upper-bound overhead and best-effort accuracy could get > benefit from it. The primitives targetting 'virtual address spaces' and the > 'physical address space' clearly showed the benefit. I am still not much convinced on the 'physical address space' use-case or the way you are presenting it. Anyways I think we start with what you have and if in future there is a use-case where regions adjustment does not make sense, we can change it then.
Re: [PATCH v23 01/15] mm: Introduce Data Access MONitor (DAMON)
Thanks for the valuable comments, Shakeel! On Wed, 23 Dec 2020 07:11:12 -0800 Shakeel Butt wrote: > First I would like you to prune your To/CC list. I will remove people not directly related with this work and didn't comment to this series yet. > > On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park wrote: > > > > From: SeongJae Park > > > > DAMON is a data access monitoring framework for the Linux kernel. The > > core mechanisms of DAMON make it > > > > - accurate (the monitoring output is useful enough for DRAM level > >performance-centric memory management; It might be inappropriate for > >CPU cache levels, though), > > - light-weight (the monitoring overhead is normally low enough to be > >applied online), and > > - scalable (the upper-bound of the overhead is in constant range > >regardless of the size of target workloads). > > > > Using this framework, hence, we can easily write efficient kernel space > > data access monitoring applications. For example, the kernel's memory > > management mechanisms can make advanced decisions using this. > > Experimental data access aware optimization works that incurring high > > access monitoring overhead could implemented again on top of this. > > *could again be implemented on* Good eye! I will fix this in the next version. > > > > > Due to its simple and flexible interface, providing user space interface > > would be also easy. Then, user space users who have some special > > workloads can write personalized applications for better understanding > > and optimizations of their workloads and systems. > > > > --- > > > > Nevertheless, this commit is defining and implementing only basic access > > check part without the overhead-accuracy handling core logic. The basic > > access check is as below. > > > > The output of DAMON says what memory regions are how frequently accessed > > for a given duration. The resolution of the access frequency is > > controlled by setting ``sampling interval`` and ``aggregation > > interval``. In detail, DAMON checks access to each page per ``sampling > > interval`` and aggregates the results. In other words, counts the > > number of the accesses to each region. After each ``aggregation > > interval`` passes, DAMON calls callback functions that previously > > registered by users so that users can read the aggregated results and > > then clears the results. This can be described in below simple > > pseudo-code:: > > > > while monitoring_on: > > for page in monitoring_target: > > if accessed(page): > > nr_accesses[page] += 1 > > if time() % aggregation_interval == 0: > > for callback in user_registered_callbacks: > > callback(monitoring_target, nr_accesses) > > for page in monitoring_target: > > nr_accesses[page] = 0 > > sleep(sampling interval) > > > > The above is a good example and I was hoping to see the almost same actual > code. Oh, this doesn't explain about the 'update'. I will update this in the next version. > > > The target regions constructed at the beginning of the monitoring and > > updated after each ``regions_update_interval``, because the target > > regions could be dynamically changed (e.g., mmap() or memory hotplug). > > The monitoring overhead of this mechanism will arbitrarily increase as > > the size of the target workload grows. > > > > The basic monitoring primitives for actual access check and dynamic > > target regions construction aren't in the core part of DAMON. Instead, > > it allows users to implement their own primitives that optimized for > > 'that are optimized for' Good catch! I will fix this in the next version. > > > their use case and configure DAMON to use those. In other words, users > > cannot use current version of DAMON without some additional works. > > > > Following commits will implement the core mechanisms for the > > overhead-accuracy control and default primitives implementations. > > > > Signed-off-by: SeongJae Park > > Reviewed-by: Leonard Foerster > > --- > > include/linux/damon.h | 167 ++ > > mm/Kconfig| 2 + > > mm/Makefile | 1 + > > mm/damon/Kconfig | 15 ++ > > mm/damon/Makefile | 3 + > > mm/damon/core.c | 316 ++ > > 6 files changed, 504 insertions(+) > > create mode 100644 include/linux/damon.h > > create mode 100644 mm/damon/Kconfig > > create mode 100644 mm/damon/Makefile > > create mode 100644 mm/damon/core.c > > > > diff --git a/include/linux/damon.h b/include/linux/damon.h > > new file mode 100644 > > index ..387fa4399fc8 > > --- /dev/null > > +++ b/include/linux/damon.h > > @@ -0,0 +1,167 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +/* > > + * DAMON api > > + * > > + * Author: SeongJae Park > > + */ > > + > > +#ifndef _DAMON_H_ > > +#define _DAMON_H_ > > + > > +#include > > +#include
Re: [PATCH v23 01/15] mm: Introduce Data Access MONitor (DAMON)
First I would like you to prune your To/CC list. On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park wrote: > > From: SeongJae Park > > DAMON is a data access monitoring framework for the Linux kernel. The > core mechanisms of DAMON make it > > - accurate (the monitoring output is useful enough for DRAM level >performance-centric memory management; It might be inappropriate for >CPU cache levels, though), > - light-weight (the monitoring overhead is normally low enough to be >applied online), and > - scalable (the upper-bound of the overhead is in constant range >regardless of the size of target workloads). > > Using this framework, hence, we can easily write efficient kernel space > data access monitoring applications. For example, the kernel's memory > management mechanisms can make advanced decisions using this. > Experimental data access aware optimization works that incurring high > access monitoring overhead could implemented again on top of this. *could again be implemented on* > > Due to its simple and flexible interface, providing user space interface > would be also easy. Then, user space users who have some special > workloads can write personalized applications for better understanding > and optimizations of their workloads and systems. > > --- > > Nevertheless, this commit is defining and implementing only basic access > check part without the overhead-accuracy handling core logic. The basic > access check is as below. > > The output of DAMON says what memory regions are how frequently accessed > for a given duration. The resolution of the access frequency is > controlled by setting ``sampling interval`` and ``aggregation > interval``. In detail, DAMON checks access to each page per ``sampling > interval`` and aggregates the results. In other words, counts the > number of the accesses to each region. After each ``aggregation > interval`` passes, DAMON calls callback functions that previously > registered by users so that users can read the aggregated results and > then clears the results. This can be described in below simple > pseudo-code:: > > while monitoring_on: > for page in monitoring_target: > if accessed(page): > nr_accesses[page] += 1 > if time() % aggregation_interval == 0: > for callback in user_registered_callbacks: > callback(monitoring_target, nr_accesses) > for page in monitoring_target: > nr_accesses[page] = 0 > sleep(sampling interval) > The above is a good example and I was hoping to see the almost same actual code. > The target regions constructed at the beginning of the monitoring and > updated after each ``regions_update_interval``, because the target > regions could be dynamically changed (e.g., mmap() or memory hotplug). > The monitoring overhead of this mechanism will arbitrarily increase as > the size of the target workload grows. > > The basic monitoring primitives for actual access check and dynamic > target regions construction aren't in the core part of DAMON. Instead, > it allows users to implement their own primitives that optimized for 'that are optimized for' > their use case and configure DAMON to use those. In other words, users > cannot use current version of DAMON without some additional works. > > Following commits will implement the core mechanisms for the > overhead-accuracy control and default primitives implementations. > > Signed-off-by: SeongJae Park > Reviewed-by: Leonard Foerster > --- > include/linux/damon.h | 167 ++ > mm/Kconfig| 2 + > mm/Makefile | 1 + > mm/damon/Kconfig | 15 ++ > mm/damon/Makefile | 3 + > mm/damon/core.c | 316 ++ > 6 files changed, 504 insertions(+) > create mode 100644 include/linux/damon.h > create mode 100644 mm/damon/Kconfig > create mode 100644 mm/damon/Makefile > create mode 100644 mm/damon/core.c > > diff --git a/include/linux/damon.h b/include/linux/damon.h > new file mode 100644 > index ..387fa4399fc8 > --- /dev/null > +++ b/include/linux/damon.h > @@ -0,0 +1,167 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * DAMON api > + * > + * Author: SeongJae Park > + */ > + > +#ifndef _DAMON_H_ > +#define _DAMON_H_ > + > +#include > +#include > +#include > + > +struct damon_ctx; > + > +/** > + * struct damon_primitive Monitoring primitives for given use cases. > + * > + * @init_target_regions: Constructs initial monitoring target regions. > + * @update_target_regions: Updates monitoring target regions. The 'region' here still does not feel right. I prefer to rename the above two to just init() and update(). > + * @prepare_access_checks: Prepares next access check of target regions. > + * @check_accesses:Checks the access of target regions. > + * @reset_aggregated: Resets aggregated accesses monitoring
[PATCH v23 01/15] mm: Introduce Data Access MONitor (DAMON)
From: SeongJae Park DAMON is a data access monitoring framework for the Linux kernel. The core mechanisms of DAMON make it - accurate (the monitoring output is useful enough for DRAM level performance-centric memory management; It might be inappropriate for CPU cache levels, though), - light-weight (the monitoring overhead is normally low enough to be applied online), and - scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads). Using this framework, hence, we can easily write efficient kernel space data access monitoring applications. For example, the kernel's memory management mechanisms can make advanced decisions using this. Experimental data access aware optimization works that incurring high access monitoring overhead could implemented again on top of this. Due to its simple and flexible interface, providing user space interface would be also easy. Then, user space users who have some special workloads can write personalized applications for better understanding and optimizations of their workloads and systems. --- Nevertheless, this commit is defining and implementing only basic access check part without the overhead-accuracy handling core logic. The basic access check is as below. The output of DAMON says what memory regions are how frequently accessed for a given duration. The resolution of the access frequency is controlled by setting ``sampling interval`` and ``aggregation interval``. In detail, DAMON checks access to each page per ``sampling interval`` and aggregates the results. In other words, counts the number of the accesses to each region. After each ``aggregation interval`` passes, DAMON calls callback functions that previously registered by users so that users can read the aggregated results and then clears the results. This can be described in below simple pseudo-code:: while monitoring_on: for page in monitoring_target: if accessed(page): nr_accesses[page] += 1 if time() % aggregation_interval == 0: for callback in user_registered_callbacks: callback(monitoring_target, nr_accesses) for page in monitoring_target: nr_accesses[page] = 0 sleep(sampling interval) The target regions constructed at the beginning of the monitoring and updated after each ``regions_update_interval``, because the target regions could be dynamically changed (e.g., mmap() or memory hotplug). The monitoring overhead of this mechanism will arbitrarily increase as the size of the target workload grows. The basic monitoring primitives for actual access check and dynamic target regions construction aren't in the core part of DAMON. Instead, it allows users to implement their own primitives that optimized for their use case and configure DAMON to use those. In other words, users cannot use current version of DAMON without some additional works. Following commits will implement the core mechanisms for the overhead-accuracy control and default primitives implementations. Signed-off-by: SeongJae Park Reviewed-by: Leonard Foerster --- include/linux/damon.h | 167 ++ mm/Kconfig| 2 + mm/Makefile | 1 + mm/damon/Kconfig | 15 ++ mm/damon/Makefile | 3 + mm/damon/core.c | 316 ++ 6 files changed, 504 insertions(+) create mode 100644 include/linux/damon.h create mode 100644 mm/damon/Kconfig create mode 100644 mm/damon/Makefile create mode 100644 mm/damon/core.c diff --git a/include/linux/damon.h b/include/linux/damon.h new file mode 100644 index ..387fa4399fc8 --- /dev/null +++ b/include/linux/damon.h @@ -0,0 +1,167 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * DAMON api + * + * Author: SeongJae Park + */ + +#ifndef _DAMON_H_ +#define _DAMON_H_ + +#include +#include +#include + +struct damon_ctx; + +/** + * struct damon_primitive Monitoring primitives for given use cases. + * + * @init_target_regions: Constructs initial monitoring target regions. + * @update_target_regions: Updates monitoring target regions. + * @prepare_access_checks: Prepares next access check of target regions. + * @check_accesses:Checks the access of target regions. + * @reset_aggregated: Resets aggregated accesses monitoring results. + * @target_valid: Determine if the target is valid. + * @cleanup: Cleans up the context. + * + * DAMON can be extended for various address spaces and usages. For this, + * users should register the low level primitives for their target address + * space and usecase via the _ctx.primitive. Then, the monitoring thread + * calls @init_target_regions and @prepare_access_checks before starting the + * monitoring, @update_target_regions after each + * _ctx.regions_update_interval, and @check_accesses, @target_valid and + *
