Hi SeonJae, On 4/26/2026 11:52 PM, SeongJae Park wrote: > TL; DR > ====== > > Extend DAMON for monitoring general data attributes other than accesses. > This is for enabling light-weight page type (e.g., belonging cgroup) > aware monitoring in short term. In long term, this will help extending > DAMON for multiple access events capture primitives (e.g., page faults > and PMU) and eventually pivotting DAMON to a "Data Attributes Monitoring > and Operations eNgine" in long term.
Very interesting. Looking forward to seeing this in upstream. > > Background: High Cost of Page Level Properties Monitoring > ========================================================= > > DAMON is initially introduced as a Data Access MONitor. It has been > extended for not only access monitoring but also data access-aware > system operations (DAMOS). But still the monitoring part is only for > data accesses. > > Data access patterns is good information, but some users need more > holistic views. Particularly, users want to show the access pattern > information together with the types of the memory. For example, users > who work for making huge pages efficiently want to know how much of > DAMON-found hot/cold regions are backed by huge pages. Users who run > multiple workloads with different cgroups want to know how much of > DAMON-found hot/cold regions belong to specific cgroups. > > For the user demand, we developed a DAMOS extension for page level > properties based monitoring [1], which has landed on 6.14. Using the > feature, users can inform the page level data properties that they are > interested in, in a flexible format that uses DAMOS filters. Then, > DAMON applies the filters to each folio of the entire DAMON region and > lets users know how many bytes of memory in each DAMON region passed the > given filters. > > This gives page level detailed and deterministic information to users. > But, because the operation is done at page level, the overhead is > proportional to the memory size. It was useful for test or debugging > purposes on a small number of machines. But it was obviously too heavy > to be enabled always on all machines running the real user workloads. > For real world workloads, it was recommended to use the feature with > user-space controlled sampling approaches. For example, users could do > the page level monitoring only once per hour, on randomly selected one > percent of machines of their fleet. If the runtime and the size of the > fleet is long and big enough, it should provide statistically meaningful > data. > > But users are too busy to implement such controls on their own. > > Data Attributes Monitoring > ========================== > > Extend DAMON to monitor not only data accesses, but also general data > attributes. Do the extension while keeping the main promise of DAMON, > the bounded and best-effort minimum overhead. > > Allow users to specify what data attributes in addition to the data > access they want to monitor. Users can install one 'data probe' per > data attribute of their interest for this purpose. The 'data probe' > should be able to be applied to any memory, and determine if the given > memory has the appropriate data attribute. E.g., if memory of physical > address 42 belongs to cgroup A. Each 'data probe' is configured with > filters that are very similar to the DAMOS filters. > > When DAMON checks if each sampling address memory of each region is > accessed since the last check, it applies data probes if registered. > Same to the number of access check-positive samples accounting > (nr_accesses), it accounts the number of each data probe-positive > samples in another per-region counters array, namely 'probe_hits'. When > DAMON resets nr_accesses every aggregation interval, it resets > 'probe_hits' together. > > Users can read 'probe_hits' just before the values are reset. In this > way, users can know how many hot/cold memory regions have data > attributes of their interest. E.g., 30 percent of this system's hot > memory is belonging to cgroup A and 80 percent of the hot cgroup A > memory is backed by huge pages. > > Patches Sequence > ================ > > First eight patches implement the core feature, interface and the > working support. Patch 1 introduces data probe data structure, namely > damon_probe. Patch 2 extends damon_ctx for installing data probes. > Patch 3 introduces another data structure for filters of each data > probe, namely damon_filter. Patch 4 updates damon_ctx commit function > to handle the probes. Patch 5 extends damon_region for the per-region > per-probe positive samples counter, namely probe_hits. Patch 6 extends > damon_operations for applying probes on the underlying DAMON operations > implementation. Patch 7 updates kdamond_fn() to invoke the probes > applying callback. Patch 8 finally implements the probes support on > paddr ops. > > Eight changes for user interface (patches 9-16) come next. Patches 9-13 > implements sysfs directories and files for setting data probes, namely > probes directory, probe directory, filters directory, filter directory > and filter directory internal files, respectively. Patch 14 connects > the user inputs that are made via the sysfs files to DAMON core. > Patch 15 implements sysfs files for showing the per-region per-probe > positive samples count, namely probe_hits. Patch 16 introduces a new > tracepoint for showing the counts via tracefs. > > Patch 14 adds a selftest for the sysfs files. > > Patches 15 and 16 documents the design and usage of the new feature, > respectively. > > Discussions > =========== > > This allows the page properties monitoring with overhead that is low > enough to be enabled always on real world workloads. Because the > sampling time for access check is reused for data attributes check, the > upper-bounded and best-effort minimum overhead of DAMON is kept. > Because the sampling memory for access check is reused for data > attributes check, additional overhead is minimum. > > Still DAMOS-based page level properties monitoring should be useful, > because it provides a deterministic page level information. When in > doubt of the sampling based information, running DAMOS-based one > together and comparing the results would be useful, for debugging and > tuning. > > Plan for Dropping RFC tag > ========================= > > The user ABI for reading probe_hits is not yet convincing. It is > exposed to users by a tracepoint and new sysfs file. For the > tracepoint, a new one namely damon:damon_aggregated_v2 is introduced. > The name is not convincing, and its internal mechanism seems to have > room to be improved before dropping RFC. For the sysfs, a file under > the DAMOS-tried region directory namely 'probe_hits' is added. Reading > it returns four probe_hits values with ',' as a separator. With the > maximum number of data probes, this should work. This can make future > changes of the limit difficult. I will try to find a better way before > dropping the RFC tag. Maybe 'probe_hits/' directory having files of > name '0' to 'N-1' for each of user-registered 'N' data probes. > > I'm currently hoping to drop the RFC tag by 7.2-rc1. > > Future Works: Short Term > ======================== > > This series is introducing only a single type of data attribute: > anonymous page. Once this is landed, I will extend it for > cgroup-belonging, so that we can do cgroup-level monitoring with low > overhead. After that, I may further work on supporting all DAMOS filter > types. And as demands are found, we could extend the types. > > This version of implementation is limiting the maximum number of data > probes to four. I will try to find a way to remove the limit in future, > if it is easy to do. I personally think it should be enough for common > use cases, though, and therefore not giving high priority at the moment. > > Future Works: Long Term > ======================= > > There are user requests for extending DAMON with detailed access > information, for example, per-CPUs/threads/read/writes monitoring. For > that, I was working [2] on extending DAMON to use page fault events as > another access check primitives, and making the infrastructure flexible > for future use of yet another access check primitive. Actually there is > another ongoing work [3] for extending DAMON with PMU events. The > motivation of the work is reducing the overhead, though. > > In my work [2], I was introducing a new interface for access sampling > primitives control. Now I think this data probe interface can be used > for that, too. That is, data access becomes just one type of data > attribute. Also, pg_idle-confirmed access, page fault-confirmed access, > and PMU event-confirmed access will be different types of data > attributes. > > The regions adjustment mechanism is currently working based on the > access information. That's because DAMON is designed for data access > monitoring. That is, data access information is the primary interest, > and therefore DAMON adjusts regions in a way that can best-present the > information. > > Once data access becomes just one of data attributes, there is no reason > to think data access that special. There might be some users not > interested in access at all but want to know the location of memory of > specific type. Data probes interface will allow doing that. Further, > we could extend the interface to let users set any data attribute as the > 'primary' attribute. Then, DAMON will split and merge regions in a way > that can best-present the 'primary' attributes. > > DAMOS will also be extended, to specify targets based on not only the > data access pattern, but all user-registered data attributes. From this > stage, we may be able to call DAMON as a "Data Attributes Monitoring and > Operations eNgine". > > [1] https://lore.kernel.org/[email protected] > [2] https://lore.kernel.org/[email protected]/ > [3] https://lore.kernel.org/[email protected] > > SeongJae Park (19): > mm/damon/core: introduce struct damon_probe > mm/damon/core: embed damon_probe objects in damon_ctx > mm/damon/core: introduce damon_filter > mm/damon/core: commit probes > mm/damon/core: introduce damon_region->probe_hits > mm/damon/core: introduce damon_ops->apply_probes > mm/damon/core: do data attributes monitoring > mm/damon/paddr: support data attributes monitoring > mm/damon/sysfs: implement probes dir > mm/damon/sysfs: implement probe dir > mm/damon/sysfs: implement filters directory > mm/damon/sysfs: implement filter dir > mm/damon/sysfs: implement filter dir files > mm/damon/sysfs: setup probes on DAMON core API parameters > mm/damon/sysfs-schemes: implement tried_region/probe_hits file > mm/damon: trace probe_hits > selftests/damon/sysfs.sh: test probes dir > Docs/mm/damon/design: document data attributes monitoring > Docs/admin-guide/mm/damon/usage: document data attributes monitoring > > Documentation/admin-guide/mm/damon/usage.rst | 44 +- > Documentation/mm/damon/design.rst | 37 ++ > include/linux/damon.h | 60 +++ > include/trace/events/damon.h | 41 ++ > mm/damon/core.c | 182 +++++++ > mm/damon/paddr.c | 45 ++ > mm/damon/sysfs-schemes.c | 30 ++ > mm/damon/sysfs.c | 502 +++++++++++++++++++ > tools/testing/selftests/damon/sysfs.sh | 48 ++ > 9 files changed, 982 insertions(+), 7 deletions(-) > > > base-commit: 8f22aa2e28454419ed2031119ad32ea4a6c9f1f1 My main concern is about potential pollution of sysfs. DAMON is already complex to set up, with a lot of knobs. Adding more configuration options may make admin's job more complex. Do you plan to support this extension in damo user space? -- Asier Gutierrez Huawei
