Re: [PATCH v2 2/2] core-api/memory-hotplug.rst: divide Locking Internal section by different locks
On Thu, Dec 06, 2018 at 08:26:22AM +0800, Wei Yang wrote: > Currently locking for memory hotplug is a little complicated. > > Generally speaking, we leverage the two global lock: > > * device_hotplug_lock > * mem_hotplug_lock > > to serialise the process. > > While for the long term, we are willing to have more fine-grained lock > to provide higher scalability. > > This patch divides Locking Internal section based on these two global > locks to help readers to understand it. Also it adds some new finding to > enrich it. > > [David: words arrangement] > > Signed-off-by: Wei Yang Reviewd-by: Mike Rapoport > --- > v2: adjustment based on David and Mike comment > --- > Documentation/core-api/memory-hotplug.rst | 27 --- > 1 file changed, 24 insertions(+), 3 deletions(-) > > diff --git a/Documentation/core-api/memory-hotplug.rst > b/Documentation/core-api/memory-hotplug.rst > index de7467e48067..51d477ad4b80 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification > queue. > Locking Internals > = > > +In addition to fine grained locks like pgdat_resize_lock, there are three > locks > +involved > + > +- device_hotplug_lock > +- mem_hotplug_lock > +- device_lock > + > +Currently, they are twisted together for all kinds of reasons. The following > +part is divided into device_hotplug_lock and mem_hotplug_lock parts > +respectively to describe those tricky situations. > + > +device_hotplug_lock > +- > + > When adding/removing memory that uses memory block devices (i.e. ordinary > RAM), > the device_hotplug_lock should be held to: > > @@ -111,13 +125,20 @@ As the device is visible to user space before taking > the device_lock(), this > can result in a lock inversion. > > onlining/offlining of memory should be done via device_online()/ > -device_offline() - to make sure it is properly synchronized to actions > -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect > online_type) > +device_offline() - to make sure it is properly synchronized to actions via > +sysfs. Even if mem_hotplug_lock is used to protect the process, because of > the > +lock inversion described above, holding device_hotplug_lock is still advised > +(to e.g. protect online_type) > + > +mem_hotplug_lock > +- > > When adding/removing/onlining/offlining memory or adding/removing > heterogeneous/device memory, we should always hold the mem_hotplug_lock in > write mode to serialise memory hotplug (e.g. access to global/zone > -variables). > +variables). Currently, we take advantage of this to serialise sparsemem's > +mem_section handling in sparse_add_one_section() and > +sparse_remove_one_section(). > > In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > mode allows for a quite efficient get_online_mems/put_online_mems > -- > 2.15.1 > -- Sincerely yours, Mike.
Re: [PATCH 2/2] core-api/memory-hotplug.rst: divide Locking Internal section by different locks
On Wed, Dec 05, 2018 at 10:34:26AM +0800, Wei Yang wrote: > Currently locking for memory hotplug is a little complicated. > > Generally speaking, we leverage the two global lock: > > * device_hotplug_lock > * mem_hotplug_lock > > to serialise the process. > > While for the long term, we are willing to have more fine-grained lock > to provide higher scalability. > > This patch divides Locking Internal section based on these two global > locks to help readers to understand it. Also it adds some new finding to > enrich it. > > [David: words arrangement] > > Signed-off-by: Wei Yang > --- > Documentation/core-api/memory-hotplug.rst | 27 --- > 1 file changed, 24 insertions(+), 3 deletions(-) > > diff --git a/Documentation/core-api/memory-hotplug.rst > b/Documentation/core-api/memory-hotplug.rst > index de7467e48067..95662b283328 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification > queue. > Locking Internals > = > > +There are three locks involved in memory-hotplug, two global lock and one > local typo: ^locks > +lock: > + > +- device_hotplug_lock > +- mem_hotplug_lock > +- device_lock > + > +Currently, they are twisted together for all kinds of reasons. The following > +part is divided into device_hotplug_lock and mem_hotplug_lock parts > +respectively to describe those tricky situations. > + > +device_hotplug_lock > +- > + > When adding/removing memory that uses memory block devices (i.e. ordinary > RAM), > the device_hotplug_lock should be held to: > > @@ -111,13 +125,20 @@ As the device is visible to user space before taking > the device_lock(), this > can result in a lock inversion. > > onlining/offlining of memory should be done via device_online()/ > -device_offline() - to make sure it is properly synchronized to actions > -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect > online_type) > +device_offline() - to make sure it is properly synchronized to actions via > +sysfs. Even mem_hotplug_lock is used to protect the process, because of the I think it should be "Even if mem_hotplug_lock ..." > +lock inversion described above, holding device_hotplug_lock is still advised > +(to e.g. protect online_type) > + > +mem_hotplug_lock > +- > > When adding/removing/onlining/offlining memory or adding/removing > heterogeneous/device memory, we should always hold the mem_hotplug_lock in > write mode to serialise memory hotplug (e.g. access to global/zone > -variables). > +variables). Currently, we take advantage of this to serialise sparsemem's > +mem_section handling in sparse_add_one_section() and > +sparse_remove_one_section(). > > In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > mode allows for a quite efficient get_online_mems/put_online_mems > -- > 2.15.1 > -- Sincerely yours, Mike.
Re: [PATCH 1/2] admin-guide/memory-hotplug.rst: remove locking internal part from admin-guide
On Wed, Dec 05, 2018 at 09:03:24AM +0100, David Hildenbrand wrote: > On 05.12.18 03:34, Wei Yang wrote: > > Locking Internal section exists in core-api documentation, which is more > > suitable for this. > > > > This patch removes the duplication part here. > > > > Signed-off-by: Wei Yang > > --- > > Documentation/admin-guide/mm/memory-hotplug.rst | 40 > > - > > 1 file changed, 40 deletions(-) > > > > diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst > > b/Documentation/admin-guide/mm/memory-hotplug.rst > > index 5c4432c96c4b..241f4ce1e387 100644 > > --- a/Documentation/admin-guide/mm/memory-hotplug.rst > > +++ b/Documentation/admin-guide/mm/memory-hotplug.rst > > @@ -392,46 +392,6 @@ Need more implementation yet > > - Notification completion of remove works by OS to firmware. > > - Guard from remove if not yet. [ ... ] > > Future Work > > === > > > > > > I reported this yesterday to Jonathan and Mike > > https://lkml.org/lkml/2018/12/3/340 Somehow I've missed it... > Anyhow > > Reviewed-by: David Hildenbrand Acked-by: Mike Rapoport > > -- > > Thanks, > > David / dhildenb > -- Sincerely yours, Mike.
[PATCH] docs/core-api: make mm-api.rst more structured
The mm-api.rst covers variety of memory management APIs under "More Memory Management Functions" section. The descriptions included there are in a random order there are quite a few of them which makes the section too long. Regrouping the documentation by subject and splitting the long "More Memory Management Functions" section into several smaller sections makes the generated html more usable. Signed-off-by: Mike Rapoport --- Documentation/core-api/mm-api.rst | 29 - 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index 5ce1ec1..c81e754 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -49,8 +49,14 @@ The Slab Cache .. kernel-doc:: mm/util.c :functions: kfree_const kvmalloc_node kvfree -More Memory Management Functions - +Virtually Contiguous Mappings += + +.. kernel-doc:: mm/vmalloc.c + :export: + +File Mapping and Page Cache +=== .. kernel-doc:: mm/readahead.c :export: @@ -58,23 +64,28 @@ More Memory Management Functions .. kernel-doc:: mm/filemap.c :export: -.. kernel-doc:: mm/memory.c +.. kernel-doc:: mm/page-writeback.c :export: -.. kernel-doc:: mm/vmalloc.c +.. kernel-doc:: mm/truncate.c :export: -.. kernel-doc:: mm/page_alloc.c - :internal: +Memory pools + .. kernel-doc:: mm/mempool.c :export: +DMA pools += + .. kernel-doc:: mm/dmapool.c :export: -.. kernel-doc:: mm/page-writeback.c - :export: +More Memory Management Functions + -.. kernel-doc:: mm/truncate.c +.. kernel-doc:: mm/memory.c :export: + +.. kernel-doc:: mm/page_alloc.c -- 2.7.4
Re: [PATCH] Link the memory allocation guide from the MM docs
On Mon, Nov 19, 2018 at 08:00:49AM -0800, Matthew Wilcox wrote: > I just went looking for the memory allocation guide in the MM docs instead > of in the core API. For the benefit of the next person who makes that > mistake, link to it from the MM docs. > > Signed-off-by: Matthew Wilcox Acked-by: Mike Rapoport > > diff --git a/Documentation/core-api/memory-allocation.rst > b/Documentation/core-api/memory-allocation.rst > index f8bb9aa120c4..8954a88ff5b7 100644 > --- a/Documentation/core-api/memory-allocation.rst > +++ b/Documentation/core-api/memory-allocation.rst > @@ -1,3 +1,5 @@ > +.. _memory-allocation: > + > === > Memory Allocation Guide > === > diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst > index c4ded22197ca..2b3ab3a1ccf3 100644 > --- a/Documentation/vm/index.rst > +++ b/Documentation/vm/index.rst > @@ -2,7 +2,9 @@ > Linux Memory Management Documentation > = > > -This is a collection of documents about Linux memory management (mm) > subsystem. > +This is a collection of documents about the Linux memory management (mm) > +subsystem. If you are looking for advice on simply allocating memory, > +see the :ref:`memory-allocation`. > > User guides for MM features > === > -- Sincerely yours, Mike.
[PATCH] docs/mm: update kmalloc kernel-doc description
Add references to GFP documentation and the memory-allocation.rst and remove
GFP_USER, GFP_DMA and GFP_NOIO descriptions.
While on it slightly change the formatting so that the list of GFP flags
will be rendered as "description" in the generated html.
Signed-off-by: Mike Rapoport
---
Probably this should go via the -mm tree as it touches include/linux/slab.h
Documentation/core-api/memory-allocation.rst | 2 +
include/linux/slab.h | 55 ++--
2 files changed, 29 insertions(+), 28 deletions(-)
diff --git a/Documentation/core-api/memory-allocation.rst
b/Documentation/core-api/memory-allocation.rst
index f8bb9aa..39f35eb 100644
--- a/Documentation/core-api/memory-allocation.rst
+++ b/Documentation/core-api/memory-allocation.rst
@@ -1,3 +1,5 @@
+.. _memory_allocation:
+
===
Memory Allocation Guide
===
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 918f374..4a342eb 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -486,48 +486,47 @@ static __always_inline void *kmalloc_large(size_t size,
gfp_t flags)
* kmalloc is the normal method of allocating memory
* for objects smaller than page size in the kernel.
*
- * The @flags argument may be one of:
+ * The @flags argument may be one of the GFP flags defined at
+ * include/linux/gfp.h and described at
+ * :ref:`Documentation/core-api/mm-api.rst `
*
- * %GFP_USER - Allocate memory on behalf of user. May sleep.
+ * The recommended usage of the @flags is described at
+ * :ref:`Documentation/core-api/memory-allocation.rst `
*
- * %GFP_KERNEL - Allocate normal kernel ram. May sleep.
+ * Below is a brief outline of the most useful GFP flags
*
- * %GFP_ATOMIC - Allocation will not sleep. May use emergency pools.
- * For example, use this inside interrupt handlers.
+ * %GFP_KERNEL
+ * Allocate normal kernel ram. May sleep.
*
- * %GFP_HIGHUSER - Allocate pages from high memory.
+ * %GFP_NOWAIT
+ * Allocation will not sleep.
*
- * %GFP_NOIO - Do not do any I/O at all while trying to get memory.
+ * %GFP_ATOMIC
+ * Allocation will not sleep. May use emergency pools.
*
- * %GFP_NOFS - Do not make any fs calls while trying to get memory.
- *
- * %GFP_NOWAIT - Allocation will not sleep.
- *
- * %__GFP_THISNODE - Allocate node-local memory only.
- *
- * %GFP_DMA - Allocation suitable for DMA.
- * Should only be used for kmalloc() caches. Otherwise, use a
- * slab created with SLAB_DMA.
+ * %GFP_HIGHUSER
+ * Allocate memory from high memory on behalf of user.
*
* Also it is possible to set different flags by OR'ing
* in one or more of the following additional @flags:
*
- * %__GFP_HIGH - This allocation has high priority and may use emergency pools.
- *
- * %__GFP_NOFAIL - Indicate that this allocation is in no way allowed to fail
- * (think twice before using).
+ * %__GFP_HIGH
+ * This allocation has high priority and may use emergency pools.
*
- * %__GFP_NORETRY - If memory is not immediately available,
- * then give up at once.
+ * %__GFP_NOFAIL
+ * Indicate that this allocation is in no way allowed to fail
+ * (think twice before using).
*
- * %__GFP_NOWARN - If allocation fails, don't issue any warnings.
+ * %__GFP_NORETRY
+ * If memory is not immediately available,
+ * then give up at once.
*
- * %__GFP_RETRY_MAYFAIL - Try really hard to succeed the allocation but fail
- * eventually.
+ * %__GFP_NOWARN
+ * If allocation fails, don't issue any warnings.
*
- * There are other flags available as well, but these are not intended
- * for general use, and so are not documented here. For a full list of
- * potential flags, always refer to linux/gfp.h.
+ * %__GFP_RETRY_MAYFAIL
+ * Try really hard to succeed the allocation but fail
+ * eventually.
*/
static __always_inline void *kmalloc(size_t size, gfp_t flags)
{
--
2.7.4
[PATCH v2] docs/admin-guide/mm/concepts.rst: grammar and style fixups
Signed-off-by: Mike Rapoport Reviewed-by: Randy Dunlap --- v2: address Matthew's feedback Documentation/admin-guide/mm/concepts.rst | 51 --- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/Documentation/admin-guide/mm/concepts.rst b/Documentation/admin-guide/mm/concepts.rst index 291699c..c2531b1 100644 --- a/Documentation/admin-guide/mm/concepts.rst +++ b/Documentation/admin-guide/mm/concepts.rst @@ -4,13 +4,13 @@ Concepts overview = -The memory management in Linux is complex system that evolved over the -years and included more and more functionality to support variety of +The memory management in Linux is a complex system that evolved over the +years and included more and more functionality to support a variety of systems from MMU-less microcontrollers to supercomputers. The memory -management for systems without MMU is called ``nommu`` and it +management for systems without an MMU is called ``nommu`` and it definitely deserves a dedicated document, which hopefully will be eventually written. Yet, although some of the concepts are the same, -here we assume that MMU is available and CPU can translate a virtual +here we assume that an MMU is available and a CPU can translate a virtual address to a physical address. .. contents:: :local: @@ -21,10 +21,10 @@ Virtual Memory Primer The physical memory in a computer system is a limited resource and even for systems that support memory hotplug there is a hard limit on the amount of memory that can be installed. The physical memory is not -necessary contiguous, it might be accessible as a set of distinct +necessarily contiguous; it might be accessible as a set of distinct address ranges. Besides, different CPU architectures, and even -different implementations of the same architecture have different view -how these address ranges defined. +different implementations of the same architecture have different views +of how these address ranges are defined. All this makes dealing directly with physical memory quite complex and to avoid this complexity a concept of virtual memory was developed. @@ -48,8 +48,8 @@ appropriate kernel configuration option. Each physical memory page can be mapped as one or more virtual pages. These mappings are described by page tables that allow -translation from virtual address used by programs to real address in -the physical memory. The page tables organized hierarchically. +translation from a virtual address used by programs to the physical +memory address. The page tables are organized hierarchically. The tables at the lowest level of the hierarchy contain physical addresses of actual pages used by the software. The tables at higher @@ -121,8 +121,8 @@ Nodes Many multi-processor machines are NUMA - Non-Uniform Memory Access - systems. In such systems the memory is arranged into banks that have different access latency depending on the "distance" from the -processor. Each bank is referred as `node` and for each node Linux -constructs an independent memory management subsystem. A node has it's +processor. Each bank is referred to as a `node` and for each node Linux +constructs an independent memory management subsystem. A node has its own set of zones, lists of free and used pages and various statistics counters. You can find more details about NUMA in :ref:`Documentation/vm/numa.rst ` and in @@ -149,9 +149,9 @@ for program's stack and heap or by explicit calls to mmap(2) system call. Usually, the anonymous mappings only define virtual memory areas that the program is allowed to access. The read accesses will result in creation of a page table entry that references a special physical -page filled with zeroes. When the program performs a write, regular +page filled with zeroes. When the program performs a write, a regular physical page will be allocated to hold the written data. The page -will be marked dirty and if the kernel will decide to repurpose it, +will be marked dirty and if the kernel decides to repurpose it, the dirty page will be swapped out. Reclaim @@ -181,8 +181,8 @@ pressure. The process of freeing the reclaimable physical memory pages and repurposing them is called (surprise!) `reclaim`. Linux can reclaim pages either asynchronously or synchronously, depending on the state -of the system. When system is not loaded, most of the memory is free -and allocation request will be satisfied immediately from the free +of the system. When the system is not loaded, most of the memory is free +and allocation requests will be satisfied immediately from the free pages supply. As the load increases, the amount of the free pages goes down and when it reaches a certain threshold (high watermark), an allocation request will awaken the ``kswapd`` daemon. It will @@ -190,7 +190,7 @@ asynchronously scan memory pages and either just free them if the data they contain is available elsewhere, or evict to the backi
[PATCH] mm/gup: fix follow_page_mask kernel-doc comment
Commit df06b37ffe5a ("mm/gup: cache dev_pagemap while pinning pages")
modified the signature of follow_page_mask() function but left the
parameter description behind.
Update the description to make the code and comments agree again.
While on it, update formatting of the return value description to match
Documentation/doc-guide/kernel-doc.rst guidelines.
Signed-off-by: Mike Rapoport
---
mm/gup.c | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index f76e77a..aa43620 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -385,11 +385,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct
*vma,
* @vma: vm_area_struct mapping @address
* @address: virtual address to look up
* @flags: flags modifying lookup behaviour
- * @page_mask: on output, *page_mask is set according to the size of the page
+ * @ctx: contains dev_pagemap for %ZONE_DEVICE memory pinning and a
+ * pointer to output page_mask
*
* @flags can have FOLL_ flags set, defined in
*
- * Returns the mapped (struct page *), %NULL if no mapping exists, or
+ * When getting pages from ZONE_DEVICE memory, the @ctx->pgmap caches
+ * the device's dev_pagemap metadata to avoid repeating expensive lookups.
+ *
+ * On output, the @ctx->page_mask is set according to the size of the page.
+ *
+ * Return: the mapped (struct page *), %NULL if no mapping exists, or
* an error pointer if there is a mapping to something not represented
* by a page descriptor (see also vm_normal_page()).
*/
--
2.7.4
Re: [PATCH] docs/admin-guide/mm/concepts.rst: grammar fixups
On Mon, Nov 05, 2018 at 11:29:27PM -0800, Randy Dunlap wrote: > On 11/5/18 10:35 PM, Mike Rapoport wrote: > > On Mon, Nov 05, 2018 at 01:12:40PM -0800, Matthew Wilcox wrote: > >> On Mon, Nov 05, 2018 at 09:58:15PM +0200, Mike Rapoport wrote: > >>> @@ -21,10 +21,10 @@ Virtual Memory Primer > >>> The physical memory in a computer system is a limited resource and > >>> even for systems that support memory hotplug there is a hard limit on > >>> the amount of memory that can be installed. The physical memory is not > >>> -necessary contiguous, it might be accessible as a set of distinct > >>> +necessary contiguous; it might be accessible as a set of distinct > >> > >> necessarily > >> > >>> address ranges. Besides, different CPU architectures, and even > >>> -different implementations of the same architecture have different view > >>> -how these address ranges defined. > >>> +different implementations of the same architecture have different views > >>> +of how these address ranges defined. > >> > >> "are defined"? > >> > >>> Each physical memory page can be mapped as one or more virtual > >>> pages. These mappings are described by page tables that allow > >>> -translation from virtual address used by programs to real address in > >>> -the physical memory. The page tables organized hierarchically. > >>> +translation from a virtual address used by programs to the real > >>> +address in the physical memory. The page tables are organized > >>> +hierarchically. > >> > >> I don't like the term "real address". Can we say "physical address in > >> memory" here, or "address of physical memory" or something? > > > > I didn't really like it as well, but I couldn't think of any better > > adjective to emphasize that address in the physical memory is "the real > > thing". > > > > Maybe the best would be to drop "real" and make it > > > > "translation from a virtual address used by programs to the > > address in the physical memory" > > physical memory address ? Works for me, thanks! > > -- > ~Randy > -- Sincerely yours, Mike.
Re: [PATCH] docs/admin-guide/mm/concepts.rst: grammar fixups
On Mon, Nov 05, 2018 at 01:12:40PM -0800, Matthew Wilcox wrote: > On Mon, Nov 05, 2018 at 09:58:15PM +0200, Mike Rapoport wrote: > > @@ -21,10 +21,10 @@ Virtual Memory Primer > > The physical memory in a computer system is a limited resource and > > even for systems that support memory hotplug there is a hard limit on > > the amount of memory that can be installed. The physical memory is not > > -necessary contiguous, it might be accessible as a set of distinct > > +necessary contiguous; it might be accessible as a set of distinct > > necessarily > > > address ranges. Besides, different CPU architectures, and even > > -different implementations of the same architecture have different view > > -how these address ranges defined. > > +different implementations of the same architecture have different views > > +of how these address ranges defined. > > "are defined"? > > > Each physical memory page can be mapped as one or more virtual > > pages. These mappings are described by page tables that allow > > -translation from virtual address used by programs to real address in > > -the physical memory. The page tables organized hierarchically. > > +translation from a virtual address used by programs to the real > > +address in the physical memory. The page tables are organized > > +hierarchically. > > I don't like the term "real address". Can we say "physical address in > memory" here, or "address of physical memory" or something? I didn't really like it as well, but I couldn't think of any better adjective to emphasize that address in the physical memory is "the real thing". Maybe the best would be to drop "real" and make it "translation from a virtual address used by programs to the address in the physical memory" > > -page filled with zeroes. When the program performs a write, regular > > +page filled with zeroes. When the program performs a write, a regular > > physical page will be allocated to hold the written data. The page > > will be marked dirty and if the kernel will decide to repurpose it, > > the dirty page will be swapped out. > > "if the kernel will decide to repurpose it" is awkward. How about > > if the kernel decides to repurpose it"? > > > OOM killer > > == > > > > -It may happen, that on a loaded machine memory will be exhausted. When > > +It may happen that on a loaded machine memory will be exhausted. When > > the kernel detects that the system runs out of memory (OOM) it invokes > > `OOM killer`. > > Its mission is simple: all it has to do is to select a > > task to sacrifice for the sake of the overall system health. The > > It is possible for the kernel to be unable to reclaim enough memory to > continue to operate. In order to save the rest of the system, it invokes > the `OOM killer` to sacrifice one task. How about: It is possible that on a loaded machine memory will be exhausted and the kernel will be unable to reclaim enough memory to continue to operate. In order to save the rest of the system, it invokes the `OOM killer`. The `OOM killer` selects a task to sacrifice for the sake of the overall system health. The selected task is killed in a hope that after it exits enough memory will be freed to continue normal operation. -- Sincerely yours, Mike.
[PATCH] docs/admin-guide/mm/concepts.rst: grammar fixups
From: Mike Rapoport Signed-off-by: Mike Rapoport Cc: Randy Dunlap --- There was a couple of grammar fixes Randy suggested a while ago, but it seems I've never sent them out. Documentation/admin-guide/mm/concepts.rst | 39 --- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/Documentation/admin-guide/mm/concepts.rst b/Documentation/admin-guide/mm/concepts.rst index 291699c..ab7a0f9 100644 --- a/Documentation/admin-guide/mm/concepts.rst +++ b/Documentation/admin-guide/mm/concepts.rst @@ -4,13 +4,13 @@ Concepts overview = -The memory management in Linux is complex system that evolved over the -years and included more and more functionality to support variety of +The memory management in Linux is a complex system that evolved over the +years and included more and more functionality to support a variety of systems from MMU-less microcontrollers to supercomputers. The memory -management for systems without MMU is called ``nommu`` and it +management for systems without an MMU is called ``nommu`` and it definitely deserves a dedicated document, which hopefully will be eventually written. Yet, although some of the concepts are the same, -here we assume that MMU is available and CPU can translate a virtual +here we assume that an MMU is available and a CPU can translate a virtual address to a physical address. .. contents:: :local: @@ -21,10 +21,10 @@ Virtual Memory Primer The physical memory in a computer system is a limited resource and even for systems that support memory hotplug there is a hard limit on the amount of memory that can be installed. The physical memory is not -necessary contiguous, it might be accessible as a set of distinct +necessary contiguous; it might be accessible as a set of distinct address ranges. Besides, different CPU architectures, and even -different implementations of the same architecture have different view -how these address ranges defined. +different implementations of the same architecture have different views +of how these address ranges defined. All this makes dealing directly with physical memory quite complex and to avoid this complexity a concept of virtual memory was developed. @@ -48,8 +48,9 @@ appropriate kernel configuration option. Each physical memory page can be mapped as one or more virtual pages. These mappings are described by page tables that allow -translation from virtual address used by programs to real address in -the physical memory. The page tables organized hierarchically. +translation from a virtual address used by programs to the real +address in the physical memory. The page tables are organized +hierarchically. The tables at the lowest level of the hierarchy contain physical addresses of actual pages used by the software. The tables at higher @@ -121,8 +122,8 @@ Nodes Many multi-processor machines are NUMA - Non-Uniform Memory Access - systems. In such systems the memory is arranged into banks that have different access latency depending on the "distance" from the -processor. Each bank is referred as `node` and for each node Linux -constructs an independent memory management subsystem. A node has it's +processor. Each bank is referred as a `node` and for each node Linux +constructs an independent memory management subsystem. A node has its own set of zones, lists of free and used pages and various statistics counters. You can find more details about NUMA in :ref:`Documentation/vm/numa.rst ` and in @@ -149,7 +150,7 @@ for program's stack and heap or by explicit calls to mmap(2) system call. Usually, the anonymous mappings only define virtual memory areas that the program is allowed to access. The read accesses will result in creation of a page table entry that references a special physical -page filled with zeroes. When the program performs a write, regular +page filled with zeroes. When the program performs a write, a regular physical page will be allocated to hold the written data. The page will be marked dirty and if the kernel will decide to repurpose it, the dirty page will be swapped out. @@ -181,8 +182,8 @@ pressure. The process of freeing the reclaimable physical memory pages and repurposing them is called (surprise!) `reclaim`. Linux can reclaim pages either asynchronously or synchronously, depending on the state -of the system. When system is not loaded, most of the memory is free -and allocation request will be satisfied immediately from the free +of the system. When the system is not loaded, most of the memory is free +and allocation requests will be satisfied immediately from the free pages supply. As the load increases, the amount of the free pages goes down and when it reaches a certain threshold (high watermark), an allocation request will awaken the ``kswapd`` daemon. It will @@ -190,7 +191,7 @@ asynchronously scan memory pages and either just free them if the data they contain is available elsewhere
Re: [PATCH 2/2] docs/vm: split memory hotplug notifier description to Documentation/core-api
On Sun, Oct 07, 2018 at 08:46:40AM -0600, Jonathan Corbet wrote: > On Fri, 5 Oct 2018 01:11:01 +0300 > Mike Rapoport wrote: > > > The memory hotplug notifier description is about kernel internals rather > > than admin/user visible API. Place it appropriately. > > > > Signed-off-by: Mike Rapoport > > One little nit... > > > Documentation/admin-guide/mm/memory-hotplug.rst| 83 > > - > > Documentation/core-api/index.rst | 2 + > > Documentation/core-api/memory-hotplug-notifier.rst | 84 > > ++ > > 3 files changed, 86 insertions(+), 83 deletions(-) > > create mode 100644 Documentation/core-api/memory-hotplug-notifier.rst > > > > diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst > > b/Documentation/admin-guide/mm/memory-hotplug.rst > > index a33090c..0b9c83e 100644 > > --- a/Documentation/admin-guide/mm/memory-hotplug.rst > > +++ b/Documentation/admin-guide/mm/memory-hotplug.rst > > @@ -31,7 +31,6 @@ be changed often. > > 6.1 Memory offline and ZONE_MOVABLE > > 6.2. How to offline memory > >7. Physical memory remove > > - 8. Memory hotplug event notifier > >9. Future Work List > > That leaves a gap in the numbering here. > > In general, the best solution to this sort of issue is to take the TOC out > entirely and let Sphinx worry about generating it. People tend not to > think about updating the TOC when they make changes elsewhere, so it often > goes out of sync with the rest of the document anyway. > > I'll apply these, but please feel free to send a patch to fix this up. Sure, below > Thanks, > > jon > >From 45ef951bd04e9d440a0158f991a188fc8ca6e7d4 Mon Sep 17 00:00:00 2001 From: Mike Rapoport Date: Sun, 7 Oct 2018 19:29:00 +0300 Subject: [PATCH] docs/admin-guide: memory-hotplug: remove table of contents Remove "manual" table of contents and leave only the ReST tag so that Sphinx will take care of TOC generation. Signed-off-by: Mike Rapoport --- Documentation/admin-guide/mm/memory-hotplug.rst | 21 - 1 file changed, 21 deletions(-) diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 0b9c83e..25157ae 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -13,27 +13,6 @@ be changed often. .. contents:: :local: -.. CONTENTS - - 1. Introduction -1.1 Purpose of memory hotplug -1.2. Phases of memory hotplug -1.3. Unit of Memory online/offline operation - 2. Kernel Configuration - 3. sysfs files for memory hotplug - 4. Physical memory hot-add phase -4.1 Hardware(Firmware) Support -4.2 Notify memory hot-add event by hand - 5. Logical Memory hot-add phase -5.1. State of memory -5.2. How to online memory - 6. Logical memory remove -6.1 Memory offline and ZONE_MOVABLE -6.2. How to offline memory - 7. Physical memory remove - 9. Future Work List - - .. note:: (1) x86_64's has special implementation for memory hotplug. -- 2.7.4
[PATCH 2/2] docs/vm: split memory hotplug notifier description to Documentation/core-api
The memory hotplug notifier description is about kernel internals rather
than admin/user visible API. Place it appropriately.
Signed-off-by: Mike Rapoport
---
Documentation/admin-guide/mm/memory-hotplug.rst| 83 -
Documentation/core-api/index.rst | 2 +
Documentation/core-api/memory-hotplug-notifier.rst | 84 ++
3 files changed, 86 insertions(+), 83 deletions(-)
create mode 100644 Documentation/core-api/memory-hotplug-notifier.rst
diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst
b/Documentation/admin-guide/mm/memory-hotplug.rst
index a33090c..0b9c83e 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -31,7 +31,6 @@ be changed often.
6.1 Memory offline and ZONE_MOVABLE
6.2. How to offline memory
7. Physical memory remove
- 8. Memory hotplug event notifier
9. Future Work List
@@ -414,88 +413,6 @@ Need more implementation yet
- Notification completion of remove works by OS to firmware.
- Guard from remove if not yet.
-Memory hotplug event notifier
-=
-
-Hotplugging events are sent to a notification queue.
-
-There are six types of notification defined in ``include/linux/memory.h``:
-
-MEM_GOING_ONLINE
- Generated before new memory becomes available in order to be able to
- prepare subsystems to handle memory. The page allocator is still unable
- to allocate from the new memory.
-
-MEM_CANCEL_ONLINE
- Generated if MEMORY_GOING_ONLINE fails.
-
-MEM_ONLINE
- Generated when memory has successfully brought online. The callback may
- allocate pages from the new memory.
-
-MEM_GOING_OFFLINE
- Generated to begin the process of offlining memory. Allocations are no
- longer possible from the memory but some of the memory to be offlined
- is still in use. The callback can be used to free memory known to a
- subsystem from the indicated memory block.
-
-MEM_CANCEL_OFFLINE
- Generated if MEMORY_GOING_OFFLINE fails. Memory is available again from
- the memory block that we attempted to offline.
-
-MEM_OFFLINE
- Generated after offlining memory is complete.
-
-A callback routine can be registered by calling::
-
- hotplug_memory_notifier(callback_func, priority)
-
-Callback functions with higher values of priority are called before callback
-functions with lower values.
-
-A callback function must have the following prototype::
-
- int callback_func(
-struct notifier_block *self, unsigned long action, void *arg);
-
-The first argument of the callback function (self) is a pointer to the block
-of the notifier chain that points to the callback function itself.
-The second argument (action) is one of the event types described above.
-The third argument (arg) passes a pointer of struct memory_notify::
-
- struct memory_notify {
- unsigned long start_pfn;
- unsigned long nr_pages;
- int status_change_nid_normal;
- int status_change_nid_high;
- int status_change_nid;
- }
-
-- start_pfn is start_pfn of online/offline memory.
-- nr_pages is # of pages of online/offline memory.
-- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask
- is (will be) set/clear, if this is -1, then nodemask status is not changed.
-- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask
- is (will be) set/clear, if this is -1, then nodemask status is not changed.
-- status_change_nid is set node id when N_MEMORY of nodemask is (will be)
- set/clear. It means a new(memoryless) node gets new memory by online and a
- node loses all memory. If this is -1, then nodemask status is not changed.
-
- If status_changed_nid* >= 0, callback should create/discard structures for
the
- node if necessary.
-
-The callback routine shall return one of the values
-NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP
-defined in ``include/linux/notifier.h``
-
-NOTIFY_DONE and NOTIFY_OK have no effect on the further processing.
-
-NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE,
-MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops
-further processing of the notification queue.
-
-NOTIFY_STOP stops further processing of the notification queue.
-
Future Work
===
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 165d7688..4f8a426 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -32,6 +32,8 @@ Core utilities
gfp_mask-from-fs-io
timekeeping
boot-time-mm
+ memory-hotplug-notifier
+
Interfaces for kernel debugging
===
diff --git a/Documentation/core-api/memory-hotplug-notifier.rst
b/Documentation/core-api/memory-hotplug-notifier.rst
new file mode 100644
index 000..35347cc
--- /dev/null
+++ b/Documentation/core-api/memory-hotplug-notifier.rst
@@ -0,0 +1
[PATCH 1/2] docs: move memory hotplug description into admin-guide/mm
The memory hotplug description in Documentation/memory-hotplug.txt is
already formatted as ReST and can be easily added to admin-guide/mm
section.
While on it, slightly update formatting to make it consistent with the
doc-guide.
Signed-off-by: Mike Rapoport
---
Documentation/admin-guide/mm/index.rst | 1 +
.../mm/memory-hotplug.rst} | 73 +++---
2 files changed, 38 insertions(+), 36 deletions(-)
rename Documentation/{memory-hotplug.txt => admin-guide/mm/memory-hotplug.rst}
(92%)
diff --git a/Documentation/admin-guide/mm/index.rst
b/Documentation/admin-guide/mm/index.rst
index ceead68..8edb35f 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -29,6 +29,7 @@ the Linux memory management.
hugetlbpage
idle_page_tracking
ksm
+ memory-hotplug
numa_memory_policy
pagemap
soft-dirty
diff --git a/Documentation/memory-hotplug.txt
b/Documentation/admin-guide/mm/memory-hotplug.rst
similarity index 92%
rename from Documentation/memory-hotplug.txt
rename to Documentation/admin-guide/mm/memory-hotplug.rst
index 7f49ebf..a33090c 100644
--- a/Documentation/memory-hotplug.txt
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -1,3 +1,5 @@
+.. _admin_guide_memory_hotplug:
+
==
Memory Hotplug
==
@@ -9,10 +11,12 @@ This document is about memory hotplug including how-to-use
and current status.
Because Memory Hotplug is still under development, contents of this text will
be changed often.
+.. contents:: :local:
+
.. CONTENTS
1. Introduction
-1.1 purpose of memory hotplug
+1.1 Purpose of memory hotplug
1.2. Phases of memory hotplug
1.3. Unit of Memory online/offline operation
2. Kernel Configuration
@@ -35,13 +39,13 @@ be changed often.
(1) x86_64's has special implementation for memory hotplug.
This text does not describe it.
-(2) This text assumes that sysfs is mounted at /sys.
+(2) This text assumes that sysfs is mounted at ``/sys``.
Introduction
-purpose of memory hotplug
+Purpose of memory hotplug
-
Memory Hotplug allows users to increase/decrease the amount of memory.
@@ -57,7 +61,6 @@ hardware which supports memory power management.
Linux memory hotplug is designed for both purpose.
-
Phases of memory hotplug
@@ -92,7 +95,6 @@ phase by hand.
(However, if you writes udev's hotplug scripts for memory hotplug, these
phases can be execute in seamless way.)
-
Unit of Memory online/offline operation
---
@@ -107,10 +109,9 @@ unit upon which memory online/offline operations are to be
performed. The
default size of a memory block is the same as memory section size unless an
architecture specifies otherwise. (see :ref:`memory_hotplug_sysfs_files`.)
-To determine the size (in bytes) of a memory block please read this file:
-
-/sys/devices/system/memory/block_size_bytes
+To determine the size (in bytes) of a memory block please read this file::
+ /sys/devices/system/memory/block_size_bytes
Kernel Configuration
@@ -119,22 +120,22 @@ To use memory hotplug feature, kernel must be compiled
with following
config options.
- For all memory hotplug:
-- Memory model -> Sparse Memory (CONFIG_SPARSEMEM)
-- Allow for memory hot-add (CONFIG_MEMORY_HOTPLUG)
+- Memory model -> Sparse Memory (``CONFIG_SPARSEMEM``)
+- Allow for memory hot-add (``CONFIG_MEMORY_HOTPLUG``)
- To enable memory removal, the following are also necessary:
-- Allow for memory hot remove(CONFIG_MEMORY_HOTREMOVE)
-- Page Migration (CONFIG_MIGRATION)
+- Allow for memory hot remove(``CONFIG_MEMORY_HOTREMOVE``)
+- Page Migration (``CONFIG_MIGRATION``)
- For ACPI memory hotplug, the following are also necessary:
-- Memory hotplug (under ACPI Support menu) (CONFIG_ACPI_HOTPLUG_MEMORY)
+- Memory hotplug (under ACPI Support menu) (``CONFIG_ACPI_HOTPLUG_MEMORY``)
- This option can be kernel module.
- As a related configuration, if your box has a feature of NUMA-node hotplug
via ACPI, then this option is necessary too.
- ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu)
- (CONFIG_ACPI_CONTAINER).
+ (``CONFIG_ACPI_CONTAINER``).
This option can be kernel module too.
@@ -145,10 +146,11 @@ sysfs files for memory hotplug
==
All memory blocks have their device information in sysfs. Each memory block
-is described under /sys/devices/system/memory as:
+is described under ``/sys/devices/system/memory`` as::
/sys/devices/system/memory/memoryXXX
- (XXX is the memory block id.)
+
+where XXX is the memory block id.
For the memory block covered by the sysfs directory. It is expected that all
memory
[PATCH 0/2] docs: ReSTify memory-hotplug description
Hi,
Recently I've noticed that Documentation/memory-hotplug.txt is
1) mostly formatted
2) in a wrong place
These patches split the memory-hotplug.txt to two parts: user/admin
interface and memory hotplug notifier API and place these parts in the
correct places, with some formatting changes.
Mike Rapoport (2):
docs: move memory hotplug description into admin-guide/mm
docs/vm: split memory hotplug notifier description to Documentation/core-api
Documentation/admin-guide/mm/index.rst | 1 +
.../mm/memory-hotplug.rst} | 152 +
Documentation/core-api/index.rst | 2 +
Documentation/core-api/memory-hotplug-notifier.rst | 84
4 files changed, 122 insertions(+), 117 deletions(-)
rename Documentation/{memory-hotplug.txt => admin-guide/mm/memory-hotplug.rst}
(77%)
create mode 100644 Documentation/core-api/memory-hotplug-notifier.rst
--
2.7.4
Re: [PATCH v3 1/2] Documentation/sphinx: allow "functions" with no parameters
On Sat, Jun 30, 2018 at 04:21:56AM -0700, Matthew Wilcox wrote: > On Sat, Jun 30, 2018 at 12:05:10AM +0300, Mike Rapoport wrote: > > @@ -488,14 +488,19 @@ doc: *title* > > .. kernel-doc:: drivers/gpu/drm/i915/intel_audio.c > > :doc: High Definition Audio over HDMI and Display Port > > > > -functions: *function* *[...]* > > +functions: *[ function ...]* > >Include documentation for each *function* in *source*. > > + If no *function* if specified, the documentaion for all functions > > "is specified". "documentation". > > > - Example:: > > + Examples:: > > > > .. kernel-doc:: lib/bitmap.c > > :functions: bitmap_parselist bitmap_parselist_user > > > > +.. kernel-doc:: lib/idr.c > > + :functions: > > Is this the right syntax? Should we rather have: > > .. kernel-doc:: lib/idr.c >:functions: * IMHO :functions: with no parameters is simpler. -- Sincerely yours, Mike. -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/2] Documentation/sphinx: add kind of "nodocs" directive
Hi, These patches allow passing "-no-doc-sections" option to scripts/kernel-doc from the sphinx generator. This allows to avoid duplicated DOC: sections when "kernel-doc:" directive is used without explicit selection of functions or function types. For instance, [1] has "IDA description" and "idr synchronization" twice. v3 changes: * add description for empty "functions" directive to Documentation/doc-guide/kernel-doc.rst v2 changes: * Use empty "functions" directive instead of "nodocs", as per Jani's suggestion Mike Rapoport (2): Documentation/sphinx: allow "functions" with no parameters docs/idr: use empty "functions" directive Documentation/core-api/idr.rst | 2 ++ Documentation/doc-guide/kernel-doc.rst | 9 +++-- Documentation/sphinx/kerneldoc.py | 10 +++--- 3 files changed, 16 insertions(+), 5 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 1/2] Documentation/sphinx: allow "functions" with no parameters
When kernel-doc:: specified in .rst document without explicit directives,
it outputs both comment and DOC: sections. If a DOC: section was explicitly
included in the same document it will be duplicated. For example, the
output generated for Documentation/core-api/idr.rst [1] has "IDA
description" in the "IDA usage" section and in the middle of the API
reference.
This patch enables using "functions" directive without parameters to output
all the documentation excluding DOC: sections.
[1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
Signed-off-by: Mike Rapoport
Acked-by: Matthew Wilcox
---
Documentation/doc-guide/kernel-doc.rst | 9 +++--
Documentation/sphinx/kerneldoc.py | 10 +++---
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/Documentation/doc-guide/kernel-doc.rst
b/Documentation/doc-guide/kernel-doc.rst
index 80383b1..8db53cd 100644
--- a/Documentation/doc-guide/kernel-doc.rst
+++ b/Documentation/doc-guide/kernel-doc.rst
@@ -488,14 +488,19 @@ doc: *title*
.. kernel-doc:: drivers/gpu/drm/i915/intel_audio.c
:doc: High Definition Audio over HDMI and Display Port
-functions: *function* *[...]*
+functions: *[ function ...]*
Include documentation for each *function* in *source*.
+ If no *function* if specified, the documentaion for all functions
+ and types in the *source* will be included.
- Example::
+ Examples::
.. kernel-doc:: lib/bitmap.c
:functions: bitmap_parselist bitmap_parselist_user
+.. kernel-doc:: lib/idr.c
+ :functions:
+
Without options, the kernel-doc directive includes all documentation comments
from the source file.
diff --git a/Documentation/sphinx/kerneldoc.py
b/Documentation/sphinx/kerneldoc.py
index fbedcc3..9d0a7f0 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
optional_arguments = 4
option_spec = {
'doc': directives.unchanged_required,
-'functions': directives.unchanged_required,
+'functions': directives.unchanged,
'export': directives.unchanged,
'internal': directives.unchanged,
}
@@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
elif 'doc' in self.options:
cmd += ['-function', str(self.options.get('doc'))]
elif 'functions' in self.options:
-for f in str(self.options.get('functions')).split():
-cmd += ['-function', f]
+functions = self.options.get('functions').split()
+if functions:
+for f in functions:
+cmd += ['-function', f]
+else:
+cmd += ['-no-doc-sections']
for pattern in export_file_patterns:
for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 2/2] docs/idr: use empty "functions" directive
to avoid duplication of DOC: sections in the middle of the API reference. Signed-off-by: Mike Rapoport Acked-by: Matthew Wilcox --- Documentation/core-api/idr.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/core-api/idr.rst b/Documentation/core-api/idr.rst index 9078a5c..d351e88 100644 --- a/Documentation/core-api/idr.rst +++ b/Documentation/core-api/idr.rst @@ -76,4 +76,6 @@ Functions and structures .. kernel-doc:: include/linux/idr.h + :functions: .. kernel-doc:: lib/idr.c + :functions: -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/2] Documentation/sphinx: allow "functions" with no parameters
On Wed, Jun 20, 2018 at 10:19:36AM +0300, Jani Nikula wrote:
> On Wed, 20 Jun 2018, Mike Rapoport wrote:
> > When kernel-doc:: specified in .rst document without explicit directives,
> > it outputs both comment and DOC: sections. If a DOC: section was explicitly
> > included in the same document it will be duplicated. For example, the
> > output generated for Documentation/core-api/idr.rst [1] has "IDA
> > description" in the "IDA usage" section and in the middle of the API
> > reference.
> >
> > This patch enables using "functions" directive without parameters to output
> > all the documentation excluding DOC: sections.
> >
> > [1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
> >
> > Signed-off-by: Mike Rapoport
> > Acked-by: Matthew Wilcox
>
> Looks good to me. Though I do realize now that I overlooked that this
> applies to not only functions, but also to other non-DOC documentation
> comments. I guess up to Jon to decide.
We can name it "everything-except-doc-sections" ;-)
> Please do give the cobbler's children some shoes, and document this in
> Documentation/doc-guide/kernel-doc.rst.
Sure. I'd just wait until there's consensus on the name :)
> Thanks,
> Jani.
>
> > ---
> > Documentation/sphinx/kerneldoc.py | 10 +++---
> > 1 file changed, 7 insertions(+), 3 deletions(-)
> >
> > diff --git a/Documentation/sphinx/kerneldoc.py
> > b/Documentation/sphinx/kerneldoc.py
> > index fbedcc3..9d0a7f0 100644
> > --- a/Documentation/sphinx/kerneldoc.py
> > +++ b/Documentation/sphinx/kerneldoc.py
> > @@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
> > optional_arguments = 4
> > option_spec = {
> > 'doc': directives.unchanged_required,
> > -'functions': directives.unchanged_required,
> > +'functions': directives.unchanged,
> > 'export': directives.unchanged,
> > 'internal': directives.unchanged,
> > }
> > @@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
> > elif 'doc' in self.options:
> > cmd += ['-function', str(self.options.get('doc'))]
> > elif 'functions' in self.options:
> > -for f in str(self.options.get('functions')).split():
> > -cmd += ['-function', f]
> > +functions = self.options.get('functions').split()
> > +if functions:
> > +for f in functions:
> > +cmd += ['-function', f]
> > +else:
> > +cmd += ['-no-doc-sections']
> >
> > for pattern in export_file_patterns:
> > for f in glob.glob(env.config.kerneldoc_srctree + '/' +
> > pattern):
>
> --
> Jani Nikula, Intel Open Source Graphics Center
>
--
Sincerely yours,
Mike.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 2/2] docs/idr: use empty "functions" directive
to avoid duplication of DOC: sections in the middle of the API reference. Signed-off-by: Mike Rapoport Acked-by: Matthew Wilcox --- Documentation/core-api/idr.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/core-api/idr.rst b/Documentation/core-api/idr.rst index 9078a5c..d351e88 100644 --- a/Documentation/core-api/idr.rst +++ b/Documentation/core-api/idr.rst @@ -76,4 +76,6 @@ Functions and structures .. kernel-doc:: include/linux/idr.h + :functions: .. kernel-doc:: lib/idr.c + :functions: -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 1/2] Documentation/sphinx: allow "functions" with no parameters
When kernel-doc:: specified in .rst document without explicit directives,
it outputs both comment and DOC: sections. If a DOC: section was explicitly
included in the same document it will be duplicated. For example, the
output generated for Documentation/core-api/idr.rst [1] has "IDA
description" in the "IDA usage" section and in the middle of the API
reference.
This patch enables using "functions" directive without parameters to output
all the documentation excluding DOC: sections.
[1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
Signed-off-by: Mike Rapoport
Acked-by: Matthew Wilcox
---
Documentation/sphinx/kerneldoc.py | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/Documentation/sphinx/kerneldoc.py
b/Documentation/sphinx/kerneldoc.py
index fbedcc3..9d0a7f0 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
optional_arguments = 4
option_spec = {
'doc': directives.unchanged_required,
-'functions': directives.unchanged_required,
+'functions': directives.unchanged,
'export': directives.unchanged,
'internal': directives.unchanged,
}
@@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
elif 'doc' in self.options:
cmd += ['-function', str(self.options.get('doc'))]
elif 'functions' in self.options:
-for f in str(self.options.get('functions')).split():
-cmd += ['-function', f]
+functions = self.options.get('functions').split()
+if functions:
+for f in functions:
+cmd += ['-function', f]
+else:
+cmd += ['-no-doc-sections']
for pattern in export_file_patterns:
for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 0/2] Documentation/sphinx: add kind of "nodocs" directive
Hi, These patches allow passing "-no-doc-sections" option to scripts/kernel-doc from the sphinx generator. This allows to avoid duplicated DOC: sections when "kernel-doc:" directive is used without explicit selection of functions or function types. For instance, [1] has "IDA description" and "idr synchronization" twice. v2 changes: * Use empty "functions" directive instead of "nodocs", as per Jani's suggestion Mike Rapoport (2): Documentation/sphinx: allow "functions" with no parameters docs/idr: use empty "functions" directive Documentation/core-api/idr.rst| 2 ++ Documentation/sphinx/kerneldoc.py | 10 +++--- 2 files changed, 9 insertions(+), 3 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Documentation/sphinx: add "nodocs" directive
On Tue, Jun 19, 2018 at 07:11:56AM -0700, Matthew Wilcox wrote:
> On Mon, Jun 18, 2018 at 10:10:28AM -0700, Matthew Wilcox wrote:
> > On Mon, Jun 18, 2018 at 04:36:34PM +0300, Mike Rapoport wrote:
> > > Hi,
> > >
> > > These patches allow passing "-no-doc-sections" option to
> > > scripts/kernel-doc
> > > from the sphinx generator.
> > >
> > > This allows to avoid duplicated DOC: sections when "kernel-doc:" directive
> > > is used without explicit selection of functions or function types. For
> > > instance, [1] has "IDA description" and "idr synchronization" twice.
> >
> > Hah, I just found an abandoned patch for this in a disused git tree.
> > I was wondering whether I needed to resurrect it. Enthusiastically,
> >
> > Acked-by: Matthew Wilcox
>
> Here's the patch I found (I couldn't refind it at the time):
>
> diff --git a/Documentation/sphinx/kerneldoc.py
> b/Documentation/sphinx/kerneldoc.py
> index d15e07f36881..d86d88da1d75 100644
> --- a/Documentation/sphinx/kerneldoc.py
> +++ b/Documentation/sphinx/kerneldoc.py
> @@ -47,6 +47,7 @@ class KernelDocDirective(Directive):
> optional_arguments = 4
> option_spec = {
> 'doc': directives.unchanged_required,
> +'nodoc': directives.unchanged,
> 'functions': directives.unchanged_required,
> 'export': directives.unchanged,
> 'internal': directives.unchanged,
> @@ -74,6 +75,8 @@ class KernelDocDirective(Directive):
> export_file_patterns = str(self.options.get('internal')).split()
> elif 'doc' in self.options:
> cmd += ['-function', str(self.options.get('doc'))]
> +elif 'nodoc' in self.options:
> +cmd += ['-no-doc-sections']
> elif 'functions' in self.options:
> for f in str(self.options.get('functions')).split():
> cmd += ['-function', f]
>
> I did it while I was trying to create good radix tree documentation,
> which led to me realising that was a Herculean task (specifically: the
> stables).
>
> I ended up doing this instead:
>
> +The Public API
> +==
> +
> +The public API can be found in . To use a
> +radix tree in your data structure, embed a :c:type:`struct radix_tree_root`
> +in it, and initialise it using ``INIT_RADIX_TREE``. You can also use
> +a file-local or global radix tree by defining a :c:type:`RADIX_TREE` as you
> +would a :c:type:`LIST_HEAD`.
> +
> +.. Not actually "internal", but I need to exclude the 'doc' paragraph, and
> + this is the best way to do it.
> +.. kernel-doc:: include/linux/radix-tree.h
> + :internal:
> +
> +.. kernel-doc:: lib/radix-tree.c
> + :export:
>
> I'm not sure if I agree with me-of-January-2017 that this is the "best"
> way to do it, but maybe that'll point to another way of achieving the
> same thing.
>
Actually, in the IDR case the same trick would work.
In more general case it's even possible to use
.. kernel-doc:: path/to/file.c
:export:
.. kernel-doc: path/to/file.c
:internal:
and get all the functions without the docs sections :)
But IMHO having a directive for this would be nicer. If there are no
objections to Jani's suggestion to use empty :functions: instead of
:nodoc: I'm going to send v2.
--
Sincerely yours,
Mike.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] Documentation/sphinx: add "nodocs" directive
On Tue, Jun 19, 2018 at 11:09:42AM +0300, Jani Nikula wrote:
> On Tue, 19 Jun 2018, Mike Rapoport wrote:
> > On Tue, Jun 19, 2018 at 10:29:20AM +0300, Jani Nikula wrote:
> >> On Tue, 19 Jun 2018, Mike Rapoport wrote:
> >> > On Mon, Jun 18, 2018 at 11:01:32PM +0300, Jani Nikula wrote:
> >> >> On Mon, 18 Jun 2018, Mike Rapoport wrote:
> >> >> > When kernel-doc:: specified in .rst document without explicit
> >> >> > directives,
> >> >> > it outputs both comment and DOC: sections. If a DOC: section was
> >> >> > explictly
> >> >> > included in the same document it will be duplicated. For example, the
> >> >> > output generated for Documentation/core-api/idr.rst [1] has "IDA
> >> >> > description" in the "IDA usage" section and in the middle of the API
> >> >> > reference.
> >> >> >
> >> >> > Addition of "nodocs" directive prevents the duplication without the
> >> >> > need to
> >> >> > explicitly define what functions should be include in the API
> >> >> > reference.
> >> >> >
> >> >> > [1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
> >> >> >
> >> >> > Signed-off-by: Mike Rapoport
> >> >> > ---
> >> >> > Documentation/sphinx/kerneldoc.py | 3 +++
> >> >> > 1 file changed, 3 insertions(+)
> >> >> >
> >> >> > diff --git a/Documentation/sphinx/kerneldoc.py
> >> >> > b/Documentation/sphinx/kerneldoc.py
> >> >> > index fbedcc3..bc5dd05 100644
> >> >> > --- a/Documentation/sphinx/kerneldoc.py
> >> >> > +++ b/Documentation/sphinx/kerneldoc.py
> >> >> > @@ -50,6 +50,7 @@ class KernelDocDirective(Directive):
> >> >> > 'functions': directives.unchanged_required,
> >> >> > 'export': directives.unchanged,
> >> >> > 'internal': directives.unchanged,
> >> >> > +'nodocs': directives.unchanged,
> >> >>
> >> >> I'm not convinved this is the prettiest way to achieve what you
> >> >> want. 'nodocs' seems kind of clunky.
> >> >>
> >> >> I'd suggest supporting 'functions' without option arguments, and turning
> >> >> that into kernel-doc -no-doc-sections.
> >> >
> >> > Do you mean something like this:
> >>
> >> Yes.
> >>
> >> >
> >> > diff --git a/Documentation/sphinx/kerneldoc.py
> >> > b/Documentation/sphinx/kerneldoc.py
> >> > index fbedcc3..9d0a7f0 100644
> >> > --- a/Documentation/sphinx/kerneldoc.py
> >> > +++ b/Documentation/sphinx/kerneldoc.py
> >> > @@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
> >> > optional_arguments = 4
> >> > option_spec = {
> >> > 'doc': directives.unchanged_required,
> >> > -'functions': directives.unchanged_required,
> >> > +'functions': directives.unchanged,
> >> > 'export': directives.unchanged,
> >> > 'internal': directives.unchanged,
> >> > }
> >> > @@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
> >> > elif 'doc' in self.options:
> >> > cmd += ['-function', str(self.options.get('doc'))]
> >> > elif 'functions' in self.options:
> >> > -for f in str(self.options.get('functions')).split():
> >> > -cmd += ['-function', f]
> >> > +functions = self.options.get('functions').split()
> >>
> >> Does .split() get upset if there's no argument? Or do you get an empty
> >> string if there are no options? I forget.
> >
> > "".split() gives an empty list.
>
> I tried to say, does self.options.get('functions') return an empty
> string or None if there are no options?
It returns an empty string.
> BR,
> Jani.
>
>
> >
> >> BR,
> >> Jani.
> >>
> >> > +if functions:
> >> > +for f in functions:
> >> > +cmd += ['-function', f]
> >> > +else:
> >> > +cmd += ['-no-doc-sections']
> >> >
> >> > for pattern in expo
Re: [PATCH 1/2] Documentation/sphinx: add "nodocs" directive
On Tue, Jun 19, 2018 at 10:29:20AM +0300, Jani Nikula wrote:
> On Tue, 19 Jun 2018, Mike Rapoport wrote:
> > On Mon, Jun 18, 2018 at 11:01:32PM +0300, Jani Nikula wrote:
> >> On Mon, 18 Jun 2018, Mike Rapoport wrote:
> >> > When kernel-doc:: specified in .rst document without explicit directives,
> >> > it outputs both comment and DOC: sections. If a DOC: section was
> >> > explictly
> >> > included in the same document it will be duplicated. For example, the
> >> > output generated for Documentation/core-api/idr.rst [1] has "IDA
> >> > description" in the "IDA usage" section and in the middle of the API
> >> > reference.
> >> >
> >> > Addition of "nodocs" directive prevents the duplication without the need
> >> > to
> >> > explicitly define what functions should be include in the API reference.
> >> >
> >> > [1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
> >> >
> >> > Signed-off-by: Mike Rapoport
> >> > ---
> >> > Documentation/sphinx/kerneldoc.py | 3 +++
> >> > 1 file changed, 3 insertions(+)
> >> >
> >> > diff --git a/Documentation/sphinx/kerneldoc.py
> >> > b/Documentation/sphinx/kerneldoc.py
> >> > index fbedcc3..bc5dd05 100644
> >> > --- a/Documentation/sphinx/kerneldoc.py
> >> > +++ b/Documentation/sphinx/kerneldoc.py
> >> > @@ -50,6 +50,7 @@ class KernelDocDirective(Directive):
> >> > 'functions': directives.unchanged_required,
> >> > 'export': directives.unchanged,
> >> > 'internal': directives.unchanged,
> >> > +'nodocs': directives.unchanged,
> >>
> >> I'm not convinved this is the prettiest way to achieve what you
> >> want. 'nodocs' seems kind of clunky.
> >>
> >> I'd suggest supporting 'functions' without option arguments, and turning
> >> that into kernel-doc -no-doc-sections.
> >
> > Do you mean something like this:
>
> Yes.
>
> >
> > diff --git a/Documentation/sphinx/kerneldoc.py
> > b/Documentation/sphinx/kerneldoc.py
> > index fbedcc3..9d0a7f0 100644
> > --- a/Documentation/sphinx/kerneldoc.py
> > +++ b/Documentation/sphinx/kerneldoc.py
> > @@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
> > optional_arguments = 4
> > option_spec = {
> > 'doc': directives.unchanged_required,
> > -'functions': directives.unchanged_required,
> > +'functions': directives.unchanged,
> > 'export': directives.unchanged,
> > 'internal': directives.unchanged,
> > }
> > @@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
> > elif 'doc' in self.options:
> > cmd += ['-function', str(self.options.get('doc'))]
> > elif 'functions' in self.options:
> > -for f in str(self.options.get('functions')).split():
> > -cmd += ['-function', f]
> > +functions = self.options.get('functions').split()
>
> Does .split() get upset if there's no argument? Or do you get an empty
> string if there are no options? I forget.
"".split() gives an empty list.
> BR,
> Jani.
>
> > +if functions:
> > +for f in functions:
> > +cmd += ['-function', f]
> > +else:
> > +cmd += ['-no-doc-sections']
> >
> > for pattern in export_file_patterns:
> > for f in glob.glob(env.config.kerneldoc_srctree + '/' +
> > pattern):
> >
> >> The usage in patch 2/2 would turn into:
> >>
> >> .. kernel-doc:: include/linux/idr.h
> >>:functions:
> >>
> >> which I think is much better overall in the rst source, complementing
> >> the places where you use :doc:.
> >>
> >> BR,
> >> Jani.
> >>
> >> > }
> >> > has_content = False
> >> >
> >> > @@ -77,6 +78,8 @@ class KernelDocDirective(Directive):
> >> > elif 'functions' in self.options:
> >> > for f in str(self.options.get('functions')).split():
> >> > cmd += ['-function', f]
> >> > +elif 'nodocs' in self.options:
> >> > +cmd += ['-no-doc-sections']
> >> >
> >> > for pattern in export_file_patterns:
> >> > for f in glob.glob(env.config.kerneldoc_srctree + '/' +
> >> > pattern):
> >>
> >> --
> >> Jani Nikula, Intel Open Source Graphics Center
> >>
>
> --
> Jani Nikula, Intel Open Source Graphics Center
>
--
Sincerely yours,
Mike.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] Documentation/sphinx: add "nodocs" directive
On Mon, Jun 18, 2018 at 11:01:32PM +0300, Jani Nikula wrote:
> On Mon, 18 Jun 2018, Mike Rapoport wrote:
> > When kernel-doc:: specified in .rst document without explicit directives,
> > it outputs both comment and DOC: sections. If a DOC: section was explictly
> > included in the same document it will be duplicated. For example, the
> > output generated for Documentation/core-api/idr.rst [1] has "IDA
> > description" in the "IDA usage" section and in the middle of the API
> > reference.
> >
> > Addition of "nodocs" directive prevents the duplication without the need to
> > explicitly define what functions should be include in the API reference.
> >
> > [1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
> >
> > Signed-off-by: Mike Rapoport
> > ---
> > Documentation/sphinx/kerneldoc.py | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/Documentation/sphinx/kerneldoc.py
> > b/Documentation/sphinx/kerneldoc.py
> > index fbedcc3..bc5dd05 100644
> > --- a/Documentation/sphinx/kerneldoc.py
> > +++ b/Documentation/sphinx/kerneldoc.py
> > @@ -50,6 +50,7 @@ class KernelDocDirective(Directive):
> > 'functions': directives.unchanged_required,
> > 'export': directives.unchanged,
> > 'internal': directives.unchanged,
> > +'nodocs': directives.unchanged,
>
> I'm not convinved this is the prettiest way to achieve what you
> want. 'nodocs' seems kind of clunky.
>
> I'd suggest supporting 'functions' without option arguments, and turning
> that into kernel-doc -no-doc-sections.
Do you mean something like this:
diff --git a/Documentation/sphinx/kerneldoc.py
b/Documentation/sphinx/kerneldoc.py
index fbedcc3..9d0a7f0 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -47,7 +47,7 @@ class KernelDocDirective(Directive):
optional_arguments = 4
option_spec = {
'doc': directives.unchanged_required,
-'functions': directives.unchanged_required,
+'functions': directives.unchanged,
'export': directives.unchanged,
'internal': directives.unchanged,
}
@@ -75,8 +75,12 @@ class KernelDocDirective(Directive):
elif 'doc' in self.options:
cmd += ['-function', str(self.options.get('doc'))]
elif 'functions' in self.options:
-for f in str(self.options.get('functions')).split():
-cmd += ['-function', f]
+functions = self.options.get('functions').split()
+if functions:
+for f in functions:
+cmd += ['-function', f]
+else:
+cmd += ['-no-doc-sections']
for pattern in export_file_patterns:
for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
> The usage in patch 2/2 would turn into:
>
> .. kernel-doc:: include/linux/idr.h
>:functions:
>
> which I think is much better overall in the rst source, complementing
> the places where you use :doc:.
>
> BR,
> Jani.
>
> > }
> > has_content = False
> >
> > @@ -77,6 +78,8 @@ class KernelDocDirective(Directive):
> > elif 'functions' in self.options:
> > for f in str(self.options.get('functions')).split():
> > cmd += ['-function', f]
> > +elif 'nodocs' in self.options:
> > +cmd += ['-no-doc-sections']
> >
> > for pattern in export_file_patterns:
> > for f in glob.glob(env.config.kerneldoc_srctree + '/' +
> > pattern):
>
> --
> Jani Nikula, Intel Open Source Graphics Center
>
--
Sincerely yours,
Mike.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] docs/idr: use "nodocs" directive
to avoid duplication of DOC: sections in the middle of the API reference. Signed-off-by: Mike Rapoport --- Documentation/core-api/idr.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/core-api/idr.rst b/Documentation/core-api/idr.rst index 9078a5c..15eadf8 100644 --- a/Documentation/core-api/idr.rst +++ b/Documentation/core-api/idr.rst @@ -76,4 +76,6 @@ Functions and structures .. kernel-doc:: include/linux/idr.h + :nodocs: .. kernel-doc:: lib/idr.c + :nodocs: -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/2] Documentation/sphinx: add "nodocs" directive
Hi, These patches allow passing "-no-doc-sections" option to scripts/kernel-doc from the sphinx generator. This allows to avoid duplicated DOC: sections when "kernel-doc:" directive is used without explicit selection of functions or function types. For instance, [1] has "IDA description" and "idr synchronization" twice. [1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html Mike Rapoport (2): Documentation/sphinx: add "nodocs" directive docs/idr: use "nodocs" directive Documentation/core-api/idr.rst| 2 ++ Documentation/sphinx/kerneldoc.py | 3 +++ 2 files changed, 5 insertions(+) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] Documentation/sphinx: add "nodocs" directive
When kernel-doc:: specified in .rst document without explicit directives,
it outputs both comment and DOC: sections. If a DOC: section was explictly
included in the same document it will be duplicated. For example, the
output generated for Documentation/core-api/idr.rst [1] has "IDA
description" in the "IDA usage" section and in the middle of the API
reference.
Addition of "nodocs" directive prevents the duplication without the need to
explicitly define what functions should be include in the API reference.
[1] https://www.kernel.org/doc/html/v4.17/core-api/idr.html
Signed-off-by: Mike Rapoport
---
Documentation/sphinx/kerneldoc.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/sphinx/kerneldoc.py
b/Documentation/sphinx/kerneldoc.py
index fbedcc3..bc5dd05 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -50,6 +50,7 @@ class KernelDocDirective(Directive):
'functions': directives.unchanged_required,
'export': directives.unchanged,
'internal': directives.unchanged,
+'nodocs': directives.unchanged,
}
has_content = False
@@ -77,6 +78,8 @@ class KernelDocDirective(Directive):
elif 'functions' in self.options:
for f in str(self.options.get('functions')).split():
cmd += ['-function', f]
+elif 'nodocs' in self.options:
+cmd += ['-no-doc-sections']
for pattern in export_file_patterns:
for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] docs/vm: move ksm and transhuge from "user" to "internals" section.
After the userspace interface description for KSM and THP was split to Documentation/admin-guide/mm, the remaining parts belong to the section describing MM internals. Signed-off-by: Mike Rapoport --- Documentation/vm/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst index 8e1cc66..c4ded22 100644 --- a/Documentation/vm/index.rst +++ b/Documentation/vm/index.rst @@ -13,8 +13,6 @@ various features of the Linux memory management .. toctree:: :maxdepth: 1 - ksm - transhuge swap_numa zswap @@ -36,6 +34,7 @@ descriptions of data structures and algorithms. hmm hwpoison hugetlbfs_reserv + ksm mmu_notifier numa overcommit-accounting @@ -45,6 +44,7 @@ descriptions of data structures and algorithms. remap_file_pages slub split_page_table_lock + transhuge unevictable-lru z3fold zsmalloc -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] docs/vm: transhuge: split userspace bits to admin-guide/mm
Hi, Here are minor updates to transparent hugepage docs. Except from minor formatting and spelling updates, these patches re-arrange the transhuge.rst so that userspace interface description will not be interleaved with the implementation details and it would be possible to split the userspace related bits to Documentation/admin-guide/mm, which is done by the third patch. Mike Rapoport (3): docs/vm: transhuge: change sections order docs/vm: transhuge: minor updates docs/vm: transhuge: split userspace bits to admin-guide/mm/transhuge Documentation/admin-guide/kernel-parameters.txt | 3 +- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/transhuge.rst | 418 Documentation/vm/transhuge.rst | 395 +- 4 files changed, 426 insertions(+), 391 deletions(-) create mode 100644 Documentation/admin-guide/mm/transhuge.rst -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] docs/vm: transhuge: change sections order
so that userspace interface and implementation description will be grouped together Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/transhuge.rst | 82 +- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/Documentation/vm/transhuge.rst b/Documentation/vm/transhuge.rst index 2c6867f..56d04cbb 100644 --- a/Documentation/vm/transhuge.rst +++ b/Documentation/vm/transhuge.rst @@ -38,31 +38,6 @@ are using hugepages but a significant speedup already happens if only one of the two is using hugepages just because of the fact the TLB miss is going to run faster. -Design -== - -- "graceful fallback": mm components which don't have transparent hugepage - knowledge fall back to breaking huge pmd mapping into table of ptes and, - if necessary, split a transparent hugepage. Therefore these components - can continue working on the regular pages or regular pte mappings. - -- if a hugepage allocation fails because of memory fragmentation, - regular pages should be gracefully allocated instead and mixed in - the same vma without any failure or significant delay and without - userland noticing - -- if some task quits and more hugepages become available (either - immediately in the buddy or through the VM), guest physical memory - backed by regular pages should be relocated on hugepages - automatically (with khugepaged) - -- it doesn't require memory reservation and in turn it uses hugepages - whenever possible (the only possible reservation here is kernelcore= - to avoid unmovable pages to fragment all the memory but such a tweak - is not specific to transparent hugepage support and it's a generic - feature that applies to all dynamic high order allocations in the - kernel) - Transparent Hugepage Support maximizes the usefulness of free memory if compared to the reservation approach of hugetlbfs by allowing all unused memory to be used as cache or other movable (or even unmovable @@ -401,6 +376,47 @@ tracer to record how long was spent in __alloc_pages_nodemask and using the mm_page_alloc tracepoint to identify which allocations were for huge pages. +Optimizing the applications +=== + +To be guaranteed that the kernel will map a 2M page immediately in any +memory region, the mmap region has to be hugepage naturally +aligned. posix_memalign() can provide that guarantee. + +Hugetlbfs += + +You can use hugetlbfs on a kernel that has transparent hugepage +support enabled just fine as always. No difference can be noted in +hugetlbfs other than there will be less overall fragmentation. All +usual features belonging to hugetlbfs are preserved and +unaffected. libhugetlbfs will also work fine as usual. + +Design principles += + +- "graceful fallback": mm components which don't have transparent hugepage + knowledge fall back to breaking huge pmd mapping into table of ptes and, + if necessary, split a transparent hugepage. Therefore these components + can continue working on the regular pages or regular pte mappings. + +- if a hugepage allocation fails because of memory fragmentation, + regular pages should be gracefully allocated instead and mixed in + the same vma without any failure or significant delay and without + userland noticing + +- if some task quits and more hugepages become available (either + immediately in the buddy or through the VM), guest physical memory + backed by regular pages should be relocated on hugepages + automatically (with khugepaged) + +- it doesn't require memory reservation and in turn it uses hugepages + whenever possible (the only possible reservation here is kernelcore= + to avoid unmovable pages to fragment all the memory but such a tweak + is not specific to transparent hugepage support and it's a generic + feature that applies to all dynamic high order allocations in the + kernel) + get_user_pages and follow_page == @@ -432,22 +448,6 @@ hugepages being returned (as it's not only checking the pfn of the page and pinning it during the copy but it pretends to migrate the memory in regular page sizes and with regular pte/pmd mappings). -Optimizing the applications -=== - -To be guaranteed that the kernel will map a 2M page immediately in any -memory region, the mmap region has to be hugepage naturally -aligned. posix_memalign() can provide that guarantee. - -Hugetlbfs -= - -You can use hugetlbfs on a kernel that has transparent hugepage -support enabled just fine as always. No difference can be noted in -hugetlbfs other than there will be less overall fragmentation. All -usual features belonging to hugetlbfs are preserved and -unaffected. libhugetlbfs will also work fine as usual. - Graceful fallback = -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a
[PATCH 2/3] docs/vm: transhuge: minor updates
Some formatting changes and addition of a sentence introducing khugepaged Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/transhuge.rst | 47 -- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/Documentation/vm/transhuge.rst b/Documentation/vm/transhuge.rst index 56d04cbb..47c7e47 100644 --- a/Documentation/vm/transhuge.rst +++ b/Documentation/vm/transhuge.rst @@ -9,14 +9,19 @@ Objective Performance critical computing applications dealing with large memory working sets are already running on top of libhugetlbfs and in turn -hugetlbfs. Transparent Hugepage Support is an alternative means of +hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of using huge pages for the backing of virtual memory with huge pages that supports the automatic promotion and demotion of page sizes and without the shortcomings of hugetlbfs. -Currently it only works for anonymous memory mappings and tmpfs/shmem. +Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems. +.. note:: + in the examples below we presume that the basic page size is 4K and + the huge page size is 2M, although the actual numbers may vary + depending on the CPU architecture. + The reason applications are running faster is because of two factors. The first factor is almost completely irrelevant and it's not of significant interest because it'll also have the downside of @@ -28,15 +33,27 @@ only matters the first time the memory is accessed for the lifetime of a memory mapping. The second long lasting and much more important factor will affect all subsequent accesses to the memory for the whole runtime of the application. The second factor consist of two -components: 1) the TLB miss will run faster (especially with -virtualization using nested pagetables but almost always also on bare -metal without virtualization) and 2) a single TLB entry will be -mapping a much larger amount of virtual memory in turn reducing the -number of TLB misses. With virtualization and nested pagetables the -TLB can be mapped of larger size only if both KVM and the Linux guest -are using hugepages but a significant speedup already happens if only -one of the two is using hugepages just because of the fact the TLB -miss is going to run faster. +components: + +1) the TLB miss will run faster (especially with virtualization using + nested pagetables but almost always also on bare metal without + virtualization) + +2) a single TLB entry will be mapping a much larger amount of virtual + memory in turn reducing the number of TLB misses. With + virtualization and nested pagetables the TLB can be mapped of + larger size only if both KVM and the Linux guest are using + hugepages but a significant speedup already happens if only one of + the two is using hugepages just because of the fact the TLB miss is + going to run faster. + +THP can be enabled system wide or restricted to certain tasks or even +memory ranges inside task's address space. Unless THP is completely +disabled, there is ``khugepaged`` daemon that scans memory and +collapses sequences of basic pages into huge pages. + +The THP behaviour is controlled via :ref:`sysfs ` +interface and using madivse(2) and prctl(2) system calls. Transparent Hugepage Support maximizes the usefulness of free memory if compared to the reservation approach of hugetlbfs by allowing all @@ -69,9 +86,14 @@ Applications that gets a lot of benefit from hugepages and that don't risk to lose memory by using hugepages, should use madvise(MADV_HUGEPAGE) on their critical mmapped regions. +.. _thp_sysfs: + sysfs = +Global THP controls +--- + Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled @@ -142,6 +164,9 @@ khugepaged will be automatically started when transparent_hugepage/enabled is set to "always" or "madvise, and it'll be automatically shutdown if it's set to "never". +Khugepaged controls +--- + khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] docs/vm: numa_memory_policy: formatting and spelling updates
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/numa_memory_policy.rst | 24 +--- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/Documentation/vm/numa_memory_policy.rst b/Documentation/vm/numa_memory_policy.rst index 8cd942c..ac0b396 100644 --- a/Documentation/vm/numa_memory_policy.rst +++ b/Documentation/vm/numa_memory_policy.rst @@ -44,14 +44,20 @@ System Default Policy allocations. Task/Process Policy - this is an optional, per-task policy. When defined for a specific task, this policy controls all page allocations made by or on behalf of the task that aren't controlled by a more specific scope. If a task does not define a task policy, then all page allocations that would have been controlled by the task policy "fall back" to the System Default Policy. + this is an optional, per-task policy. When defined for a + specific task, this policy controls all page allocations made + by or on behalf of the task that aren't controlled by a more + specific scope. If a task does not define a task policy, then + all page allocations that would have been controlled by the + task policy "fall back" to the System Default Policy. The task policy applies to the entire address space of a task. Thus, it is inheritable, and indeed is inherited, across both fork() [clone() w/o the CLONE_VM flag] and exec*(). This allows a parent task to establish the task policy for a child task exec()'d from an executable image that has no awareness of memory policy. See the - MEMORY POLICY APIS section, below, for an overview of the system call + :ref:`Memory Policy APIs ` section, + below, for an overview of the system call that a task may use to set/change its task/process policy. In a multi-threaded task, task policies apply only to the thread @@ -70,12 +76,13 @@ Task/Process Policy VMA Policy A "VMA" or "Virtual Memory Area" refers to a range of a task's virtual address space. A task may define a specific policy for a range - of its virtual address space. See the MEMORY POLICIES APIS section, + of its virtual address space. See the + :ref:`Memory Policy APIs ` section, below, for an overview of the mbind() system call used to set a VMA policy. A VMA policy will govern the allocation of pages that back - this region ofthe address space. Any regions of the task's + this region of the address space. Any regions of the task's address space that don't have an explicit VMA policy will fall back to the task policy, which may itself fall back to the System Default Policy. @@ -117,7 +124,7 @@ VMA Policy Shared Policy Conceptually, shared policies apply to "memory objects" mapped shared into one or more tasks' distinct address spaces. An - application installs a shared policies the same way as VMA + application installs shared policies the same way as VMA policies--using the mbind() system call specifying a range of virtual addresses that map the shared object. However, unlike VMA policies, which can be considered to be an attribute of a @@ -135,7 +142,7 @@ Shared Policy Although hugetlbfs segments now support lazy allocation, their support for shared policy has not been completed. - As mentioned above :ref:`VMA policies `, + As mentioned above in :ref:`VMA policies ` section, allocations of page cache pages for regular files mmap()ed with MAP_SHARED ignore any VMA policy installed on the virtual address range backed by the shared file mapping. Rather, @@ -245,7 +252,7 @@ MPOL_F_STATIC_NODES the user should not be remapped if the task or VMA's set of allowed nodes changes after the memory policy has been defined. - Without this flag, anytime a mempolicy is rebound because of a + Without this flag, any time a mempolicy is rebound because of a change in the set of allowed nodes, the node (Preferred) or nodemask (Bind, Interleave) is remapped to the new set of allowed nodes. This may result in nodes being used that were @@ -389,7 +396,10 @@ follows: or by prefaulting the entire shared memory region into memory and locking it down. However, this might not be appropriate for all applications. +.. _memory_policy_apis: + Memory Policy APIs +== Linux supports 3 system calls for controlling memory policy. These APIS always affect only the calling task, the calling task's address space, or -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] docs/vm: move numa_memory_policy.rst to admin-guide/mm
Hi,
These patches include minor formatting and spelling updates to
Documentation/vm/numa_memory_policy.rst and move this file to
Documentation/admin-guide/mm.
Mike Rapoport (3):
docs/vm: numa_memory_policy: formatting and spelling updates
docs/vm: numa_memory_policy: s/Linux memory policy/NUMA memory policy/
docs/vm: move numa_memory_policy.rst to Documentation/admin-guide/mm
Documentation/admin-guide/mm/hugetlbpage.rst | 2 +-
Documentation/admin-guide/mm/index.rst | 1 +
.../{vm => admin-guide/mm}/numa_memory_policy.rst | 38 ++
Documentation/filesystems/proc.txt | 2 +-
Documentation/filesystems/tmpfs.txt| 5 +--
Documentation/vm/00-INDEX | 2 --
Documentation/vm/index.rst | 1 -
Documentation/vm/numa.rst | 2 +-
8 files changed, 31 insertions(+), 22 deletions(-)
rename Documentation/{vm => admin-guide/mm}/numa_memory_policy.rst (95%)
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] docs/vm: move numa_memory_policy.rst to Documentation/admin-guide/mm
The document describes userspace API and as such it belongs to
Documentation/admin-guide/mm
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/admin-guide/mm/hugetlbpage.rst| 2 +-
Documentation/admin-guide/mm/index.rst | 1 +
Documentation/{vm => admin-guide/mm}/numa_memory_policy.rst | 0
Documentation/filesystems/proc.txt | 2 +-
Documentation/filesystems/tmpfs.txt | 5 +++--
Documentation/vm/00-INDEX | 2 --
Documentation/vm/index.rst | 1 -
Documentation/vm/numa.rst | 2 +-
8 files changed, 7 insertions(+), 8 deletions(-)
rename Documentation/{vm => admin-guide/mm}/numa_memory_policy.rst (100%)
diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst
b/Documentation/admin-guide/mm/hugetlbpage.rst
index a8b0806..1cc0bc7 100644
--- a/Documentation/admin-guide/mm/hugetlbpage.rst
+++ b/Documentation/admin-guide/mm/hugetlbpage.rst
@@ -220,7 +220,7 @@ memory policy mode--bind, preferred, local or
interleave--may be used. The
resulting effect on persistent huge page allocation is as follows:
#. Regardless of mempolicy mode [see
- :ref:`Documentation/vm/numa_memory_policy.rst `],
+ :ref:`Documentation/admin-guide/mm/numa_memory_policy.rst
`],
persistent huge pages will be distributed across the node or nodes
specified in the mempolicy as if "interleave" had been specified.
However, if a node in the policy does not contain sufficient contiguous
diff --git a/Documentation/admin-guide/mm/index.rst
b/Documentation/admin-guide/mm/index.rst
index ad28644..a69aa69 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -24,6 +24,7 @@ the Linux memory management.
hugetlbpage
idle_page_tracking
ksm
+ numa_memory_policy
pagemap
soft-dirty
userfaultfd
diff --git a/Documentation/vm/numa_memory_policy.rst
b/Documentation/admin-guide/mm/numa_memory_policy.rst
similarity index 100%
rename from Documentation/vm/numa_memory_policy.rst
rename to Documentation/admin-guide/mm/numa_memory_policy.rst
diff --git a/Documentation/filesystems/proc.txt
b/Documentation/filesystems/proc.txt
index ef53f80..520f6a8 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -566,7 +566,7 @@ address policymapping details
Where:
"address" is the starting address for the mapping;
-"policy" reports the NUMA memory policy set for the mapping (see
vm/numa_memory_policy.txt);
+"policy" reports the NUMA memory policy set for the mapping (see
Documentation/admin-guide/mm/numa_memory_policy.rst);
"mapping details" summarizes mapping data such as mapping type, page usage
counters,
node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page
size, in KB, that is backing the mapping up.
diff --git a/Documentation/filesystems/tmpfs.txt
b/Documentation/filesystems/tmpfs.txt
index 627389a..d06e9a5 100644
--- a/Documentation/filesystems/tmpfs.txt
+++ b/Documentation/filesystems/tmpfs.txt
@@ -105,8 +105,9 @@ policy for the file will revert to "default" policy.
NUMA memory allocation policies have optional flags that can be used in
conjunction with their modes. These optional flags can be specified
when tmpfs is mounted by appending them to the mode before the NodeList.
-See Documentation/vm/numa_memory_policy.rst for a list of all available
-memory allocation policy mode flags and their effect on memory policy.
+See Documentation/admin-guide/mm/numa_memory_policy.rst for a list of
+all available memory allocation policy mode flags and their effect on
+memory policy.
=static is equivalent toMPOL_F_STATIC_NODES
=relative is equivalent toMPOL_F_RELATIVE_NODES
diff --git a/Documentation/vm/00-INDEX b/Documentation/vm/00-INDEX
index f8a96ca..f4a4f3e 100644
--- a/Documentation/vm/00-INDEX
+++ b/Documentation/vm/00-INDEX
@@ -22,8 +22,6 @@ mmu_notifier.rst
- a note about clearing pte/pmd and mmu notifications
numa.rst
- information about NUMA specific code in the Linux vm.
-numa_memory_policy.rst
- - documentation of concepts and APIs of the 2.6 memory policy support.
overcommit-accounting.rst
- description of the Linux kernels overcommit handling modes.
page_frags.rst
diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
index ed58cb9..8e1cc66 100644
--- a/Documentation/vm/index.rst
+++ b/Documentation/vm/index.rst
@@ -14,7 +14,6 @@ various features of the Linux memory management
:maxdepth: 1
ksm
- numa_memory_policy
transhuge
swap_numa
zswap
diff --git a/Documentation/vm/numa.rst b/Documentation/vm/numa.rst
index aada84b..185d8a5 100644
--- a/Documentation/vm/numa.rst
+++ b/Documentation/vm
[PATCH 2/3] docs/vm: numa_memory_policy: s/Linux memory policy/NUMA memory policy/
The document describes NUMA memory policy and as it is a part of the Linux documentation it's obvious that this is Linux memory policy. Besides, "Linux memory policy" may refer to other policies, e.g. memory hotplug policy, and using term NUMA makes the documentation less ambiguous. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/numa_memory_policy.rst | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/vm/numa_memory_policy.rst b/Documentation/vm/numa_memory_policy.rst index ac0b396..d78c5b3 100644 --- a/Documentation/vm/numa_memory_policy.rst +++ b/Documentation/vm/numa_memory_policy.rst @@ -1,10 +1,10 @@ .. _numa_memory_policy: -=== -Linux Memory Policy -=== +== +NUMA Memory Policy +== -What is Linux Memory Policy? +What is NUMA Memory Policy? In the Linux kernel, "memory policy" determines from which node the kernel will @@ -162,7 +162,7 @@ Shared Policy Components of Memory Policies - -A Linux memory policy consists of a "mode", optional mode flags, and +A NUMA memory policy consists of a "mode", optional mode flags, and an optional set of nodes. The mode determines the behavior of the policy, the optional mode flags determine the behavior of the mode, and the optional set of nodes can be viewed as the arguments to the @@ -172,7 +172,7 @@ Internally, memory policies are implemented by a reference counted structure, struct mempolicy. Details of this structure will be discussed in context, below, as required to explain the behavior. -Linux memory policy supports the following 4 behavioral modes: +NUMA memory policy supports the following 4 behavioral modes: Default Mode--MPOL_DEFAULT This mode is only used in the memory policy APIs. Internally, @@ -245,7 +245,7 @@ MPOL_INTERLEAVED address range or file. During system boot up, the temporary interleaved system default policy works in this mode. -Linux memory policy supports the following optional mode flags: +NUMA memory policy supports the following optional mode flags: MPOL_F_STATIC_NODES This flag specifies that the nodemask passed by -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/7] docs/vm: ksm: (mostly) formatting updates
Aside from the formatting: * fixed typos * added section and sub-section headers * moved ksmd overview after the description of KSM origins Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.rst | 110 +-- 1 file changed, 69 insertions(+), 41 deletions(-) diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst index 87e7eef..786d460 100644 --- a/Documentation/vm/ksm.rst +++ b/Documentation/vm/ksm.rst @@ -4,34 +4,50 @@ Kernel Samepage Merging === +Overview + + KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, added to the Linux kernel in 2.6.32. See ``mm/ksm.c`` for its implementation, and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ -The KSM daemon ksmd periodically scans those areas of user memory which -have been registered with it, looking for pages of identical content which -can be replaced by a single write-protected page (which is automatically -copied if a process later wants to update its content). - KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data. +The KSM daemon ksmd periodically scans those areas of user memory +which have been registered with it, looking for pages of identical +content which can be replaced by a single write-protected page (which +is automatically copied if a process later wants to update its +content). The amount of pages that KSM daemon scans in a single pass +and the time between the passes are configured using :ref:`sysfs +intraface ` + KSM only merges anonymous (private) pages, never pagecache (file) pages. KSM's merged pages were originally locked into kernel memory, but can now be swapped out just like other user pages (but sharing is broken when they are swapped back in: ksmd must rediscover their identity and merge again). +Controlling KSM with madvise + + KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) -system call: int madvise(addr, length, MADV_MERGEABLE). +system call:: + + int madvise(addr, length, MADV_MERGEABLE) + +The app may call + +:: + + int madvise(addr, length, MADV_UNMERGEABLE) -The app may call int madvise(addr, length, MADV_UNMERGEABLE) to cancel -that advice and restore unshared pages: whereupon KSM unmerges whatever -it merged in that range. Note: this unmerging call may suddenly require -more memory than is available - possibly failing with EAGAIN, but more -probably arousing the Out-Of-Memory killer. +to cancel that advice and restore unshared pages: whereupon KSM +unmerges whatever it merged in that range. Note: this unmerging call +may suddenly require more memory than is available - possibly failing +with EAGAIN, but more probably arousing the Out-Of-Memory killer. If KSM is not configured into the running kernel, madvise MADV_MERGEABLE and MADV_UNMERGEABLE simply fail with EINVAL. If the running kernel was @@ -43,7 +59,7 @@ MADV_UNMERGEABLE is applied to a range which was never MADV_MERGEABLE. If a region of memory must be split into at least one new MADV_MERGEABLE or MADV_UNMERGEABLE region, the madvise may return ENOMEM if the process -will exceed vm.max_map_count (see Documentation/sysctl/vm.txt). +will exceed ``vm.max_map_count`` (see Documentation/sysctl/vm.txt). Like other madvise calls, they are intended for use on mapped areas of the user address space: they will report ENOMEM if the specified range @@ -54,21 +70,28 @@ Applications should be considerate in their use of MADV_MERGEABLE, restricting its use to areas likely to benefit. KSM's scans may use a lot of processing power: some installations will disable KSM for that reason. +.. _ksm_sysfs: + +KSM daemon sysfs interface +== + The KSM daemon is controlled by sysfs files in ``/sys/kernel/mm/ksm/``, readable by all but writable only by root: pages_to_scan -how many present pages to scan before ksmd goes to sleep -e.g. ``echo 100 > /sys/kernel/mm/ksm/pages_to_scan`` Default: 100 -(chosen for demonstration purposes) +how many pages to scan before ksmd goes to sleep +e.g. ``echo 100 > /sys/kernel/mm/ksm/pages_to_scan``. + +Default: 100 (chosen for demonstration purposes) sleep_millisecs how many milliseconds ksmd should sleep before next scan -e.g. ``echo 20 > /sys/kernel/mm/ksm/sleep_millisecs`` Default: 20 -(chosen for demonstration purposes) +e.g. ``echo 20 > /sys/kernel/mm/ksm/sleep_millisecs`` + +Default: 20 (chosen for demonstration purposes) merge_across_nodes -specifies if pa
[PATCH 6/7] docs/vm: ksm: udpate description of stable_node_{dups,chains}
Remove implementation details from sysfs parameter descriptions. Also move the paragraph discussing fragmentation issues and their possible solution to the "Design" section. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.rst | 21 ++--- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst index 18d7c71..afcf5a8 100644 --- a/Documentation/vm/ksm.rst +++ b/Documentation/vm/ksm.rst @@ -170,10 +170,9 @@ pages_volatile full_scans how many times all mergeable areas have been scanned stable_node_chains -number of stable node chains allocated, this is effectively the number of KSM pages that hit the ``max_page_sharing`` limit stable_node_dups -number of stable node dups queued into the stable_node chains +number of duplicated KSM pages A high ratio of ``pages_sharing`` to ``pages_shared`` indicates good sharing, but a high ratio of ``pages_unshared`` to ``pages_sharing`` @@ -185,15 +184,6 @@ The maximum possible ``pages_sharing/pages_shared`` ratio is limited by the ``max_page_sharing`` tunable. To increase the ratio ``max_page_sharing`` must be increased accordingly. -The ``stable_node_dups/stable_node_chains`` ratio is also affected by the -``max_page_sharing`` tunable, and an high ratio may indicate fragmentation -in the stable_node dups, which could be solved by introducing -fragmentation algorithms in ksmd which would refile rmap_items from -one stable_node dup to another stable_node dup, in order to free up -stable_node "dups" with few rmap_items in them, but that may increase -the ksmd CPU usage and possibly slowdown the readonly computations on -the KSM pages of the applications. - Design == @@ -247,6 +237,15 @@ deduplication factor at the expense of slower worst case for rmap walks for any KSM page which can happen during swapping, compaction, NUMA balancing and page migration. +The ``stable_node_dups/stable_node_chains`` ratio is also affected by the +``max_page_sharing`` tunable, and an high ratio may indicate fragmentation +in the stable_node dups, which could be solved by introducing +fragmentation algorithms in ksmd which would refile rmap_items from +one stable_node dup to another stable_node dup, in order to free up +stable_node "dups" with few rmap_items in them, but that may increase +the ksmd CPU usage and possibly slowdown the readonly computations on +the KSM pages of the applications. + The whole list of stable_node "dups" linked in the stable_node "chains" is scanned periodically in order to prune stale stable_nodes. The frequency of such scans is defined by -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 7/7] docs/vm: ksm: split userspace interface to admin-guide/mm/ksm.rst
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/ksm.rst | 189 + Documentation/vm/ksm.rst | 176 +- 3 files changed, 191 insertions(+), 175 deletions(-) create mode 100644 Documentation/admin-guide/mm/ksm.rst diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst index 6c8b554..ad28644 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -23,6 +23,7 @@ the Linux memory management. hugetlbpage idle_page_tracking + ksm pagemap soft-dirty userfaultfd diff --git a/Documentation/admin-guide/mm/ksm.rst b/Documentation/admin-guide/mm/ksm.rst new file mode 100644 index 000..9303786 --- /dev/null +++ b/Documentation/admin-guide/mm/ksm.rst @@ -0,0 +1,189 @@ +.. _admin_guide_ksm: + +=== +Kernel Samepage Merging +=== + +Overview + + +KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, +added to the Linux kernel in 2.6.32. See ``mm/ksm.c`` for its implementation, +and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ + +KSM was originally developed for use with KVM (where it was known as +Kernel Shared Memory), to fit more virtual machines into physical memory, +by sharing the data common between them. But it can be useful to any +application which generates many instances of the same data. + +The KSM daemon ksmd periodically scans those areas of user memory +which have been registered with it, looking for pages of identical +content which can be replaced by a single write-protected page (which +is automatically copied if a process later wants to update its +content). The amount of pages that KSM daemon scans in a single pass +and the time between the passes are configured using :ref:`sysfs +intraface ` + +KSM only merges anonymous (private) pages, never pagecache (file) pages. +KSM's merged pages were originally locked into kernel memory, but can now +be swapped out just like other user pages (but sharing is broken when they +are swapped back in: ksmd must rediscover their identity and merge again). + +Controlling KSM with madvise + + +KSM only operates on those areas of address space which an application +has advised to be likely candidates for merging, by using the madvise(2) +system call:: + + int madvise(addr, length, MADV_MERGEABLE) + +The app may call + +:: + + int madvise(addr, length, MADV_UNMERGEABLE) + +to cancel that advice and restore unshared pages: whereupon KSM +unmerges whatever it merged in that range. Note: this unmerging call +may suddenly require more memory than is available - possibly failing +with EAGAIN, but more probably arousing the Out-Of-Memory killer. + +If KSM is not configured into the running kernel, madvise MADV_MERGEABLE +and MADV_UNMERGEABLE simply fail with EINVAL. If the running kernel was +built with CONFIG_KSM=y, those calls will normally succeed: even if the +the KSM daemon is not currently running, MADV_MERGEABLE still registers +the range for whenever the KSM daemon is started; even if the range +cannot contain any pages which KSM could actually merge; even if +MADV_UNMERGEABLE is applied to a range which was never MADV_MERGEABLE. + +If a region of memory must be split into at least one new MADV_MERGEABLE +or MADV_UNMERGEABLE region, the madvise may return ENOMEM if the process +will exceed ``vm.max_map_count`` (see Documentation/sysctl/vm.txt). + +Like other madvise calls, they are intended for use on mapped areas of +the user address space: they will report ENOMEM if the specified range +includes unmapped gaps (though working on the intervening mapped areas), +and might fail with EAGAIN if not enough memory for internal structures. + +Applications should be considerate in their use of MADV_MERGEABLE, +restricting its use to areas likely to benefit. KSM's scans may use a lot +of processing power: some installations will disable KSM for that reason. + +.. _ksm_sysfs: + +KSM daemon sysfs interface +== + +The KSM daemon is controlled by sysfs files in ``/sys/kernel/mm/ksm/``, +readable by all but writable only by root: + +pages_to_scan +how many pages to scan before ksmd goes to sleep +e.g. ``echo 100 > /sys/kernel/mm/ksm/pages_to_scan``. + +Default: 100 (chosen for demonstration purposes) + +sleep_millisecs +how many milliseconds ksmd should sleep before next scan +e.g. ``echo 20 > /sys/kernel/mm/ksm/sleep_millisecs`` + +Default: 20 (chosen for demonstration purposes) + +merge_across_nodes +specifies if pages from different NUMA nodes can be merged. +When set to 0, ksm merges only pages which physically reside +in the memory area of same NUMA node. That
[PATCH 5/7] docs/vm: ksm: update stable_node_chains_prune_millisecs description
Make the description of stable_node_chains_prune_millisecs sysfs parameter less implementation aware and add a few words about this parameter in the "Design" section. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.rst | 19 +++ 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst index 00961b8..18d7c71 100644 --- a/Documentation/vm/ksm.rst +++ b/Documentation/vm/ksm.rst @@ -150,14 +150,12 @@ max_page_sharing traversals are always schedule friendly themselves. stable_node_chains_prune_millisecs -How frequently to walk the whole list of stable_node "dups" -linked in the stable_node "chains" in order to prune stale -stable_nodes. Smaller milllisecs values will free up the KSM -metadata with lower latency, but they will make ksmd use more -CPU during the scan. This only applies to the stable_node -chains so it's a noop if not a single KSM page hit the -``max_page_sharing`` yet (there would be no stable_node chains in -such case). +specifies how frequently KSM checks the metadata of the pages +that hit the deduplication limit for stale information. +Smaller milllisecs values will free up the KSM metadata with +lower latency, but they will make ksmd use more CPU during the +scan. It's a noop if not a single KSM page hit the +``max_page_sharing`` yet. The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``: @@ -249,6 +247,11 @@ deduplication factor at the expense of slower worst case for rmap walks for any KSM page which can happen during swapping, compaction, NUMA balancing and page migration. +The whole list of stable_node "dups" linked in the stable_node +"chains" is scanned periodically in order to prune stale stable_nodes. +The frequency of such scans is defined by +``stable_node_chains_prune_millisecs`` sysfs tunable. + Reference - .. kernel-doc:: mm/ksm.c -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/7] mm/ksm: docs: extend overview comment and make it "DOC:"
The existing comment provides a good overview of KSM implementation. Let's
update it to reflect recent additions of "chain" and "dup" variants of the
stable tree nodes and mark it as "DOC:" for inclusion into the KSM
documentation.
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
mm/ksm.c | 19 ++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index 58c2741..54155b1 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -51,7 +51,9 @@
#define DO_NUMA(x) do { } while (0)
#endif
-/*
+/**
+ * DOC: Overview
+ *
* A few notes about the KSM scanning process,
* to make it easier to understand the data structures below:
*
@@ -67,6 +69,21 @@
* this tree is fully assured to be working (except when pages are unmapped),
* and therefore this tree is called the stable tree.
*
+ * The stable tree node includes information required for reverse
+ * mapping from a KSM page to virtual addresses that map this page.
+ *
+ * In order to avoid large latencies of the rmap walks on KSM pages,
+ * KSM maintains two types of nodes in the stable tree:
+ *
+ * * the regular nodes that keep the reverse mapping structures in a
+ * linked list
+ * * the "chains" that link nodes ("dups") that represent the same
+ * write protected memory content, but each "dup" corresponds to a
+ * different KSM page copy of that content
+ *
+ * Internally, the regular nodes, "dups" and "chains" are represented
+ * using the same :c:type:`struct stable_node` structure.
+ *
* In addition to the stable tree, KSM uses a second data structure called the
* unstable tree: this tree holds pointers to pages which have been found to
* be "unchanged for a period of time". The unstable tree sorts these pages
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/7] docs/vm: ksm: add "Design" section
Include the KSM description from the source code comment, add a subsection about reverse mapping and include kernel-doc references for KSM data structures. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.rst | 39 +++ 1 file changed, 39 insertions(+) diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst index 786d460..0e5a085 100644 --- a/Documentation/vm/ksm.rst +++ b/Documentation/vm/ksm.rst @@ -206,6 +206,45 @@ stable_node "dups" with few rmap_items in them, but that may increase the ksmd CPU usage and possibly slowdown the readonly computations on the KSM pages of the applications. +Design +== + +Overview + + +.. kernel-doc:: mm/ksm.c + :DOC: Overview + +Reverse mapping +--- +KSM maintains reverse mapping information for KSM pages in the stable +tree. + +If a KSM page is shared between less than ``max_page_sharing`` VMAs, +the node of the stable tree that represents such KSM page points to a +list of :c:type:`struct rmap_item` and the ``page->mapping`` of the +KSM page points to the stable tree node. + +When the sharing passes this threshold, KSM adds a second dimension to +the stable tree. The tree node becomes a "chain" that links one or +more "dups". Each "dup" keeps reverse mapping information for a KSM +page with ``page->mapping`` pointing to that "dup". + +Every "chain" and all "dups" linked into a "chain" enforce the +invariant that they represent the same write protected memory content, +even if each "dup" will be pointed by a different KSM page copy of +that content. + +This way the stable tree lookup computational complexity is unaffected +if compared to an unlimited list of reverse mappings. It is still +enforced that there cannot be KSM page content duplicates in the +stable tree itself. + +Reference +- +.. kernel-doc:: mm/ksm.c + :functions: mm_slot ksm_scan stable_node rmap_item + -- Izik Eidus, Hugh Dickins, 17 Nov 2009 -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/7] docs/vm: update KSM documentation
Hi,
These patches extend KSM documentation with high level design overview and
some details about reverse mappings and split the userspace interface
description to Documentation/admin-guide/mm.
The description of some KSM sysfs attributes is changed so that it won't
include implementation detail. The description of these implementation
details are moved to the new "Design" section.
The last patch in the series depends on the patchset that create
Documentation/admin-guide/mm [1], all the rest applies cleanly to the
current docs-next.
[1] https://lkml.org/lkml/2018/4/18/110
Mike Rapoport (7):
mm/ksm: docs: extend overview comment and make it "DOC:"
docs/vm: ksm: (mostly) formatting updates
docs/vm: ksm: add "Design" section
docs/vm: ksm: reshuffle text between "sysfs" and "design" sections
docs/vm: ksm: update stable_node_chains_prune_millisecs description
docs/vm: ksm: udpate description of stable_node_{dups,chains}
docs/vm: ksm: split userspace interface to admin-guide/mm/ksm.rst
Documentation/admin-guide/mm/index.rst | 1 +
Documentation/admin-guide/mm/ksm.rst | 189 ++
Documentation/vm/ksm.rst | 234 ++---
mm/ksm.c | 19 ++-
4 files changed, 277 insertions(+), 166 deletions(-)
create mode 100644 Documentation/admin-guide/mm/ksm.rst
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/7] docs/vm: ksm: reshuffle text between "sysfs" and "design" sections
The description of "max_page_sharing" sysfs attribute includes lots of implementation details that more naturally belong in the "Design" section. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.rst | 51 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst index 0e5a085..00961b8 100644 --- a/Documentation/vm/ksm.rst +++ b/Documentation/vm/ksm.rst @@ -133,31 +133,21 @@ use_zero_pages max_page_sharing Maximum sharing allowed for each KSM page. This enforces a -deduplication limit to avoid the virtual memory rmap lists to -grow too large. The minimum value is 2 as a newly created KSM -page will have at least two sharers. The rmap walk has O(N) -complexity where N is the number of rmap_items (i.e. virtual -mappings) that are sharing the page, which is in turn capped -by ``max_page_sharing``. So this effectively spreads the linear -O(N) computational complexity from rmap walk context over -different KSM pages. The ksmd walk over the stable_node -"chains" is also O(N), but N is the number of stable_node -"dups", not the number of rmap_items, so it has not a -significant impact on ksmd performance. In practice the best -stable_node "dup" candidate will be kept and found at the head -of the "dups" list. The higher this value the faster KSM will -merge the memory (because there will be fewer stable_node dups -queued into the stable_node chain->hlist to check for pruning) -and the higher the deduplication factor will be, but the -slowest the worst case rmap walk could be for any given KSM -page. Slowing down the rmap_walk means there will be higher +deduplication limit to avoid high latency for virtual memory +operations that involve traversal of the virtual mappings that +share the KSM page. The minimum value is 2 as a newly created +KSM page will have at least two sharers. The higher this value +the faster KSM will merge the memory and the higher the +deduplication factor will be, but the slower the worst case +virtual mappings traversal could be for any given KSM +page. Slowing down this traversal means there will be higher latency for certain virtual memory operations happening during swapping, compaction, NUMA balancing and page migration, in turn decreasing responsiveness for the caller of those virtual memory operations. The scheduler latency of other tasks not -involved with the VM operations doing the rmap walk is not -affected by this parameter as the rmap walks are always -schedule friendly themselves. +involved with the VM operations doing the virtual mappings +traversal is not affected by this parameter as these +traversals are always schedule friendly themselves. stable_node_chains_prune_millisecs How frequently to walk the whole list of stable_node "dups" @@ -240,6 +230,25 @@ if compared to an unlimited list of reverse mappings. It is still enforced that there cannot be KSM page content duplicates in the stable tree itself. +The deduplication limit enforced by ``max_page_sharing`` is required +to avoid the virtual memory rmap lists to grow too large. The rmap +walk has O(N) complexity where N is the number of rmap_items +(i.e. virtual mappings) that are sharing the page, which is in turn +capped by ``max_page_sharing``. So this effectively spreads the linear +O(N) computational complexity from rmap walk context over different +KSM pages. The ksmd walk over the stable_node "chains" is also O(N), +but N is the number of stable_node "dups", not the number of +rmap_items, so it has not a significant impact on ksmd performance. In +practice the best stable_node "dup" candidate will be kept and found +at the head of the "dups" list. + +High values of ``max_page_sharing`` result in faster memory merging +(because there will be fewer stable_node dups queued into the +stable_node chain->hlist to check for pruning) and higher +deduplication factor at the expense of slower worst case for rmap +walks for any KSM page which can happen during swapping, compaction, +NUMA balancing and page migration. + Reference - .. kernel-doc:: mm/ksm.c -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/7] docs/vm: pagemap: formatting and spelling updates
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/pagemap.rst | 16 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/vm/pagemap.rst b/Documentation/vm/pagemap.rst index d54b4bf..9644bc0 100644 --- a/Documentation/vm/pagemap.rst +++ b/Documentation/vm/pagemap.rst @@ -13,7 +13,7 @@ There are four components to pagemap: * ``/proc/pid/pagemap``. This file lets a userspace process find out which physical frame each virtual page is mapped to. It contains one 64-bit value for each virtual page, containing the following data (from - fs/proc/task_mmu.c, above pagemap_read): + ``fs/proc/task_mmu.c``, above pagemap_read): * Bits 0-54 page frame number (PFN) if present * Bits 0-4 swap type if swapped @@ -36,7 +36,7 @@ There are four components to pagemap: precisely which pages are mapped (or in swap) and comparing mapped pages between processes. - Efficient users of this interface will use /proc/pid/maps to + Efficient users of this interface will use ``/proc/pid/maps`` to determine which areas of memory are actually mapped and llseek to skip over unmapped regions. @@ -79,11 +79,11 @@ There are four components to pagemap: memory cgroup each page is charged to, indexed by PFN. Only available when CONFIG_MEMCG is set. -Short descriptions to the page flags: -= +Short descriptions to the page flags + 0 - LOCKED - page is being locked for exclusive access, eg. by undergoing read/write IO + page is being locked for exclusive access, e.g. by undergoing read/write IO 7 - SLAB page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator When compound page is used, SLUB/SLQB will only set this flag on the head @@ -132,7 +132,7 @@ IO related page flags ie. for file backed page: (in-memory data revision >= on-disk one) 4 - DIRTY page has been written to, hence contains new data - ie. for file backed page: (in-memory data revision > on-disk one) + i.e. for file backed page: (in-memory data revision > on-disk one) 8 - WRITEBACK page is being synced to disk @@ -145,7 +145,7 @@ LRU related page flags page is in the active LRU list 18 - UNEVICTABLE page is in the unevictable (non-)LRU list It is somehow pinned and - not a candidate for LRU page reclaims, eg. ramfs pages, + not a candidate for LRU page reclaims, e.g. ramfs pages, shmctl(SHM_LOCK) and mlock() memory segments 2 - REFERENCED page has been referenced since last LRU list enqueue/requeue @@ -156,7 +156,7 @@ LRU related page flags 12 - ANON a memory mapped page that is not part of a file 13 - SWAPCACHE - page is mapped to swap space, ie. has an associated swap entry + page is mapped to swap space, i.e. has an associated swap entry 14 - SWAPBACKED page is backed by swap/RAM -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/7] docs/vm: hugetlbpage: move section about kernel development to hugetlbfs_reserv
The hugetlbpage describes hugetlbfs from the user perspective and newer hugetlbfs_reserv document targets kernel developers. Hence the section about hugetlbfs kernel development naturally belongs there. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/hugetlbfs_reserv.rst | 8 Documentation/vm/hugetlbpage.rst | 8 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/vm/hugetlbfs_reserv.rst b/Documentation/vm/hugetlbfs_reserv.rst index 36a87a2..9d20076 100644 --- a/Documentation/vm/hugetlbfs_reserv.rst +++ b/Documentation/vm/hugetlbfs_reserv.rst @@ -583,5 +583,13 @@ of cpusets or memory policy there is no guarantee that huge pages will be available on the required nodes. This is true even if there are a sufficient number of global reservations. +Hugetlbfs regression testing + +The most complete set of hugetlb tests are in the libhugetlbfs repository. +If you modify any hugetlb related code, use the libhugetlbfs test suite +to check for regressions. In addition, if you add any new hugetlb +functionality, please add appropriate tests to libhugetlbfs. + +-- Mike Kravetz, 7 April 2017 diff --git a/Documentation/vm/hugetlbpage.rst b/Documentation/vm/hugetlbpage.rst index 99ad5d9..2b374d1 100644 --- a/Documentation/vm/hugetlbpage.rst +++ b/Documentation/vm/hugetlbpage.rst @@ -379,11 +379,3 @@ The `libhugetlbfs`_ library provides a wide range of userspace tools to help with huge page usability, environment setup, and control. .. _libhugetlbfs: https://github.com/libhugetlbfs/libhugetlbfs - -Kernel development regression testing -= - -The most complete set of hugetlb tests are in the libhugetlbfs repository. -If you modify any hugetlb related code, use the libhugetlbfs test suite -to check for regressions. In addition, if you add any new hugetlb -functionality, please add appropriate tests to libhugetlbfs. -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 6/7] docs/admin-guide/mm: start moving here files from Documentation/vm
Several documents in Documentation/vm fit quite well into the "admin/user
guide" category. The documents that don't overload the reader with lots of
implementation details and provide coherent description of certain feature
can be moved to Documentation/admin-guide/mm.
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/ABI/stable/sysfs-devices-node | 2 +-
Documentation/ABI/testing/sysfs-kernel-mm-hugepages | 2 +-
Documentation/{vm => admin-guide/mm}/hugetlbpage.rst| 0
Documentation/{vm => admin-guide/mm}/idle_page_tracking.rst | 2 +-
Documentation/admin-guide/mm/index.rst | 9 +
Documentation/{vm => admin-guide/mm}/pagemap.rst| 6 +++---
Documentation/{vm => admin-guide/mm}/soft-dirty.rst | 0
Documentation/{vm => admin-guide/mm}/userfaultfd.rst| 0
Documentation/filesystems/proc.txt | 6 --
Documentation/sysctl/vm.txt | 4 ++--
Documentation/vm/00-INDEX | 10 --
Documentation/vm/hwpoison.rst | 2 +-
Documentation/vm/index.rst | 5 -
fs/Kconfig | 2 +-
fs/proc/task_mmu.c | 4 ++--
mm/Kconfig | 5 +++--
16 files changed, 28 insertions(+), 31 deletions(-)
rename Documentation/{vm => admin-guide/mm}/hugetlbpage.rst (100%)
rename Documentation/{vm => admin-guide/mm}/idle_page_tracking.rst (98%)
rename Documentation/{vm => admin-guide/mm}/pagemap.rst (96%)
rename Documentation/{vm => admin-guide/mm}/soft-dirty.rst (100%)
rename Documentation/{vm => admin-guide/mm}/userfaultfd.rst (100%)
diff --git a/Documentation/ABI/stable/sysfs-devices-node
b/Documentation/ABI/stable/sysfs-devices-node
index b38f4b7..3e90e1f 100644
--- a/Documentation/ABI/stable/sysfs-devices-node
+++ b/Documentation/ABI/stable/sysfs-devices-node
@@ -90,4 +90,4 @@ Date: December 2009
Contact: Lee Schermerhorn <lee.schermerh...@hp.com>
Description:
The node's huge page size control/query attributes.
- See Documentation/vm/hugetlbpage.rst
\ No newline at end of file
+ See Documentation/admin-guide/mm/hugetlbpage.rst
\ No newline at end of file
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-hugepages
b/Documentation/ABI/testing/sysfs-kernel-mm-hugepages
index 5140b23..fdaa216 100644
--- a/Documentation/ABI/testing/sysfs-kernel-mm-hugepages
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-hugepages
@@ -12,4 +12,4 @@ Description:
free_hugepages
surplus_hugepages
resv_hugepages
- See Documentation/vm/hugetlbpage.rst for details.
+ See Documentation/admin-guide/mm/hugetlbpage.rst for details.
diff --git a/Documentation/vm/hugetlbpage.rst
b/Documentation/admin-guide/mm/hugetlbpage.rst
similarity index 100%
rename from Documentation/vm/hugetlbpage.rst
rename to Documentation/admin-guide/mm/hugetlbpage.rst
diff --git a/Documentation/vm/idle_page_tracking.rst
b/Documentation/admin-guide/mm/idle_page_tracking.rst
similarity index 98%
rename from Documentation/vm/idle_page_tracking.rst
rename to Documentation/admin-guide/mm/idle_page_tracking.rst
index d1c4609..92e3a25 100644
--- a/Documentation/vm/idle_page_tracking.rst
+++ b/Documentation/admin-guide/mm/idle_page_tracking.rst
@@ -65,7 +65,7 @@ workload one should:
are not reclaimable, he or she can filter them out using
``/proc/kpageflags``.
-See Documentation/vm/pagemap.rst for more information about
+See Documentation/admin-guide/mm/pagemap.rst for more information about
``/proc/pid/pagemap``, ``/proc/kpageflags``, and ``/proc/kpagecgroup``.
.. _impl_details:
diff --git a/Documentation/admin-guide/mm/index.rst
b/Documentation/admin-guide/mm/index.rst
index c47c16e..6c8b554 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -17,3 +17,12 @@ are described in Documentation/sysctl/vm.txt and in `man 5
proc`_.
Here we document in detail how to interact with various mechanisms in
the Linux memory management.
+
+.. toctree::
+ :maxdepth: 1
+
+ hugetlbpage
+ idle_page_tracking
+ pagemap
+ soft-dirty
+ userfaultfd
diff --git a/Documentation/vm/pagemap.rst
b/Documentation/admin-guide/mm/pagemap.rst
similarity index 96%
rename from Documentation/vm/pagemap.rst
rename to Documentation/admin-guide/mm/pagemap.rst
index 7ba8cbd..053ca64 100644
--- a/Documentation/vm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -18,7 +18,7 @@ There are four components to pagemap:
* Bits 0-54 page frame number (PFN) if present
* Bits 0-4 swap type if swapped
[PATCH 7/7] docs/admin-guide/mm: convert plain text cross references to hyperlinks
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/admin-guide/mm/hugetlbpage.rst| 3 ++- Documentation/admin-guide/mm/idle_page_tracking.rst | 5 +++-- Documentation/admin-guide/mm/pagemap.rst| 18 +++--- 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 2b374d1..a8b0806 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -219,7 +219,8 @@ When adjusting the persistent hugepage count via ``nr_hugepages_mempolicy``, any memory policy mode--bind, preferred, local or interleave--may be used. The resulting effect on persistent huge page allocation is as follows: -#. Regardless of mempolicy mode [see Documentation/vm/numa_memory_policy.rst], +#. Regardless of mempolicy mode [see + :ref:`Documentation/vm/numa_memory_policy.rst `], persistent huge pages will be distributed across the node or nodes specified in the mempolicy as if "interleave" had been specified. However, if a node in the policy does not contain sufficient contiguous diff --git a/Documentation/admin-guide/mm/idle_page_tracking.rst b/Documentation/admin-guide/mm/idle_page_tracking.rst index 92e3a25..6f7b7ca 100644 --- a/Documentation/admin-guide/mm/idle_page_tracking.rst +++ b/Documentation/admin-guide/mm/idle_page_tracking.rst @@ -65,8 +65,9 @@ workload one should: are not reclaimable, he or she can filter them out using ``/proc/kpageflags``. -See Documentation/admin-guide/mm/pagemap.rst for more information about -``/proc/pid/pagemap``, ``/proc/kpageflags``, and ``/proc/kpagecgroup``. +See :ref:`Documentation/admin-guide/mm/pagemap.rst ` for more +information about ``/proc/pid/pagemap``, ``/proc/kpageflags``, and +``/proc/kpagecgroup``. .. _impl_details: diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index 053ca64..577af85 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -18,7 +18,8 @@ There are four components to pagemap: * Bits 0-54 page frame number (PFN) if present * Bits 0-4 swap type if swapped * Bits 5-54 swap offset if swapped -* Bit 55pte is soft-dirty (see Documentation/admin-guide/mm/soft-dirty.rst) +* Bit 55pte is soft-dirty (see + :ref:`Documentation/admin-guide/mm/soft-dirty.rst `) * Bit 56page exclusively mapped (since 4.2) * Bits 57-60 zero * Bit 61page is file-page or shared-anon (since 3.5) @@ -97,9 +98,11 @@ Short descriptions to the page flags A compound page with order N consists of 2^N physically contiguous pages. A compound page with order 2 takes the form of "HTTT", where H donates its head page and T donates its tail page(s). The major consumers of compound -pages are hugeTLB pages (Documentation/admin-guide/mm/hugetlbpage.rst), the SLUB etc. -memory allocators and various device drivers. However in this interface, -only huge/giga pages are made visible to end users. +pages are hugeTLB pages +(:ref:`Documentation/admin-guide/mm/hugetlbpage.rst `), +the SLUB etc. memory allocators and various device drivers. +However in this interface, only huge/giga pages are made visible +to end users. 16 - COMPOUND_TAIL A compound page tail (see description above). 17 - HUGE @@ -118,9 +121,10 @@ Short descriptions to the page flags zero page for pfn_zero or huge_zero page 25 - IDLE page has not been accessed since it was marked idle (see -Documentation/admin-guide/mm/idle_page_tracking.rst). Note that this flag may be -stale in case the page was accessed via a PTE. To make sure the flag -is up-to-date one has to read ``/sys/kernel/mm/page_idle/bitmap`` first. +:ref:`Documentation/admin-guide/mm/idle_page_tracking.rst `). +Note that this flag may be stale in case the page was accessed via +a PTE. To make sure the flag is up-to-date one has to read +``/sys/kernel/mm/page_idle/bitmap`` first. IO related page flags - -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 5/7] docs/admin-guide: introduce basic index for mm documentation
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/admin-guide/index.rst| 1 + Documentation/admin-guide/mm/index.rst | 19 +++ 2 files changed, 20 insertions(+) create mode 100644 Documentation/admin-guide/mm/index.rst diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst index 5bb9161..cac906f 100644 --- a/Documentation/admin-guide/index.rst +++ b/Documentation/admin-guide/index.rst @@ -63,6 +63,7 @@ configure specific aspects of kernel behavior to your liking. pm/index thunderbolt LSM/index + mm/index .. only:: subproject and html diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst new file mode 100644 index 000..c47c16e --- /dev/null +++ b/Documentation/admin-guide/mm/index.rst @@ -0,0 +1,19 @@ += +Memory Management += + +Linux memory management subsystem is responsible, as the name implies, +for managing the memory in the system. This includes implemnetation of +virtual memory and demand paging, memory allocation both for kernel +internal structures and user space programms, mapping of files into +processes address space and many other cool things. + +Linux memory management is a complex system with many configurable +settings. Most of these settings are available via ``/proc`` +filesystem and can be quired and adjusted using ``sysctl``. These APIs +are described in Documentation/sysctl/vm.txt and in `man 5 proc`_. + +.. _man 5 proc: http://man7.org/linux/man-pages/man5/proc.5.html + +Here we document in detail how to interact with various mechanisms in +the Linux memory management. -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/7] docs/vm: pagemap: change document title
"pagemap from the Userspace Perspective" is not very descriptive for unaware readers. Since the document describes how to examine a process page tables, let's title it "Examining Process Page Tables" Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/pagemap.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/vm/pagemap.rst b/Documentation/vm/pagemap.rst index 9644bc0..7ba8cbd 100644 --- a/Documentation/vm/pagemap.rst +++ b/Documentation/vm/pagemap.rst @@ -1,8 +1,8 @@ .. _pagemap: -== -pagemap from the Userspace Perspective -== += +Examining Process Page Tables += pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow userspace programs to examine the page tables and related information by -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/7] docs/vm: start moving files do Documentation/admin-guide`
Hi,
These pacthes begin categorizing memory management documentation. The
documents that describe userspace APIs and do not overload the reader with
implementation details can be moved to Documentation/admin-guide, so let's
do it :)
Mike Rapoport (7):
docs/vm: hugetlbpage: minor improvements
docs/vm: hugetlbpage: move section about kernel development to
hugetlbfs_reserv
docs/vm: pagemap: formatting and spelling updates
docs/vm: pagemap: change document title
docs/admin-guide: introduce basic index for mm documentation
docs/admin-guide/mm: start moving here files from Documentation/vm
docs/admin-guide/mm: convert plain text cross references to hyperlinks
Documentation/ABI/stable/sysfs-devices-node| 2 +-
.../ABI/testing/sysfs-kernel-mm-hugepages | 2 +-
Documentation/admin-guide/index.rst| 1 +
.../{vm => admin-guide/mm}/hugetlbpage.rst | 28 +++
.../{vm => admin-guide/mm}/idle_page_tracking.rst | 5 +--
Documentation/admin-guide/mm/index.rst | 28 +++
Documentation/{vm => admin-guide/mm}/pagemap.rst | 40 --
.../{vm => admin-guide/mm}/soft-dirty.rst | 0
.../{vm => admin-guide/mm}/userfaultfd.rst | 0
Documentation/filesystems/proc.txt | 6 ++--
Documentation/sysctl/vm.txt| 4 +--
Documentation/vm/00-INDEX | 10 --
Documentation/vm/hugetlbfs_reserv.rst | 8 +
Documentation/vm/hwpoison.rst | 2 +-
Documentation/vm/index.rst | 5 ---
fs/Kconfig | 2 +-
fs/proc/task_mmu.c | 4 +--
mm/Kconfig | 5 +--
18 files changed, 89 insertions(+), 63 deletions(-)
rename Documentation/{vm => admin-guide/mm}/hugetlbpage.rst (95%)
rename Documentation/{vm => admin-guide/mm}/idle_page_tracking.rst (96%)
create mode 100644 Documentation/admin-guide/mm/index.rst
rename Documentation/{vm => admin-guide/mm}/pagemap.rst (83%)
rename Documentation/{vm => admin-guide/mm}/soft-dirty.rst (100%)
rename Documentation/{vm => admin-guide/mm}/userfaultfd.rst (100%)
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/7] docs/vm: hugetlbpage: minor improvements
* fixed mistypes * added internal cross-references for sections Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/hugetlbpage.rst | 17 ++--- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/Documentation/vm/hugetlbpage.rst b/Documentation/vm/hugetlbpage.rst index a5da14b..99ad5d9 100644 --- a/Documentation/vm/hugetlbpage.rst +++ b/Documentation/vm/hugetlbpage.rst @@ -87,7 +87,7 @@ memory pressure. Once a number of huge pages have been pre-allocated to the kernel huge page pool, a user with appropriate privilege can use either the mmap system call or shared memory system calls to use the huge pages. See the discussion of -Using Huge Pages, below. +:ref:`Using Huge Pages `, below. The administrator can allocate persistent huge pages on the kernel boot command line by specifying the "hugepages=N" parameter, where 'N' = the @@ -115,8 +115,9 @@ over all the set of allowed nodes specified by the NUMA memory policy of the task that modifies ``nr_hugepages``. The default for the allowed nodes--when the task has default memory policy--is all on-line nodes with memory. Allowed nodes with insufficient available, contiguous memory for a huge page will be -silently skipped when allocating persistent huge pages. See the discussion -below of the interaction of task memory policy, cpusets and per node attributes +silently skipped when allocating persistent huge pages. See the +:ref:`discussion below ` +of the interaction of task memory policy, cpusets and per node attributes with the allocation and freeing of persistent huge pages. The success or failure of huge page allocation depends on the amount of @@ -158,7 +159,7 @@ normal page pool. Caveat: Shrinking the persistent huge page pool via ``nr_hugepages`` such that it becomes less than the number of huge pages in use will convert the balance of the in-use huge pages to surplus huge pages. This will occur even if -the number of surplus pages it would exceed the overcommit value. As long as +the number of surplus pages would exceed the overcommit value. As long as this condition holds--that is, until ``nr_hugepages+nr_overcommit_hugepages`` is increased sufficiently, or the surplus huge pages go out of use and are freed-- no more surplus huge pages will be allowed to be allocated. @@ -187,6 +188,7 @@ Inside each of these directories, the same set of files will exist:: which function as described above for the default huge page-sized case. +.. _mem_policy_and_hp_alloc: Interaction of Task Memory Policy with Huge Page Allocation/Freeing === @@ -282,6 +284,7 @@ Note that the number of overcommit and reserve pages remain global quantities, as we don't know until fault time, when the faulting task's mempolicy is applied, from which node the huge page allocation will be attempted. +.. _using_huge_pages: Using Huge Pages @@ -295,7 +298,7 @@ type hugetlbfs:: min_size=,nr_inodes= none /mnt/huge This command mounts a (pseudo) filesystem of type hugetlbfs on the directory -``/mnt/huge``. Any files created on ``/mnt/huge`` uses huge pages. +``/mnt/huge``. Any file created on ``/mnt/huge`` uses huge pages. The ``uid`` and ``gid`` options sets the owner and group of the root of the file system. By default the ``uid`` and ``gid`` of the current process @@ -345,8 +348,8 @@ applications are going to use only shmat/shmget system calls or mmap with MAP_HUGETLB. For an example of how to use mmap with MAP_HUGETLB see :ref:`map_hugetlb ` below. -Users who wish to use hugetlb memory via shared memory segment should be a -member of a supplementary group and system admin needs to configure that gid +Users who wish to use hugetlb memory via shared memory segment should be +members of a supplementary group and system admin needs to configure that gid into ``/proc/sys/vm/hugetlb_shm_group``. It is possible for same or different applications to use any combination of mmaps and shm* calls, though the mount of filesystem will be required for using mmap calls without MAP_HUGETLB. -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/32] docs/vm: convert to ReST format
On Fri, Apr 13, 2018 at 01:55:51PM -0600, Jonathan Corbet wrote: > Sorry for the silence, I'm pedaling as fast as I can, honest... > > On Sun, 1 Apr 2018 09:38:58 +0300 > Mike Rapoport <r...@linux.vnet.ibm.com> wrote: > > > My thinking was to start with mechanical RST conversion and then to start > > working on the contents and ordering of the documentation. Some of the > > existing files, e.g. ksm.txt, can be moved as is into the appropriate > > places, others, like transhuge.txt should be at least split into admin/user > > and developer guides. > > > > Another problem with many of the existing mm docs is that they are rather > > developer notes and it wouldn't be really straight forward to assign them > > to a particular topic. > > All this sounds good. > > > I believe that keeping the mm docs together will give better visibility of > > what (little) mm documentation we have and will make the updates easier. > > The documents that fit well into a certain topic could be linked there. For > > instance: > > ...but this sounds like just the opposite...? > > I've had this conversation with folks in a number of subsystems. > Everybody wants to keep their documentation together in one place - it's > easier for the developers after all. But for the readers I think it's > objectively worse. It perpetuates the mess that Documentation/ is, and > forces readers to go digging through all kinds of inappropriate material > in the hope of finding something that tells them what they need to know. > > So I would *really* like to split the documentation by audience, as has > been done for a number of other kernel subsystems (and eventually all, I > hope). > > I can go ahead and apply the RST conversion, that seems like a step in > the right direction regardless. But I sure hope we don't really have to > keep it as an unorganized jumble of stuff... I didn't mean we should keep it as unorganized jumble of stuff and I agree that splitting the documentation by audience is better because developers are already know how to find it :) I just thought that putting the doc into the place should not be done immediately after mechanical ReST conversion but rather after improving the contents. Although I'd agree that part of the documentation in Documentation/vm is in pretty good shape already. > Thanks, > > jon > -- Sincerely yours, Mike. -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/32] docs/vm: convert to ReST format
Jon, Andrew, How do you suggest to continue with this? On Sun, Apr 01, 2018 at 09:38:58AM +0300, Mike Rapoport wrote: > (added akpm) > > On Thu, Mar 29, 2018 at 03:46:07PM -0600, Jonathan Corbet wrote: > > On Wed, 21 Mar 2018 21:22:16 +0200 > > Mike Rapoport <r...@linux.vnet.ibm.com> wrote: > > > > > These patches convert files in Documentation/vm to ReST format, add an > > > initial index and link it to the top level documentation. > > > > > > There are no contents changes in the documentation, except few spelling > > > fixes. The relatively large diffstat stems from the indentation and > > > paragraph wrapping changes. > > > > > > I've tried to keep the formatting as consistent as possible, but I could > > > miss some places that needed markup and add some markup where it was not > > > necessary. > > > > So I've been pondering on these for a bit. It looks like a reasonable and > > straightforward RST conversion, no real complaints there. But I do have a > > couple of concerns... > > > > One is that, as we move documentation into RST, I'm really trying to > > organize it a bit so that it is better tuned to the various audiences we > > have. For example, ksm.txt is going to be of interest to sysadmin types, > > who might want to tune it. mmu_notifier.txt is of interest to ... > > somebody, but probably nobody who is thinking in user space. And so on. > > > > So I would really like to see this material split up and put into the > > appropriate places in the RST hierarchy - admin-guide for administrative > > stuff, core-api for kernel development topics, etc. That, of course, > > could be done separately from the RST conversion, but I suspect I know > > what will (or will not) happen if we agree to defer that for now :) > > Well, I was actually planning on doing that ;-) > > My thinking was to start with mechanical RST conversion and then to start > working on the contents and ordering of the documentation. Some of the > existing files, e.g. ksm.txt, can be moved as is into the appropriate > places, others, like transhuge.txt should be at least split into admin/user > and developer guides. > > Another problem with many of the existing mm docs is that they are rather > developer notes and it wouldn't be really straight forward to assign them > to a particular topic. > > I believe that keeping the mm docs together will give better visibility of > what (little) mm documentation we have and will make the updates easier. > The documents that fit well into a certain topic could be linked there. For > instance: > > - > diff --git a/Documentation/admin-guide/index.rst > b/Documentation/admin-guide/index.rst > index 5bb9161..8f6c6e6 100644 > --- a/Documentation/admin-guide/index.rst > +++ b/Documentation/admin-guide/index.rst > @@ -63,6 +63,7 @@ configure specific aspects of kernel behavior to your > liking. > pm/index > thunderbolt > LSM/index > + vm/index > > .. only:: subproject and html > > diff --git a/Documentation/admin-guide/vm/index.rst > b/Documentation/admin-guide/vm/index.rst > new file mode 100644 > index 000..d86f1c8 > --- /dev/null > +++ b/Documentation/admin-guide/vm/index.rst > @@ -0,0 +1,5 @@ > +== > +Knobs and Buttons for Memory Management Tuning > +== > + > +* :ref:`ksm ` > - > > > The other is the inevitable merge conflicts that changing that many doc > > files will create. Sending the patches through Andrew could minimize > > that, I guess, or at least make it his problem. Alternatively, we could > > try to do it as an end-of-merge-window sort of thing. I can try to manage > > that, but an ack or two from the mm crowd would be nice to have. > > I can rebase on top of Andrew's tree if that would help to minimize the > merge conflicts. > > > Thanks, > > > > jon > > > > -- > Sincerely yours, > Mike. > -- Sincerely yours, Mike. -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/32] docs/vm: convert to ReST format
(added akpm) On Thu, Mar 29, 2018 at 03:46:07PM -0600, Jonathan Corbet wrote: > On Wed, 21 Mar 2018 21:22:16 +0200 > Mike Rapoport <r...@linux.vnet.ibm.com> wrote: > > > These patches convert files in Documentation/vm to ReST format, add an > > initial index and link it to the top level documentation. > > > > There are no contents changes in the documentation, except few spelling > > fixes. The relatively large diffstat stems from the indentation and > > paragraph wrapping changes. > > > > I've tried to keep the formatting as consistent as possible, but I could > > miss some places that needed markup and add some markup where it was not > > necessary. > > So I've been pondering on these for a bit. It looks like a reasonable and > straightforward RST conversion, no real complaints there. But I do have a > couple of concerns... > > One is that, as we move documentation into RST, I'm really trying to > organize it a bit so that it is better tuned to the various audiences we > have. For example, ksm.txt is going to be of interest to sysadmin types, > who might want to tune it. mmu_notifier.txt is of interest to ... > somebody, but probably nobody who is thinking in user space. And so on. > > So I would really like to see this material split up and put into the > appropriate places in the RST hierarchy - admin-guide for administrative > stuff, core-api for kernel development topics, etc. That, of course, > could be done separately from the RST conversion, but I suspect I know > what will (or will not) happen if we agree to defer that for now :) Well, I was actually planning on doing that ;-) My thinking was to start with mechanical RST conversion and then to start working on the contents and ordering of the documentation. Some of the existing files, e.g. ksm.txt, can be moved as is into the appropriate places, others, like transhuge.txt should be at least split into admin/user and developer guides. Another problem with many of the existing mm docs is that they are rather developer notes and it wouldn't be really straight forward to assign them to a particular topic. I believe that keeping the mm docs together will give better visibility of what (little) mm documentation we have and will make the updates easier. The documents that fit well into a certain topic could be linked there. For instance: - diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst index 5bb9161..8f6c6e6 100644 --- a/Documentation/admin-guide/index.rst +++ b/Documentation/admin-guide/index.rst @@ -63,6 +63,7 @@ configure specific aspects of kernel behavior to your liking. pm/index thunderbolt LSM/index + vm/index .. only:: subproject and html diff --git a/Documentation/admin-guide/vm/index.rst b/Documentation/admin-guide/vm/index.rst new file mode 100644 index 000..d86f1c8 --- /dev/null +++ b/Documentation/admin-guide/vm/index.rst @@ -0,0 +1,5 @@ +== +Knobs and Buttons for Memory Management Tuning +== + +* :ref:`ksm ` - > The other is the inevitable merge conflicts that changing that many doc > files will create. Sending the patches through Andrew could minimize > that, I guess, or at least make it his problem. Alternatively, we could > try to do it as an end-of-merge-window sort of thing. I can try to manage > that, but an ack or two from the mm crowd would be nice to have. I can rebase on top of Andrew's tree if that would help to minimize the merge conflicts. > Thanks, > > jon > -- Sincerely yours, Mike. -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 02/32] docs/vm: balance: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/balance | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/Documentation/vm/balance b/Documentation/vm/balance index 9645954..6a1fadf 100644 --- a/Documentation/vm/balance +++ b/Documentation/vm/balance @@ -1,3 +1,9 @@ +.. _balance: + + +Memory Balancing + + Started Jan 2000 by Kanoj Sarcar <ka...@sgi.com> Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as @@ -62,11 +68,11 @@ for non-sleepable allocations. Second, the HIGHMEM zone is also balanced, so as to give a fighting chance for replace_with_highmem() to get a HIGHMEM page, as well as to ensure that HIGHMEM allocations do not fall back into regular zone. This also makes sure that HIGHMEM pages -are not leaked (for example, in situations where a HIGHMEM page is in +are not leaked (for example, in situations where a HIGHMEM page is in the swapcache but is not being used by anyone) kswapd also needs to know about the zones it should balance. kswapd is -primarily needed in a situation where balancing can not be done, +primarily needed in a situation where balancing can not be done, probably because all allocation requests are coming from intr context and all process contexts are sleeping. For 2.3, kswapd does not really need to balance the highmem zone, since intr context does not request @@ -89,7 +95,8 @@ pages is below watermark[WMARK_LOW]; in which case zone_wake_kswapd is also set. (Good) Ideas that I have heard: + 1. Dynamic experience should influence balancing: number of failed requests -for a zone can be tracked and fed into the balancing scheme (ja...@mbay.net) + for a zone can be tracked and fed into the balancing scheme (ja...@mbay.net) 2. Implement a replace_with_highmem()-like replace_with_regular() to preserve -dma pages. (l...@tantalophile.demon.co.uk) + dma pages. (l...@tantalophile.demon.co.uk) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 01/32] docs/vm: active_mm.txt convert to ReST format
Just add a label for cross-referencing and indent the text to make it ``literal`` Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/active_mm.txt | 174 + 1 file changed, 91 insertions(+), 83 deletions(-) diff --git a/Documentation/vm/active_mm.txt b/Documentation/vm/active_mm.txt index dbf4581..c84471b 100644 --- a/Documentation/vm/active_mm.txt +++ b/Documentation/vm/active_mm.txt @@ -1,83 +1,91 @@ -List: linux-kernel -Subject:Re: active_mm -From: Linus Torvalds -Date: 1999-07-30 21:36:24 - -Cc'd to linux-kernel, because I don't write explanations all that often, -and when I do I feel better about more people reading them. - -On Fri, 30 Jul 1999, David Mosberger wrote: -> -> Is there a brief description someplace on how "mm" vs. "active_mm" in -> the task_struct are supposed to be used? (My apologies if this was -> discussed on the mailing lists---I just returned from vacation and -> wasn't able to follow linux-kernel for a while). - -Basically, the new setup is: - - - we have "real address spaces" and "anonymous address spaces". The - difference is that an anonymous address space doesn't care about the - user-level page tables at all, so when we do a context switch into an - anonymous address space we just leave the previous address space - active. - - The obvious use for a "anonymous address space" is any thread that - doesn't need any user mappings - all kernel threads basically fall into - this category, but even "real" threads can temporarily say that for - some amount of time they are not going to be interested in user space, - and that the scheduler might as well try to avoid wasting time on - switching the VM state around. Currently only the old-style bdflush - sync does that. - - - "tsk->mm" points to the "real address space". For an anonymous process, - tsk->mm will be NULL, for the logical reason that an anonymous process - really doesn't _have_ a real address space at all. - - - however, we obviously need to keep track of which address space we - "stole" for such an anonymous user. For that, we have "tsk->active_mm", - which shows what the currently active address space is. - - The rule is that for a process with a real address space (ie tsk->mm is - non-NULL) the active_mm obviously always has to be the same as the real - one. - - For a anonymous process, tsk->mm == NULL, and tsk->active_mm is the - "borrowed" mm while the anonymous process is running. When the - anonymous process gets scheduled away, the borrowed address space is - returned and cleared. - -To support all that, the "struct mm_struct" now has two counters: a -"mm_users" counter that is how many "real address space users" there are, -and a "mm_count" counter that is the number of "lazy" users (ie anonymous -users) plus one if there are any real users. - -Usually there is at least one real user, but it could be that the real -user exited on another CPU while a lazy user was still active, so you do -actually get cases where you have a address space that is _only_ used by -lazy users. That is often a short-lived state, because once that thread -gets scheduled away in favour of a real thread, the "zombie" mm gets -released because "mm_users" becomes zero. - -Also, a new rule is that _nobody_ ever has "init_mm" as a real MM any -more. "init_mm" should be considered just a "lazy context when no other -context is available", and in fact it is mainly used just at bootup when -no real VM has yet been created. So code that used to check - - if (current->mm == _mm) - -should generally just do - - if (!current->mm) - -instead (which makes more sense anyway - the test is basically one of "do -we have a user context", and is generally done by the page fault handler -and things like that). - -Anyway, I put a pre-patch-2.3.13-1 on ftp.kernel.org just a moment ago, -because it slightly changes the interfaces to accommodate the alpha (who -would have thought it, but the alpha actually ends up having one of the -ugliest context switch codes - unlike the other architectures where the MM -and register state is separate, the alpha PALcode joins the two, and you -need to switch both together). - -(From http://marc.info/?l=linux-kernel=93337278602211=2) +.. _active_mm: + += +Active MM += + +:: + + List: linux-kernel + Subject:Re: active_mm + From: Linus Torvalds + Date: 1999-07-30 21:36:24 + + Cc'd to linux-kernel, because I don't write explanations all that often, + and when I do I feel better about more people reading them. + + On Fri, 30 Jul 1999, David Mosberger wrote: + > + > Is there a brief
[PATCH 00/32] docs/vm: convert to ReST format
Hi,
These patches convert files in Documentation/vm to ReST format, add an
initial index and link it to the top level documentation.
There are no contents changes in the documentation, except few spelling
fixes. The relatively large diffstat stems from the indentation and
paragraph wrapping changes.
I've tried to keep the formatting as consistent as possible, but I could
miss some places that needed markup and add some markup where it was not
necessary.
Mike Rapoport (32):
docs/vm: active_mm.txt convert to ReST format
docs/vm: balance: convert to ReST format
docs/vm: cleancache.txt: convert to ReST format
docs/vm: frontswap.txt: convert to ReST format
docs/vm: highmem.txt: convert to ReST format
docs/vm: hmm.txt: convert to ReST format
docs/vm: hugetlbpage.txt: convert to ReST format
docs/vm: hugetlbfs_reserv.txt: convert to ReST format
docs/vm: hwpoison.txt: convert to ReST format
docs/vm: idle_page_tracking.txt: convert to ReST format
docs/vm: ksm.txt: convert to ReST format
docs/vm: mmu_notifier.txt: convert to ReST format
docs/vm: numa_memory_policy.txt: convert to ReST format
docs/vm: overcommit-accounting: convert to ReST format
docs/vm: page_frags convert to ReST format
docs/vm: numa: convert to ReST format
docs/vm: pagemap.txt: convert to ReST format
docs/vm: page_migration: convert to ReST format
docs/vm: page_owner: convert to ReST format
docs/vm: remap_file_pages.txt: conert to ReST format
docs/vm: slub.txt: convert to ReST format
docs/vm: soft-dirty.txt: convert to ReST format
docs/vm: split_page_table_lock: convert to ReST format
docs/vm: swap_numa.txt: convert to ReST format
docs/vm: transhuge.txt: convert to ReST format
docs/vm: unevictable-lru.txt: convert to ReST format
docs/vm: userfaultfd.txt: convert to ReST format
docs/vm: z3fold.txt: convert to ReST format
docs/vm: zsmalloc.txt: convert to ReST format
docs/vm: zswap.txt: convert to ReST format
docs/vm: rename documentation files to .rst
docs/vm: add index.rst and link MM documentation to top level index
Documentation/ABI/stable/sysfs-devices-node| 2 +-
.../ABI/testing/sysfs-kernel-mm-hugepages | 2 +-
Documentation/ABI/testing/sysfs-kernel-mm-ksm | 2 +-
Documentation/ABI/testing/sysfs-kernel-slab| 4 +-
Documentation/admin-guide/kernel-parameters.txt| 12 +-
Documentation/dev-tools/kasan.rst | 2 +-
Documentation/filesystems/proc.txt | 4 +-
Documentation/filesystems/tmpfs.txt| 2 +-
Documentation/index.rst| 3 +-
Documentation/sysctl/vm.txt| 6 +-
Documentation/vm/00-INDEX | 58 +--
Documentation/vm/active_mm.rst | 91
Documentation/vm/active_mm.txt | 83
Documentation/vm/{balance => balance.rst} | 15 +-
.../vm/{cleancache.txt => cleancache.rst} | 105 +++--
Documentation/vm/conf.py | 10 +
Documentation/vm/{frontswap.txt => frontswap.rst} | 59 ++-
Documentation/vm/{highmem.txt => highmem.rst} | 87 ++--
Documentation/vm/{hmm.txt => hmm.rst} | 66 ++-
.../{hugetlbfs_reserv.txt => hugetlbfs_reserv.rst} | 212 +
.../vm/{hugetlbpage.txt => hugetlbpage.rst}| 243 ++-
Documentation/vm/{hwpoison.txt => hwpoison.rst}| 141 +++---
...le_page_tracking.txt => idle_page_tracking.rst} | 55 ++-
Documentation/vm/index.rst | 56 +++
Documentation/vm/ksm.rst | 183
Documentation/vm/ksm.txt | 178
Documentation/vm/mmu_notifier.rst | 99 +
Documentation/vm/mmu_notifier.txt | 93
Documentation/vm/{numa => numa.rst}| 6 +-
Documentation/vm/numa_memory_policy.rst| 485 +
Documentation/vm/numa_memory_policy.txt| 452 ---
Documentation/vm/overcommit-accounting | 80
Documentation/vm/overcommit-accounting.rst | 87
Documentation/vm/{page_frags => page_frags.rst}| 5 +-
.../vm/{page_migration => page_migration.rst} | 149 ---
.../vm/{page_owner.txt => page_owner.rst} | 34 +-
Documentation/vm/{pagemap.txt => pagemap.rst} | 170
.../{remap_file_pages.txt => remap_file_pages.rst} | 6 +
Documentation/vm/slub.rst | 361 +++
Documentation/vm/slub.txt | 342 ---
.../vm/{soft-dirty.txt => soft-dirty.rst} | 20 +-
...t_page_table_lock => split_page_table_lock.rst} | 12 +-
Documentation/vm/{swap_numa.txt => swap_numa.rst} | 55 ++-
Documentation/vm/{transhuge.txt => transhuge.rst} | 286 +++-
.../{unevictable-lru.txt =>
[PATCH 04/32] docs/vm: frontswap.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/frontswap.txt | 59 ++ 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/Documentation/vm/frontswap.txt b/Documentation/vm/frontswap.txt index c71a019..1979f43 100644 --- a/Documentation/vm/frontswap.txt +++ b/Documentation/vm/frontswap.txt @@ -1,13 +1,20 @@ +.. _frontswap: + += +Frontswap += + Frontswap provides a "transcendent memory" interface for swap pages. In some environments, dramatic performance savings may be obtained because swapped pages are saved in RAM (or a RAM-like device) instead of a swap disk. -(Note, frontswap -- and cleancache (merged at 3.0) -- are the "frontends" +(Note, frontswap -- and :ref:`cleancache` (merged at 3.0) -- are the "frontends" and the only necessary changes to the core kernel for transcendent memory; all other supporting code -- the "backends" -- is implemented as drivers. -See the LWN.net article "Transcendent memory in a nutshell" for a detailed -overview of frontswap and related kernel parts: -https://lwn.net/Articles/454795/ ) +See the LWN.net article `Transcendent memory in a nutshell`_ +for a detailed overview of frontswap and related kernel parts) + +.. _Transcendent memory in a nutshell: https://lwn.net/Articles/454795/ Frontswap is so named because it can be thought of as the opposite of a "backing" store for a swap device. The storage is assumed to be @@ -50,19 +57,27 @@ or the store fails AND the page is invalidated. This ensures stale data may never be obtained from frontswap. If properly configured, monitoring of frontswap is done via debugfs in -the /sys/kernel/debug/frontswap directory. The effectiveness of +the `/sys/kernel/debug/frontswap` directory. The effectiveness of frontswap can be measured (across all swap devices) with: -failed_stores - how many store attempts have failed -loads - how many loads were attempted (all should succeed) -succ_stores- how many store attempts have succeeded -invalidates- how many invalidates were attempted +``failed_stores`` + how many store attempts have failed + +``loads`` + how many loads were attempted (all should succeed) + +``succ_stores`` + how many store attempts have succeeded + +``invalidates`` + how many invalidates were attempted A backend implementation may provide additional metrics. FAQ +=== -1) Where's the value? +* Where's the value? When a workload starts swapping, performance falls through the floor. Frontswap significantly increases performance in many such workloads by @@ -117,8 +132,8 @@ A KVM implementation is underway and has been RFC'ed to lkml. And, using frontswap, investigation is also underway on the use of NVM as a memory extension technology. -2) Sure there may be performance advantages in some situations, but - what's the space/time overhead of frontswap? +* Sure there may be performance advantages in some situations, but + what's the space/time overhead of frontswap? If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles into nothingness and the only overhead is a few extra bytes per swapon'ed @@ -148,8 +163,8 @@ pressure that can potentially outweigh the other advantages. A backend, such as zcache, must implement policies to carefully (but dynamically) manage memory limits to ensure this doesn't happen. -3) OK, how about a quick overview of what this frontswap patch does - in terms that a kernel hacker can grok? +* OK, how about a quick overview of what this frontswap patch does + in terms that a kernel hacker can grok? Let's assume that a frontswap "backend" has registered during kernel initialization; this registration indicates that this @@ -188,9 +203,9 @@ and (potentially) a swap device write are replaced by a "frontswap backend store" and (possibly) a "frontswap backend loads", which are presumably much faster. -4) Can't frontswap be configured as a "special" swap device that is - just higher priority than any real swap device (e.g. like zswap, - or maybe swap-over-nbd/NFS)? +* Can't frontswap be configured as a "special" swap device that is + just higher priority than any real swap device (e.g. like zswap, + or maybe swap-over-nbd/NFS)? No. First, the existing swap subsystem doesn't allow for any kind of swap hierarchy. Perhaps it could be rewritten to accommodate a hierarchy, @@ -240,9 +255,9 @@ installation, frontswap is useless. Swapless portable devices can still use frontswap but a backend for such devices must configure some kind of "ghost" swap device and ensure that it is never used. -5) Why this weird definition about "duplicate stores"? If a page - has been previously successfully stored, can't it always be - successfully overwritten? +* Why t
[PATCH 03/32] docs/vm: cleancache.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/cleancache.txt | 105 1 file changed, 62 insertions(+), 43 deletions(-) diff --git a/Documentation/vm/cleancache.txt b/Documentation/vm/cleancache.txt index e4b49df..68cba91 100644 --- a/Documentation/vm/cleancache.txt +++ b/Documentation/vm/cleancache.txt @@ -1,4 +1,11 @@ -MOTIVATION +.. _cleancache: + +== +Cleancache +== + +Motivation +== Cleancache is a new optional feature provided by the VFS layer that potentially dramatically increases page cache effectiveness for @@ -21,9 +28,10 @@ Transcendent memory "drivers" for cleancache are currently implemented in Xen (using hypervisor memory) and zcache (using in-kernel compressed memory) and other implementations are in development. -FAQs are included below. +:ref:`FAQs ` are included below. -IMPLEMENTATION OVERVIEW +Implementation Overview +=== A cleancache "backend" that provides transcendent memory registers itself to the kernel's cleancache "frontend" by calling cleancache_register_ops, @@ -80,22 +88,33 @@ different Linux threads are simultaneously putting and invalidating a page with the same handle, the results are indeterminate. Callers must lock the page to ensure serial behavior. -CLEANCACHE PERFORMANCE METRICS +Cleancache Performance Metrics +== If properly configured, monitoring of cleancache is done via debugfs in -the /sys/kernel/debug/cleancache directory. The effectiveness of cleancache +the `/sys/kernel/debug/cleancache` directory. The effectiveness of cleancache can be measured (across all filesystems) with: -succ_gets - number of gets that were successful -failed_gets- number of gets that failed -puts - number of puts attempted (all "succeed") -invalidates- number of invalidates attempted +``succ_gets`` + number of gets that were successful + +``failed_gets`` + number of gets that failed + +``puts`` + number of puts attempted (all "succeed") + +``invalidates`` + number of invalidates attempted A backend implementation may provide additional metrics. +.. _faq: + FAQ +=== -1) Where's the value? (Andrew Morton) +* Where's the value? (Andrew Morton) Cleancache provides a significant performance benefit to many workloads in many environments with negligible overhead by improving the @@ -137,8 +156,8 @@ device that stores pages of data in a compressed state. And the proposed "RAMster" driver shares RAM across multiple physical systems. -2) Why does cleancache have its sticky fingers so deep inside the - filesystems and VFS? (Andrew Morton and Christoph Hellwig) +* Why does cleancache have its sticky fingers so deep inside the + filesystems and VFS? (Andrew Morton and Christoph Hellwig) The core hooks for cleancache in VFS are in most cases a single line and the minimum set are placed precisely where needed to maintain @@ -168,9 +187,9 @@ filesystems in the future. The total impact of the hooks to existing fs and mm files is only about 40 lines added (not counting comments and blank lines). -3) Why not make cleancache asynchronous and batched so it can - more easily interface with real devices with DMA instead - of copying each individual page? (Minchan Kim) +* Why not make cleancache asynchronous and batched so it can more + easily interface with real devices with DMA instead of copying each + individual page? (Minchan Kim) The one-page-at-a-time copy semantics simplifies the implementation on both the frontend and backend and also allows the backend to @@ -182,8 +201,8 @@ are avoided. While the interface seems odd for a "real device" or for real kernel-addressable RAM, it makes perfect sense for transcendent memory. -4) Why is non-shared cleancache "exclusive"? And where is the - page "invalidated" after a "get"? (Minchan Kim) +* Why is non-shared cleancache "exclusive"? And where is the + page "invalidated" after a "get"? (Minchan Kim) The main reason is to free up space in transcendent memory and to avoid unnecessary cleancache_invalidate calls. If you want inclusive, @@ -193,7 +212,7 @@ be easily extended to add a "get_no_invalidate" call. The invalidate is done by the cleancache backend implementation. -5) What's the performance impact? +* What's the performance impact? Performance analysis has been presented at OLS'09 and LCA'10. Briefly, performance gains can be significant on most workloads, @@ -206,7 +225,7 @@ single-core systems with slow memory-copy speeds, cleancache has little value, but in newer multicore machines, especially consolidated/virtualized machines, it has great value. -6) How do I add cleancache support for filesystem X? (Boaz Harrash) +* How
[PATCH 07/32] docs/vm: hugetlbpage.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/hugetlbpage.txt | 243 ++- 1 file changed, 139 insertions(+), 104 deletions(-) diff --git a/Documentation/vm/hugetlbpage.txt b/Documentation/vm/hugetlbpage.txt index faf077d..3bb0d99 100644 --- a/Documentation/vm/hugetlbpage.txt +++ b/Documentation/vm/hugetlbpage.txt @@ -1,3 +1,11 @@ +.. _hugetlbpage: + += +HugeTLB Pages += + +Overview + The intent of this file is to give a brief summary of hugetlbpage support in the Linux kernel. This support is built on top of multiple page size support @@ -18,53 +26,59 @@ First the Linux kernel needs to be built with the CONFIG_HUGETLBFS automatically when CONFIG_HUGETLBFS is selected) configuration options. -The /proc/meminfo file provides information about the total number of +The ``/proc/meminfo`` file provides information about the total number of persistent hugetlb pages in the kernel's huge page pool. It also displays default huge page size and information about the number of free, reserved and surplus huge pages in the pool of huge pages of default size. The huge page size is needed for generating the proper alignment and size of the arguments to system calls that map huge page regions. -The output of "cat /proc/meminfo" will include lines like: +The output of ``cat /proc/meminfo`` will include lines like:: -. -HugePages_Total: uuu -HugePages_Free: vvv -HugePages_Rsvd: www -HugePages_Surp: xxx -Hugepagesize:yyy kB -Hugetlb: zzz kB + HugePages_Total: uuu + HugePages_Free: vvv + HugePages_Rsvd: www + HugePages_Surp: xxx + Hugepagesize:yyy kB + Hugetlb: zzz kB where: -HugePages_Total is the size of the pool of huge pages. -HugePages_Free is the number of huge pages in the pool that are not yet -allocated. -HugePages_Rsvd is short for "reserved," and is the number of huge pages for -which a commitment to allocate from the pool has been made, -but no allocation has yet been made. Reserved huge pages -guarantee that an application will be able to allocate a -huge page from the pool of huge pages at fault time. -HugePages_Surp is short for "surplus," and is the number of huge pages in -the pool above the value in /proc/sys/vm/nr_hugepages. The -maximum number of surplus huge pages is controlled by -/proc/sys/vm/nr_overcommit_hugepages. -Hugepagesizeis the default hugepage size (in Kb). -Hugetlb is the total amount of memory (in kB), consumed by huge -pages of all sizes. -If huge pages of different sizes are in use, this number -will exceed HugePages_Total * Hugepagesize. To get more -detailed information, please, refer to -/sys/kernel/mm/hugepages (described below). - - -/proc/filesystems should also show a filesystem of type "hugetlbfs" configured -in the kernel. - -/proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge + +HugePages_Total + is the size of the pool of huge pages. +HugePages_Free + is the number of huge pages in the pool that are not yet +allocated. +HugePages_Rsvd + is short for "reserved," and is the number of huge pages for +which a commitment to allocate from the pool has been made, +but no allocation has yet been made. Reserved huge pages +guarantee that an application will be able to allocate a +huge page from the pool of huge pages at fault time. +HugePages_Surp + is short for "surplus," and is the number of huge pages in +the pool above the value in ``/proc/sys/vm/nr_hugepages``. The +maximum number of surplus huge pages is controlled by +``/proc/sys/vm/nr_overcommit_hugepages``. +Hugepagesize + is the default hugepage size (in Kb). +Hugetlb +is the total amount of memory (in kB), consumed by huge +pages of all sizes. +If huge pages of different sizes are in use, this number +will exceed HugePages_Total \* Hugepagesize. To get more +detailed information, please, refer to +``/sys/kernel/mm/hugepages`` (described below). + + +``/proc/filesystems`` should also show a filesystem of type "hugetlbfs" +configured in the kernel. + +``/proc/sys/vm/nr_hugepages`` indicates the current number of "persistent" huge pages in the kernel's huge page pool. "Persistent" huge pages will be returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages -by increasing or decreasing the value of 'nr_hugepages'. +by increasing or decreasing the value of ``nr_huge
[PATCH 08/32] docs/vm: hugetlbfs_reserv.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/vm/hugetlbfs_reserv.txt | 212 ++
1 file changed, 135 insertions(+), 77 deletions(-)
diff --git a/Documentation/vm/hugetlbfs_reserv.txt
b/Documentation/vm/hugetlbfs_reserv.txt
index 9aca09a..36a87a2 100644
--- a/Documentation/vm/hugetlbfs_reserv.txt
+++ b/Documentation/vm/hugetlbfs_reserv.txt
@@ -1,6 +1,13 @@
-Hugetlbfs Reservation Overview
---
-Huge pages as described at 'Documentation/vm/hugetlbpage.txt' are typically
+.. _hugetlbfs_reserve:
+
+=
+Hugetlbfs Reservation
+=
+
+Overview
+
+
+Huge pages as described at :ref:`hugetlbpage` are typically
preallocated for application use. These huge pages are instantiated in a
task's address space at page fault time if the VMA indicates huge pages are
to be used. If no huge page exists at page fault time, the task is sent
@@ -17,47 +24,55 @@ describe how huge page reserve processing is done in the
v4.10 kernel.
Audience
-
+
This description is primarily targeted at kernel developers who are modifying
hugetlbfs code.
The Data Structures
+===
+
resv_huge_pages
This is a global (per-hstate) count of reserved huge pages. Reserved
huge pages are only available to the task which reserved them.
Therefore, the number of huge pages generally available is computed
- as (free_huge_pages - resv_huge_pages).
+ as (``free_huge_pages - resv_huge_pages``).
Reserve Map
- A reserve map is described by the structure:
- struct resv_map {
- struct kref refs;
- spinlock_t lock;
- struct list_head regions;
- long adds_in_progress;
- struct list_head region_cache;
- long region_cache_count;
- };
+ A reserve map is described by the structure::
+
+ struct resv_map {
+ struct kref refs;
+ spinlock_t lock;
+ struct list_head regions;
+ long adds_in_progress;
+ struct list_head region_cache;
+ long region_cache_count;
+ };
+
There is one reserve map for each huge page mapping in the system.
The regions list within the resv_map describes the regions within
- the mapping. A region is described as:
- struct file_region {
- struct list_head link;
- long from;
- long to;
- };
+ the mapping. A region is described as::
+
+ struct file_region {
+ struct list_head link;
+ long from;
+ long to;
+ };
+
The 'from' and 'to' fields of the file region structure are huge page
indices into the mapping. Depending on the type of mapping, a
region in the reserv_map may indicate reservations exist for the
range, or reservations do not exist.
Flags for MAP_PRIVATE Reservations
These are stored in the bottom bits of the reservation map pointer.
- #define HPAGE_RESV_OWNER(1UL << 0) Indicates this task is the
- owner of the reservations associated with the mapping.
- #define HPAGE_RESV_UNMAPPED (1UL << 1) Indicates task originally
- mapping this range (and creating reserves) has unmapped a
- page from this task (the child) due to a failed COW.
+
+ ``#define HPAGE_RESV_OWNER(1UL << 0)``
+ Indicates this task is the owner of the reservations
+ associated with the mapping.
+ ``#define HPAGE_RESV_UNMAPPED (1UL << 1)``
+ Indicates task originally mapping this range (and creating
+ reserves) has unmapped a page from this task (the child)
+ due to a failed COW.
Page Flags
The PagePrivate page flag is used to indicate that a huge page
reservation must be restored when the huge page is freed. More
@@ -65,12 +80,14 @@ Page Flags
Reservation Map Location (Private or Shared)
-
+
+
A huge page mapping or segment is either private or shared. If private,
it is typically only available to a single address space (task). If shared,
it can be mapped into multiple address spaces (tasks). The location and
semantics of the reservation map is significantly different for two types
of mappings. Location differences are:
+
- For private mappings, the reservation map hangs off the the VMA structure.
Specifically, vma->vm_private_data. This reserve map is created at the
time the mapping (mmap(MAP_PRIVATE)) is created.
@@ -82,15 +99,15 @@ of mappings. Location difference
[PATCH 09/32] docs/vm: hwpoison.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/hwpoison.txt | 141 +- 1 file changed, 70 insertions(+), 71 deletions(-) diff --git a/Documentation/vm/hwpoison.txt b/Documentation/vm/hwpoison.txt index e912d7e..b1a8c24 100644 --- a/Documentation/vm/hwpoison.txt +++ b/Documentation/vm/hwpoison.txt @@ -1,7 +1,14 @@ +.. hwpoison: + + +hwpoison + + What is hwpoison? += Upcoming Intel CPUs have support for recovering from some memory errors -(``MCA recovery''). This requires the OS to declare a page "poisoned", +(``MCA recovery``). This requires the OS to declare a page "poisoned", kill the processes associated with it and avoid using it in the future. This patchkit implements the necessary infrastructure in the VM. @@ -46,9 +53,10 @@ address. This in theory allows other applications to handle memory failures too. The expection is that near all applications won't do that, but some very specialized ones might. +Failure recovery modes +== -There are two (actually three) modi memory failure recovery can be in: +There are two (actually three) modes memory failure recovery can be in: vm.memory_failure_recovery sysctl set to zero: All memory failures cause a panic. Do not attempt recovery. @@ -67,9 +75,8 @@ late kill This is best for memory error unaware applications and default Note some pages are always handled as late kill. - -User control: +User control + vm.memory_failure_recovery See sysctl.txt @@ -79,11 +86,19 @@ vm.memory_failure_early_kill PR_MCE_KILL Set early/late kill mode/revert to system default - arg1: PR_MCE_KILL_CLEAR: Revert to system default - arg1: PR_MCE_KILL_SET: arg2 defines thread specific mode - PR_MCE_KILL_EARLY: Early kill - PR_MCE_KILL_LATE: Late kill - PR_MCE_KILL_DEFAULT: Use system global default + + arg1: PR_MCE_KILL_CLEAR: + Revert to system default + arg1: PR_MCE_KILL_SET: + arg2 defines thread specific mode + + PR_MCE_KILL_EARLY: + Early kill + PR_MCE_KILL_LATE: + Late kill + PR_MCE_KILL_DEFAULT + Use system global default + Note that if you want to have a dedicated thread which handles the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise, @@ -92,77 +107,64 @@ PR_MCE_KILL PR_MCE_KILL_GET return current mode +Testing +=== - -Testing: - -madvise(MADV_HWPOISON, ) - (as root) - Poison a page in the process for testing - +* madvise(MADV_HWPOISON, ) (as root) - Poison a page in the + process for testing -hwpoison-inject module through debugfs +* hwpoison-inject module through debugfs ``/sys/kernel/debug/hwpoison/`` -/sys/kernel/debug/hwpoison/ + corrupt-pfn + Inject hwpoison fault at PFN echoed into this file. This does + some early filtering to avoid corrupted unintended pages in test suites. -corrupt-pfn + unpoison-pfn + Software-unpoison page at PFN echoed into this file. This way + a page can be reused again. This only works for Linux + injected failures, not for real memory failures. -Inject hwpoison fault at PFN echoed into this file. This does -some early filtering to avoid corrupted unintended pages in test suites. + Note these injection interfaces are not stable and might change between + kernel versions -unpoison-pfn + corrupt-filter-dev-major, corrupt-filter-dev-minor + Only handle memory failures to pages associated with the file + system defined by block device major/minor. -1U is the + wildcard value. This should be only used for testing with + artificial injection. -Software-unpoison page at PFN echoed into this file. This -way a page can be reused again. -This only works for Linux injected failures, not for real -memory failures. + corrupt-filter-memcg + Limit injection to pages owned by memgroup. Specified by inode + number of the memcg. -Note these injection interfaces are not stable and might change between -kernel versions + Example:: -corrupt-filter-dev-major -corrupt-filter-dev-minor + mkdir /sys/fs/cgroup/mem/hwpoison -Only handle memory failures to pages associated with the file system defined -by block device major/minor. -1U is the wildcard value. -This should be only used for testing with artificial injection. + usemem -m 100 -s 1000 & + echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks -corrupt-filter-memcg + memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ') + echo $memcg_ino > /debug/hwpoison/cor
[PATCH 12/32] docs/vm: mmu_notifier.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/vm/mmu_notifier.txt | 108 --
1 file changed, 57 insertions(+), 51 deletions(-)
diff --git a/Documentation/vm/mmu_notifier.txt
b/Documentation/vm/mmu_notifier.txt
index 23b4625..47baa1c 100644
--- a/Documentation/vm/mmu_notifier.txt
+++ b/Documentation/vm/mmu_notifier.txt
@@ -1,7 +1,10 @@
+.. _mmu_notifier:
+
When do you need to notify inside page table lock ?
+===
When clearing a pte/pmd we are given a choice to notify the event through
-(notify version of *_clear_flush call mmu_notifier_invalidate_range) under
+(notify version of \*_clear_flush call mmu_notifier_invalidate_range) under
the page table lock. But that notification is not necessary in all cases.
For secondary TLB (non CPU TLB) like IOMMU TLB or device TLB (when device use
@@ -18,6 +21,7 @@ a page that might now be used by some completely different
task.
Case B is more subtle. For correctness it requires the following sequence to
happen:
+
- take page table lock
- clear page table entry and notify ([pmd/pte]p_huge_clear_flush_notify())
- set page table entry to point to new page
@@ -28,58 +32,60 @@ the device.
Consider the following scenario (device use a feature similar to ATS/PASID):
-Two address addrA and addrB such that |addrA - addrB| >= PAGE_SIZE we assume
+Two address addrA and addrB such that \|addrA - addrB\| >= PAGE_SIZE we assume
they are write protected for COW (other case of B apply too).
-[Time N]
-CPU-thread-0 {try to write to addrA}
-CPU-thread-1 {try to write to addrB}
-CPU-thread-2 {}
-CPU-thread-3 {}
-DEV-thread-0 {read addrA and populate device TLB}
-DEV-thread-2 {read addrB and populate device TLB}
-[Time N+1] --
-CPU-thread-0 {COW_step0: {mmu_notifier_invalidate_range_start(addrA)}}
-CPU-thread-1 {COW_step0: {mmu_notifier_invalidate_range_start(addrB)}}
-CPU-thread-2 {}
-CPU-thread-3 {}
-DEV-thread-0 {}
-DEV-thread-2 {}
-[Time N+2] --
-CPU-thread-0 {COW_step1: {update page table to point to new page for addrA}}
-CPU-thread-1 {COW_step1: {update page table to point to new page for addrB}}
-CPU-thread-2 {}
-CPU-thread-3 {}
-DEV-thread-0 {}
-DEV-thread-2 {}
-[Time N+3] --
-CPU-thread-0 {preempted}
-CPU-thread-1 {preempted}
-CPU-thread-2 {write to addrA which is a write to new page}
-CPU-thread-3 {}
-DEV-thread-0 {}
-DEV-thread-2 {}
-[Time N+3] --
-CPU-thread-0 {preempted}
-CPU-thread-1 {preempted}
-CPU-thread-2 {}
-CPU-thread-3 {write to addrB which is a write to new page}
-DEV-thread-0 {}
-DEV-thread-2 {}
-[Time N+4] --
-CPU-thread-0 {preempted}
-CPU-thread-1 {COW_step3: {mmu_notifier_invalidate_range_end(addrB)}}
-CPU-thread-2 {}
-CPU-thread-3 {}
-DEV-thread-0 {}
-DEV-thread-2 {}
-[Time N+5] --
-CPU-thread-0 {preempted}
-CPU-thread-1 {}
-CPU-thread-2 {}
-CPU-thread-3 {}
-DEV-thread-0 {read addrA from old page}
-DEV-thread-2 {read addrB from new page}
+::
+
+ [Time N]
+ CPU-thread-0 {try to write to addrA}
+ CPU-thread-1 {try to write to addrB}
+ CPU-thread-2 {}
+ CPU-thread-3 {}
+ DEV-thread-0 {read addrA and populate device TLB}
+ DEV-thread-2 {read addrB and populate device TLB}
+ [Time N+1] --
+ CPU-thread-0 {COW_step0: {mmu_notifier_invalidate_range_start(addrA)}}
+ CPU-thread-1 {COW_step0: {mmu_notifier_invalidate_range_start(addrB)}}
+ CPU-thread-2 {}
+ CPU-thread-3 {}
+ DEV-thread-0 {}
+ DEV-thread-2 {}
+ [Time N+2] --
+ CPU-thread-0 {COW_step1: {update page table to point to new page for addrA}}
+ CPU-thread-1 {COW_step1: {update page table to point to new page for addrB}}
+ CPU-thread-2 {}
+ CPU-thread-3 {}
+ DEV-thread-0 {}
+ DEV-thread-2 {}
+ [Time N+3] --
+ CPU-thread-0 {preempted}
+ CPU-thread-1 {preempted}
+ CPU-thread-2 {write to addrA which is a write to new page}
+ CPU-thread-3 {}
+ DEV-thread-0 {}
+ DEV-thread-2 {}
+ [Time N+3] --
+ CPU-thread-0 {preempted}
+ CPU-thread-1 {preempted}
+ CPU-thread-2 {}
+ CPU-thread-3 {write to addrB which is a write to new page}
+ DEV-thread-0 {}
+ DEV-thread-2 {}
+ [Time N+4]
[PATCH 10/32] docs/vm: idle_page_tracking.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/idle_page_tracking.txt | 55 + 1 file changed, 36 insertions(+), 19 deletions(-) diff --git a/Documentation/vm/idle_page_tracking.txt b/Documentation/vm/idle_page_tracking.txt index 85dcc3b..9cbe6f8 100644 --- a/Documentation/vm/idle_page_tracking.txt +++ b/Documentation/vm/idle_page_tracking.txt @@ -1,4 +1,11 @@ -MOTIVATION +.. _idle_page_tracking: + +== +Idle Page Tracking +== + +Motivation +== The idle page tracking feature allows to track which memory pages are being accessed by a workload and which are idle. This information can be useful for @@ -8,10 +15,14 @@ or deciding where to place the workload within a compute cluster. It is enabled by CONFIG_IDLE_PAGE_TRACKING=y. -USER API +.. _user_api: -The idle page tracking API is located at /sys/kernel/mm/page_idle. Currently, -it consists of the only read-write file, /sys/kernel/mm/page_idle/bitmap. +User API + + +The idle page tracking API is located at ``/sys/kernel/mm/page_idle``. +Currently, it consists of the only read-write file, +``/sys/kernel/mm/page_idle/bitmap``. The file implements a bitmap where each bit corresponds to a memory page. The bitmap is represented by an array of 8-byte integers, and the page at PFN #i is @@ -19,8 +30,9 @@ mapped to bit #i%64 of array element #i/64, byte order is native. When a bit is set, the corresponding page is idle. A page is considered idle if it has not been accessed since it was marked idle -(for more details on what "accessed" actually means see the IMPLEMENTATION -DETAILS section). To mark a page idle one has to set the bit corresponding to +(for more details on what "accessed" actually means see the :ref:`Implementation +Details ` section). +To mark a page idle one has to set the bit corresponding to the page by writing to the file. A value written to the file is OR-ed with the current bitmap value. @@ -30,9 +42,9 @@ page types (e.g. SLAB pages) an attempt to mark a page idle is silently ignored, and hence such pages are never reported idle. For huge pages the idle flag is set only on the head page, so one has to read -/proc/kpageflags in order to correctly count idle huge pages. +``/proc/kpageflags`` in order to correctly count idle huge pages. -Reading from or writing to /sys/kernel/mm/page_idle/bitmap will return +Reading from or writing to ``/sys/kernel/mm/page_idle/bitmap`` will return -EINVAL if you are not starting the read/write on an 8-byte boundary, or if the size of the read/write is not a multiple of 8 bytes. Writing to this file beyond max PFN will return -ENXIO. @@ -41,21 +53,25 @@ That said, in order to estimate the amount of pages that are not used by a workload one should: 1. Mark all the workload's pages as idle by setting corresponding bits in -/sys/kernel/mm/page_idle/bitmap. The pages can be found by reading -/proc/pid/pagemap if the workload is represented by a process, or by -filtering out alien pages using /proc/kpagecgroup in case the workload is -placed in a memory cgroup. +``/sys/kernel/mm/page_idle/bitmap``. The pages can be found by reading +``/proc/pid/pagemap`` if the workload is represented by a process, or by +filtering out alien pages using ``/proc/kpagecgroup`` in case the workload +is placed in a memory cgroup. 2. Wait until the workload accesses its working set. - 3. Read /sys/kernel/mm/page_idle/bitmap and count the number of bits set. If -one wants to ignore certain types of pages, e.g. mlocked pages since they -are not reclaimable, he or she can filter them out using /proc/kpageflags. + 3. Read ``/sys/kernel/mm/page_idle/bitmap`` and count the number of bits set. +If one wants to ignore certain types of pages, e.g. mlocked pages since they +are not reclaimable, he or she can filter them out using +``/proc/kpageflags``. + +See Documentation/vm/pagemap.txt for more information about +``/proc/pid/pagemap``, ``/proc/kpageflags``, and ``/proc/kpagecgroup``. -See Documentation/vm/pagemap.txt for more information about /proc/pid/pagemap, -/proc/kpageflags, and /proc/kpagecgroup. +.. _impl_details: -IMPLEMENTATION DETAILS +Implementation Details +== The kernel internally keeps track of accesses to user memory pages in order to reclaim unreferenced pages first on memory shortage conditions. A page is @@ -77,7 +93,8 @@ When a dirty page is written to swap or disk as a result of memory reclaim or exceeding the dirty memory limit, it is not marked referenced. The idle memory tracking feature adds a new page flag, the Idle flag. This flag -is set manually, by writing to /sys/kernel/mm/page_idle/bitmap (see the USER API +is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` (see the +:ref:`User API ` section), and cleared automatically whenever a pag
[PATCH 13/32] docs/vm: numa_memory_policy.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/numa_memory_policy.txt | 533 +--- 1 file changed, 283 insertions(+), 250 deletions(-) diff --git a/Documentation/vm/numa_memory_policy.txt b/Documentation/vm/numa_memory_policy.txt index 622b927..8cd942c 100644 --- a/Documentation/vm/numa_memory_policy.txt +++ b/Documentation/vm/numa_memory_policy.txt @@ -1,5 +1,11 @@ +.. _numa_memory_policy: + +=== +Linux Memory Policy +=== What is Linux Memory Policy? + In the Linux kernel, "memory policy" determines from which node the kernel will allocate memory in a NUMA system or in an emulated NUMA system. Linux has @@ -9,35 +15,36 @@ document attempts to describe the concepts and APIs of the 2.6 memory policy support. Memory policies should not be confused with cpusets -(Documentation/cgroup-v1/cpusets.txt) +(``Documentation/cgroup-v1/cpusets.txt``) which is an administrative mechanism for restricting the nodes from which memory may be allocated by a set of processes. Memory policies are a programming interface that a NUMA-aware application can take advantage of. When both cpusets and policies are applied to a task, the restrictions of the cpuset -takes priority. See "MEMORY POLICIES AND CPUSETS" below for more details. +takes priority. See :ref:`Memory Policies and cpusets ` +below for more details. -MEMORY POLICY CONCEPTS +Memory Policy Concepts +== Scope of Memory Policies + The Linux kernel supports _scopes_ of memory policy, described here from most general to most specific: -System Default Policy: this policy is "hard coded" into the kernel. It -is the policy that governs all page allocations that aren't controlled -by one of the more specific policy scopes discussed below. When the -system is "up and running", the system default policy will use "local -allocation" described below. However, during boot up, the system -default policy will be set to interleave allocations across all nodes -with "sufficient" memory, so as not to overload the initial boot node -with boot-time allocations. - -Task/Process Policy: this is an optional, per-task policy. When defined -for a specific task, this policy controls all page allocations made by or -on behalf of the task that aren't controlled by a more specific scope. -If a task does not define a task policy, then all page allocations that -would have been controlled by the task policy "fall back" to the System -Default Policy. +System Default Policy + this policy is "hard coded" into the kernel. It is the policy + that governs all page allocations that aren't controlled by + one of the more specific policy scopes discussed below. When + the system is "up and running", the system default policy will + use "local allocation" described below. However, during boot + up, the system default policy will be set to interleave + allocations across all nodes with "sufficient" memory, so as + not to overload the initial boot node with boot-time + allocations. + +Task/Process Policy + this is an optional, per-task policy. When defined for a specific task, this policy controls all page allocations made by or on behalf of the task that aren't controlled by a more specific scope. If a task does not define a task policy, then all page allocations that would have been controlled by the task policy "fall back" to the System Default Policy. The task policy applies to the entire address space of a task. Thus, it is inheritable, and indeed is inherited, across both fork() @@ -58,56 +65,66 @@ most general to most specific: changes its task policy remain where they were allocated based on the policy at the time they were allocated. -VMA Policy: A "VMA" or "Virtual Memory Area" refers to a range of a task's -virtual address space. A task may define a specific policy for a range -of its virtual address space. See the MEMORY POLICIES APIS section, -below, for an overview of the mbind() system call used to set a VMA -policy. - -A VMA policy will govern the allocation of pages that back this region of -the address space. Any regions of the task's address space that don't -have an explicit VMA policy will fall back to the task policy, which may -itself fall back to the System Default Policy. - -VMA policies have a few complicating details: - - VMA policy applies ONLY to anonymous pages. These include pages - allocated for anonymous segments, such as the task stack and heap, and - any regions of the address space mmap()ed with the MAP_ANON
[PATCH 17/32] docs/vm: pagemap.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/pagemap.txt | 164 +++ 1 file changed, 89 insertions(+), 75 deletions(-) diff --git a/Documentation/vm/pagemap.txt b/Documentation/vm/pagemap.txt index eafcefa..bd6d717 100644 --- a/Documentation/vm/pagemap.txt +++ b/Documentation/vm/pagemap.txt @@ -1,13 +1,16 @@ -pagemap, from the userspace perspective +.. _pagemap: + +== +pagemap from the Userspace Perspective +== pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow userspace programs to examine the page tables and related information by -reading files in /proc. +reading files in ``/proc``. There are four components to pagemap: - * /proc/pid/pagemap. This file lets a userspace process find out which + * ``/proc/pid/pagemap``. This file lets a userspace process find out which physical frame each virtual page is mapped to. It contains one 64-bit value for each virtual page, containing the following data (from fs/proc/task_mmu.c, above pagemap_read): @@ -37,24 +40,24 @@ There are four components to pagemap: determine which areas of memory are actually mapped and llseek to skip over unmapped regions. - * /proc/kpagecount. This file contains a 64-bit count of the number of + * ``/proc/kpagecount``. This file contains a 64-bit count of the number of times each page is mapped, indexed by PFN. - * /proc/kpageflags. This file contains a 64-bit set of flags for each + * ``/proc/kpageflags``. This file contains a 64-bit set of flags for each page, indexed by PFN. - The flags are (from fs/proc/page.c, above kpageflags_read): - - 0. LOCKED - 1. ERROR - 2. REFERENCED - 3. UPTODATE - 4. DIRTY - 5. LRU - 6. ACTIVE - 7. SLAB - 8. WRITEBACK - 9. RECLAIM + The flags are (from ``fs/proc/page.c``, above kpageflags_read): + +0. LOCKED +1. ERROR +2. REFERENCED +3. UPTODATE +4. DIRTY +5. LRU +6. ACTIVE +7. SLAB +8. WRITEBACK +9. RECLAIM 10. BUDDY 11. MMAP 12. ANON @@ -72,98 +75,108 @@ There are four components to pagemap: 24. ZERO_PAGE 25. IDLE - * /proc/kpagecgroup. This file contains a 64-bit inode number of the + * ``/proc/kpagecgroup``. This file contains a 64-bit inode number of the memory cgroup each page is charged to, indexed by PFN. Only available when CONFIG_MEMCG is set. Short descriptions to the page flags: - - 0. LOCKED -page is being locked for exclusive access, eg. by undergoing read/write IO - - 7. SLAB -page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator -When compound page is used, SLUB/SLQB will only set this flag on the head -page; SLOB will not flag it at all. - -10. BUDDY += + +0 - LOCKED + page is being locked for exclusive access, eg. by undergoing read/write IO +7 - SLAB + page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator + When compound page is used, SLUB/SLQB will only set this flag on the head + page; SLOB will not flag it at all. +10 - BUDDY a free memory block managed by the buddy system allocator The buddy system organizes free memory in blocks of various orders. An order N block has 2^N physically contiguous pages, with the BUDDY flag set for and _only_ for the first page. - -15. COMPOUND_HEAD -16. COMPOUND_TAIL +15 - COMPOUND_HEAD A compound page with order N consists of 2^N physically contiguous pages. A compound page with order 2 takes the form of "HTTT", where H donates its head page and T donates its tail page(s). The major consumers of compound pages are hugeTLB pages (Documentation/vm/hugetlbpage.txt), the SLUB etc. memory allocators and various device drivers. However in this interface, only huge/giga pages are made visible to end users. -17. HUGE +16 - COMPOUND_TAIL +A compound page tail (see description above). +17 - HUGE this is an integral part of a HugeTLB page - -19. HWPOISON +19 - HWPOISON hardware detected memory corruption on this page: don't touch the data! - -20. NOPAGE +20 - NOPAGE no page frame exists at the requested address - -21. KSM +21 - KSM identical memory pages dynamically shared between one or more processes - -22. THP +22 - THP contiguous pages which construct transparent hugepages - -23. BALLOON +23 - BALLOON balloon compaction page - -24. ZERO_PAGE +24 - ZERO_PAGE zero page for pfn_zero or huge_zero page - -25. IDLE +25 - IDLE page has not been accessed since it was marked idle (see Documentation/vm/idle_page_tracking.txt). Note that this flag may be stale in case the page was accessed via a PTE. To make sure the flag -is up-to-date one has to read /sys/kernel/mm/page_idle/bitmap f
[PATCH 16/32] docs/vm: numa: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/numa | 4 1 file changed, 4 insertions(+) diff --git a/Documentation/vm/numa b/Documentation/vm/numa index a31b85b..c81e7c5 100644 --- a/Documentation/vm/numa +++ b/Documentation/vm/numa @@ -1,6 +1,10 @@ +.. _numa: + Started Nov 1999 by Kanoj Sarcar <ka...@sgi.com> += What is NUMA? += This question can be answered from a couple of perspectives: the hardware view and the Linux software view. -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 15/32] docs/vm: page_frags convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/page_frags | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/vm/page_frags b/Documentation/vm/page_frags index a671456..637cc49 100644 --- a/Documentation/vm/page_frags +++ b/Documentation/vm/page_frags @@ -1,5 +1,8 @@ +.. _page_frags: + +== Page fragments --- +== A page fragment is an arbitrary-length arbitrary-offset area of memory which resides within a 0 or higher order compound page. Multiple -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 18/32] docs/vm: page_migration: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/page_migration | 149 +--- 1 file changed, 77 insertions(+), 72 deletions(-) diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration index 0478ae2..07b67a8 100644 --- a/Documentation/vm/page_migration +++ b/Documentation/vm/page_migration @@ -1,5 +1,8 @@ +.. _page_migration: + +== Page migration --- +== Page migration allows the moving of the physical location of pages between nodes in a numa system while the process is running. This means that the @@ -20,7 +23,7 @@ Page migration functions are provided by the numactl package by Andi Kleen (a version later than 0.9.3 is required. Get it from ftp://oss.sgi.com/www/projects/libnuma/download/). numactl provides libnuma which provides an interface similar to other numa functionality for page -migration. cat /proc//numa_maps allows an easy review of where the +migration. cat ``/proc//numa_maps`` allows an easy review of where the pages of a process are located. See also the numa_maps documentation in the proc(5) man page. @@ -56,8 +59,8 @@ description for those trying to use migrate_pages() from the kernel (for userspace usage see the Andi Kleen's numactl package mentioned above) and then a low level description of how the low level details work. -A. In kernel use of migrate_pages() +In kernel use of migrate_pages() + 1. Remove pages from the LRU. @@ -78,8 +81,8 @@ A. In kernel use of migrate_pages() the new page for each page that is considered for moving. -B. How migrate_pages() works - +How migrate_pages() works += migrate_pages() does several passes over its list of pages. A page is moved if all references to a page are removable at the time. The page has @@ -142,8 +145,8 @@ Steps: 20. The new page is moved to the LRU and can be scanned by the swapper etc again. -C. Non-LRU page migration -- +Non-LRU page migration +== Although original migration aimed for reducing the latency of memory access for NUMA, compaction who want to create high-order page is also main customer. @@ -164,89 +167,91 @@ migration path. If a driver want to make own pages movable, it should define three functions which are function pointers of struct address_space_operations. -1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); +1. ``bool (*isolate_page) (struct page *page, isolate_mode_t mode);`` -What VM expects on isolate_page function of driver is to return *true* -if driver isolates page successfully. On returing true, VM marks the page -as PG_isolated so concurrent isolation in several CPUs skip the page -for isolation. If a driver cannot isolate the page, it should return *false*. + What VM expects on isolate_page function of driver is to return *true* + if driver isolates page successfully. On returing true, VM marks the page + as PG_isolated so concurrent isolation in several CPUs skip the page + for isolation. If a driver cannot isolate the page, it should return *false*. -Once page is successfully isolated, VM uses page.lru fields so driver -shouldn't expect to preserve values in that fields. + Once page is successfully isolated, VM uses page.lru fields so driver + shouldn't expect to preserve values in that fields. -2. int (*migratepage) (struct address_space *mapping, - struct page *newpage, struct page *oldpage, enum migrate_mode); +2. ``int (*migratepage) (struct address_space *mapping,`` +| ``struct page *newpage, struct page *oldpage, enum migrate_mode);`` -After isolation, VM calls migratepage of driver with isolated page. -The function of migratepage is to move content of the old page to new page -and set up fields of struct page newpage. Keep in mind that you should -indicate to the VM the oldpage is no longer movable via __ClearPageMovable() -under page_lock if you migrated the oldpage successfully and returns -MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver -can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time -because VM interprets -EAGAIN as "temporal migration failure". On returning -any error except -EAGAIN, VM will give up the page migration without retrying -in this time. + After isolation, VM calls migratepage of driver with isolated page. + The function of migratepage is to move content of the old page to new page + and set up fields of struct page newpage. Keep in mind that you should + indicate to the VM the oldpage is no longer movable via __ClearPageMovable() + under page_lock if you migrated the oldpage successfully and returns + MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver + can return -EAGAIN
[PATCH 21/32] docs/vm: slub.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/slub.txt | 357 -- 1 file changed, 188 insertions(+), 169 deletions(-) diff --git a/Documentation/vm/slub.txt b/Documentation/vm/slub.txt index 8465241..3a775fd 100644 --- a/Documentation/vm/slub.txt +++ b/Documentation/vm/slub.txt @@ -1,5 +1,8 @@ +.. _slub: + +== Short users guide for SLUB --- +== The basic philosophy of SLUB is very different from SLAB. SLAB requires rebuilding the kernel to activate debug options for all @@ -8,18 +11,19 @@ SLUB can enable debugging only for selected slabs in order to avoid an impact on overall system performance which may make a bug more difficult to find. -In order to switch debugging on one can add an option "slub_debug" +In order to switch debugging on one can add an option ``slub_debug`` to the kernel command line. That will enable full debugging for all slabs. -Typically one would then use the "slabinfo" command to get statistical -data and perform operation on the slabs. By default slabinfo only lists +Typically one would then use the ``slabinfo`` command to get statistical +data and perform operation on the slabs. By default ``slabinfo`` only lists slabs that have data in them. See "slabinfo -h" for more options when -running the command. slabinfo can be compiled with +running the command. ``slabinfo`` can be compiled with +:: -gcc -o slabinfo tools/vm/slabinfo.c + gcc -o slabinfo tools/vm/slabinfo.c -Some of the modes of operation of slabinfo require that slub debugging +Some of the modes of operation of ``slabinfo`` require that slub debugging be enabled on the command line. F.e. no tracking information will be available without debugging on and validation can only partially be performed if debugging was not switched on. @@ -27,14 +31,17 @@ be performed if debugging was not switched on. Some more sophisticated uses of slub_debug: --- -Parameters may be given to slub_debug. If none is specified then full +Parameters may be given to ``slub_debug``. If none is specified then full debugging is enabled. Format: -slub_debug= Enable options for all slabs +slub_debug= + Enable options for all slabs slub_debug=, - Enable options only for select slabs + Enable options only for select slabs + + +Possible debug options are:: -Possible debug options are F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS Sorry SLAB legacy issues) Z Red zoning @@ -47,18 +54,18 @@ Possible debug options are - Switch all debugging off (useful if the kernel is configured with CONFIG_SLUB_DEBUG_ON) -F.e. in order to boot just with sanity checks and red zoning one would specify: +F.e. in order to boot just with sanity checks and red zoning one would specify:: slub_debug=FZ -Trying to find an issue in the dentry cache? Try +Trying to find an issue in the dentry cache? Try:: slub_debug=,dentry to only enable debugging on the dentry cache. Red zoning and tracking may realign the slab. We can just apply sanity checks -to the dentry cache with +to the dentry cache with:: slub_debug=F,dentry @@ -66,15 +73,15 @@ Debugging options may require the minimum possible slab order to increase as a result of storing the metadata (for example, caches with PAGE_SIZE object sizes). This has a higher liklihood of resulting in slab allocation errors in low memory situations or if there's high fragmentation of memory. To -switch off debugging for such caches by default, use +switch off debugging for such caches by default, use:: slub_debug=O In case you forgot to enable debugging on the kernel command line: It is possible to enable debugging manually when the kernel is up. Look at the -contents of: +contents of:: -/sys/kernel/slab// + /sys/kernel/slab// Look at the writable files. Writing 1 to them will enable the corresponding debug option. All options can be set on a slab that does @@ -86,98 +93,103 @@ Careful with tracing: It may spew out lots of information and never stop if used on the wrong slab. Slab merging - + If no debug options are specified then SLUB may merge similar slabs together in order to reduce overhead and increase cache hotness of objects. -slabinfo -a displays which slabs were merged together. +``slabinfo -a`` displays which slabs were merged together. Slab validation +=== SLUB can validate all object if the kernel was booted with slub_debug. In -order to do so you must have the slabinfo tool. Then you can do +order to do so you must have the ``slabinfo`` tool. Then you can do +:
[PATCH 20/32] docs/vm: remap_file_pages.txt: conert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/remap_file_pages.txt | 6 ++ 1 file changed, 6 insertions(+) diff --git a/Documentation/vm/remap_file_pages.txt b/Documentation/vm/remap_file_pages.txt index f609142..7bef671 100644 --- a/Documentation/vm/remap_file_pages.txt +++ b/Documentation/vm/remap_file_pages.txt @@ -1,3 +1,9 @@ +.. _remap_file_pages: + +== +remap_file_pages() system call +== + The remap_file_pages() system call is used to create a nonlinear mapping, that is, a mapping in which the pages of the file are mapped into a nonsequential order in memory. The advantage of using remap_file_pages() -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 23/32] docs/vm: split_page_table_lock: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/split_page_table_lock | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/Documentation/vm/split_page_table_lock b/Documentation/vm/split_page_table_lock index 62842a8..889b00b 100644 --- a/Documentation/vm/split_page_table_lock +++ b/Documentation/vm/split_page_table_lock @@ -1,3 +1,6 @@ +.. _split_page_table_lock: + += Split page table lock = @@ -11,6 +14,7 @@ access to the table. At the moment we use split lock for PTE and PMD tables. Access to higher level tables protected by mm->page_table_lock. There are helpers to lock/unlock a table and other accessor functions: + - pte_offset_map_lock() maps pte and takes PTE table lock, returns pointer to the taken lock; @@ -34,12 +38,13 @@ Split page table lock for PMD tables is enabled, if it's enabled for PTE tables and the architecture supports it (see below). Hugetlb and split page table lock -- += Hugetlb can support several page sizes. We use split lock only for PMD level, but not for PUD. Hugetlb-specific helpers: + - huge_pte_lock() takes pmd split lock for PMD_SIZE page, mm->page_table_lock otherwise; @@ -47,7 +52,7 @@ Hugetlb-specific helpers: returns pointer to table lock; Support of split page table lock by an architecture +=== There's no need in special enabling of PTE split page table lock: everything required is done by pgtable_page_ctor() and pgtable_page_dtor(), @@ -73,7 +78,7 @@ NOTE: pgtable_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must be handled properly. page->ptl -- += page->ptl is used to access split page table lock, where 'page' is struct page of page containing the table. It shares storage with page->private @@ -81,6 +86,7 @@ page of page containing the table. It shares storage with page->private To avoid increasing size of struct page and have best performance, we use a trick: + - if spinlock_t fits into long, we use page->ptr as spinlock, so we can avoid indirect access and save a cache line. - if size of spinlock_t is bigger then size of long, we use page->ptl as -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 22/32] docs/vm: soft-dirty.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/soft-dirty.txt | 20 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/Documentation/vm/soft-dirty.txt b/Documentation/vm/soft-dirty.txt index 55684d1..cb0cfd6 100644 --- a/Documentation/vm/soft-dirty.txt +++ b/Documentation/vm/soft-dirty.txt @@ -1,34 +1,38 @@ -SOFT-DIRTY PTEs +.. _soft_dirty: - The soft-dirty is a bit on a PTE which helps to track which pages a task +=== +Soft-Dirty PTEs +=== + +The soft-dirty is a bit on a PTE which helps to track which pages a task writes to. In order to do this tracking one should 1. Clear soft-dirty bits from the task's PTEs. - This is done by writing "4" into the /proc/PID/clear_refs file of the + This is done by writing "4" into the ``/proc/PID/clear_refs`` file of the task in question. 2. Wait some time. 3. Read soft-dirty bits from the PTEs. - This is done by reading from the /proc/PID/pagemap. The bit 55 of the + This is done by reading from the ``/proc/PID/pagemap``. The bit 55 of the 64-bit qword is the soft-dirty one. If set, the respective PTE was written to since step 1. - Internally, to do this tracking, the writable bit is cleared from PTEs +Internally, to do this tracking, the writable bit is cleared from PTEs when the soft-dirty bit is cleared. So, after this, when the task tries to modify a page at some virtual address the #PF occurs and the kernel sets the soft-dirty bit on the respective PTE. - Note, that although all the task's address space is marked as r/o after the +Note, that although all the task's address space is marked as r/o after the soft-dirty bits clear, the #PF-s that occur after that are processed fast. This is so, since the pages are still mapped to physical memory, and thus all the kernel does is finds this fact out and puts both writable and soft-dirty bits on the PTE. - While in most cases tracking memory changes by #PF-s is more than enough +While in most cases tracking memory changes by #PF-s is more than enough there is still a scenario when we can lose soft dirty bits -- a task unmaps a previously mapped memory region and then maps a new one at exactly the same place. When unmap is called, the kernel internally clears PTE values @@ -36,7 +40,7 @@ including soft dirty bits. To notify user space application about such memory region renewal the kernel always marks new memory regions (and expanded regions) as soft dirty. - This feature is actively used by the checkpoint-restore project. You +This feature is actively used by the checkpoint-restore project. You can find more details about it on http://criu.org -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 25/32] docs/vm: transhuge.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/transhuge.txt | 286 - 1 file changed, 166 insertions(+), 120 deletions(-) diff --git a/Documentation/vm/transhuge.txt b/Documentation/vm/transhuge.txt index 4dde03b..569d182 100644 --- a/Documentation/vm/transhuge.txt +++ b/Documentation/vm/transhuge.txt @@ -1,6 +1,11 @@ -= Transparent Hugepage Support = +.. _transhuge: -== Objective == + +Transparent Hugepage Support + + +Objective += Performance critical computing applications dealing with large memory working sets are already running on top of libhugetlbfs and in turn @@ -33,7 +38,8 @@ are using hugepages but a significant speedup already happens if only one of the two is using hugepages just because of the fact the TLB miss is going to run faster. -== Design == +Design +== - "graceful fallback": mm components which don't have transparent hugepage knowledge fall back to breaking huge pmd mapping into table of ptes and, @@ -88,16 +94,17 @@ Applications that gets a lot of benefit from hugepages and that don't risk to lose memory by using hugepages, should use madvise(MADV_HUGEPAGE) on their critical mmapped regions. -== sysfs == +sysfs += Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled -system wide. This can be achieved with one of: +system wide. This can be achieved with one of:: -echo always >/sys/kernel/mm/transparent_hugepage/enabled -echo madvise >/sys/kernel/mm/transparent_hugepage/enabled -echo never >/sys/kernel/mm/transparent_hugepage/enabled + echo always >/sys/kernel/mm/transparent_hugepage/enabled + echo madvise >/sys/kernel/mm/transparent_hugepage/enabled + echo never >/sys/kernel/mm/transparent_hugepage/enabled It's also possible to limit defrag efforts in the VM to generate anonymous hugepages in case they're not immediately free to madvise @@ -108,44 +115,53 @@ use hugepages later instead of regular pages. This isn't always guaranteed, but it may be more likely in case the allocation is for a MADV_HUGEPAGE region. -echo always >/sys/kernel/mm/transparent_hugepage/defrag -echo defer >/sys/kernel/mm/transparent_hugepage/defrag -echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag -echo madvise >/sys/kernel/mm/transparent_hugepage/defrag -echo never >/sys/kernel/mm/transparent_hugepage/defrag - -"always" means that an application requesting THP will stall on allocation -failure and directly reclaim pages and compact memory in an effort to -allocate a THP immediately. This may be desirable for virtual machines -that benefit heavily from THP use and are willing to delay the VM start -to utilise them. - -"defer" means that an application will wake kswapd in the background -to reclaim pages and wake kcompactd to compact memory so that THP is -available in the near future. It's the responsibility of khugepaged -to then install the THP pages later. - -"defer+madvise" will enter direct reclaim and compaction like "always", but -only for regions that have used madvise(MADV_HUGEPAGE); all other regions -will wake kswapd in the background to reclaim pages and wake kcompactd to -compact memory so that THP is available in the near future. - -"madvise" will enter direct reclaim like "always" but only for regions -that are have used madvise(MADV_HUGEPAGE). This is the default behaviour. - -"never" should be self-explanatory. +:: + + echo always >/sys/kernel/mm/transparent_hugepage/defrag + echo defer >/sys/kernel/mm/transparent_hugepage/defrag + echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag + echo madvise >/sys/kernel/mm/transparent_hugepage/defrag + echo never >/sys/kernel/mm/transparent_hugepage/defrag + +always + means that an application requesting THP will stall on + allocation failure and directly reclaim pages and compact + memory in an effort to allocate a THP immediately. This may be + desirable for virtual machines that benefit heavily from THP + use and are willing to delay the VM start to utilise them. + +defer + means that an application will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory so that + THP is available in the near future. It's the responsibility + of khugepaged to then install the THP pages later. + +defer+madvise + will enter direct reclaim and compaction like ``always``, but + only for regions that have used madvise(MADV_HUGEPAGE); all + other regions will wake kswapd in the background to reclaim + pages and w
[PATCH 24/32] docs/vm: swap_numa.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/swap_numa.txt | 55 +- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/Documentation/vm/swap_numa.txt b/Documentation/vm/swap_numa.txt index d5960c9..e0466f2 100644 --- a/Documentation/vm/swap_numa.txt +++ b/Documentation/vm/swap_numa.txt @@ -1,5 +1,8 @@ +.. _swap_numa: + +=== Automatically bind swap device to numa node +=== If the system has more than one swap device and swap device has the node information, we can make use of this information to decide which swap @@ -7,15 +10,16 @@ device to use in get_swap_pages() to get better performance. How to use this feature +=== Swap device has priority and that decides the order of it to be used. To make use of automatically binding, there is no need to manipulate priority settings for swap devices. e.g. on a 2 node machine, assume 2 swap devices swapA and swapB, with swapA attached to node 0 and swapB attached to node 1, are going -to be swapped on. Simply swapping them on by doing: -# swapon /dev/swapA -# swapon /dev/swapB +to be swapped on. Simply swapping them on by doing:: + + # swapon /dev/swapA + # swapon /dev/swapB Then node 0 will use the two swap devices in the order of swapA then swapB and node 1 will use the two swap devices in the order of swapB then swapA. Note @@ -24,32 +28,39 @@ that the order of them being swapped on doesn't matter. A more complex example on a 4 node machine. Assume 6 swap devices are going to be swapped on: swapA and swapB are attached to node 0, swapC is attached to node 1, swapD and swapE are attached to node 2 and swapF is attached to node3. -The way to swap them on is the same as above: -# swapon /dev/swapA -# swapon /dev/swapB -# swapon /dev/swapC -# swapon /dev/swapD -# swapon /dev/swapE -# swapon /dev/swapF - -Then node 0 will use them in the order of: -swapA/swapB -> swapC -> swapD -> swapE -> swapF +The way to swap them on is the same as above:: + + # swapon /dev/swapA + # swapon /dev/swapB + # swapon /dev/swapC + # swapon /dev/swapD + # swapon /dev/swapE + # swapon /dev/swapF + +Then node 0 will use them in the order of:: + + swapA/swapB -> swapC -> swapD -> swapE -> swapF + swapA and swapB will be used in a round robin mode before any other swap device. -node 1 will use them in the order of: -swapC -> swapA -> swapB -> swapD -> swapE -> swapF +node 1 will use them in the order of:: + + swapC -> swapA -> swapB -> swapD -> swapE -> swapF + +node 2 will use them in the order of:: + + swapD/swapE -> swapA -> swapB -> swapC -> swapF -node 2 will use them in the order of: -swapD/swapE -> swapA -> swapB -> swapC -> swapF Similaly, swapD and swapE will be used in a round robin mode before any other swap devices. -node 3 will use them in the order of: -swapF -> swapA -> swapB -> swapC -> swapD -> swapE +node 3 will use them in the order of:: + + swapF -> swapA -> swapB -> swapC -> swapD -> swapE Implementation details --- +== The current code uses a priority based list, swap_avail_list, to decide which swap device to use and if multiple swap devices share the same -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 30/32] docs/vm: zswap.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/zswap.txt | 71 +++--- 1 file changed, 42 insertions(+), 29 deletions(-) diff --git a/Documentation/vm/zswap.txt b/Documentation/vm/zswap.txt index 0b3a114..1444ecd 100644 --- a/Documentation/vm/zswap.txt +++ b/Documentation/vm/zswap.txt @@ -1,4 +1,11 @@ -Overview: +.. _zswap: + += +zswap += + +Overview + Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a @@ -7,32 +14,34 @@ for potentially reduced swap I/O. This trade-off can also result in a significant performance improvement if reads from the compressed cache are faster than reads from a swap device. -NOTE: Zswap is a new feature as of v3.11 and interacts heavily with memory -reclaim. This interaction has not been fully explored on the large set of -potential configurations and workloads that exist. For this reason, zswap -is a work in progress and should be considered experimental. +.. note:: + Zswap is a new feature as of v3.11 and interacts heavily with memory + reclaim. This interaction has not been fully explored on the large set of + potential configurations and workloads that exist. For this reason, zswap + is a work in progress and should be considered experimental. + + Some potential benefits: -Some potential benefits: * Desktop/laptop users with limited RAM capacities can mitigate the -   performance impact of swapping. + performance impact of swapping. * Overcommitted guests that share a common I/O resource can -   dramatically reduce their swap I/O pressure, avoiding heavy handed I/O -throttling by the hypervisor. This allows more work to get done with less -impact to the guest workload and guests sharing the I/O subsystem + dramatically reduce their swap I/O pressure, avoiding heavy handed I/O + throttling by the hypervisor. This allows more work to get done with less + impact to the guest workload and guests sharing the I/O subsystem * Users with SSDs as swap devices can extend the life of the device by -   drastically reducing life-shortening writes. + drastically reducing life-shortening writes. Zswap evicts pages from compressed cache on an LRU basis to the backing swap device when the compressed pool reaches its size limit. This requirement had been identified in prior community discussions. Zswap is disabled by default but can be enabled at boot time by setting -the "enabled" attribute to 1 at boot time. ie: zswap.enabled=1. Zswap +the ``enabled`` attribute to 1 at boot time. ie: ``zswap.enabled=1``. Zswap can also be enabled and disabled at runtime using the sysfs interface. An example command to enable zswap at runtime, assuming sysfs is mounted -at /sys, is: +at ``/sys``, is:: -echo 1 > /sys/module/zswap/parameters/enabled + echo 1 > /sys/module/zswap/parameters/enabled When zswap is disabled at runtime it will stop storing pages that are being swapped out. However, it will _not_ immediately write out or fault @@ -43,7 +52,8 @@ pages out of the compressed pool, a swapoff on the swap device(s) will fault back into memory all swapped out pages, including those in the compressed pool. -Design: +Design +== Zswap receives pages for compression through the Frontswap API and is able to evict pages from its own compressed pool on an LRU basis and write them back to @@ -53,12 +63,12 @@ Zswap makes use of zpool for the managing the compressed memory pool. Each allocation in zpool is not directly accessible by address. Rather, a handle is returned by the allocation routine and that handle must be mapped before being accessed. The compressed memory pool grows on demand and shrinks as compressed -pages are freed. The pool is not preallocated. By default, a zpool of type -zbud is created, but it can be selected at boot time by setting the "zpool" -attribute, e.g. zswap.zpool=zbud. It can also be changed at runtime using the -sysfs "zpool" attribute, e.g. +pages are freed. The pool is not preallocated. By default, a zpool +of type zbud is created, but it can be selected at boot time by +setting the ``zpool`` attribute, e.g. ``zswap.zpool=zbud``. It can +also be changed at runtime using the sysfs ``zpool`` attribute, e.g.:: -echo zbud > /sys/module/zswap/parameters/zpool + echo zbud > /sys/module/zswap/parameters/zpool The zbud type zpool allocates exactly 1 page to store 2 compressed pages, which means the compression ratio will always be 2:1 or worse (because of half-full @@ -83,14 +93,16 @@ via frontswap, to free the compressed entry. Zswap seeks to be simple in its policies. Sysfs attributes allow for one user controlled policy: + * max_pool_percent - The maximum percentage of memory that the compressed -pool can occupy. + pool can occu
[PATCH 27/32] docs/vm: userfaultfd.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/userfaultfd.txt | 66 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/Documentation/vm/userfaultfd.txt b/Documentation/vm/userfaultfd.txt index bb2f945..5048cf6 100644 --- a/Documentation/vm/userfaultfd.txt +++ b/Documentation/vm/userfaultfd.txt @@ -1,6 +1,11 @@ -= Userfaultfd = +.. _userfaultfd: -== Objective == +=== +Userfaultfd +=== + +Objective += Userfaults allow the implementation of on-demand paging from userland and more generally they allow userland to take control of various @@ -9,7 +14,8 @@ memory page faults, something otherwise only the kernel code could do. For example userfaults allows a proper and more optimal implementation of the PROT_NONE+SIGSEGV trick. -== Design == +Design +== Userfaults are delivered and resolved through the userfaultfd syscall. @@ -41,7 +47,8 @@ different processes without them being aware about what is going on themselves on the same region the manager is already tracking, which is a corner case that would currently return -EBUSY). -== API == +API +=== When first opened the userfaultfd must be enabled invoking the UFFDIO_API ioctl specifying a uffdio_api.api value set to UFFD_API (or @@ -101,7 +108,8 @@ UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an half copied page since it'll keep userfaulting until the copy has finished. -== QEMU/KVM == +QEMU/KVM + QEMU/KVM is using the userfaultfd syscall to implement postcopy live migration. Postcopy live migration is one form of memory @@ -163,7 +171,8 @@ sending the same page twice (in case the userfault is read by the postcopy thread just before UFFDIO_COPY|ZEROPAGE runs in the migration thread). -== Non-cooperative userfaultfd == +Non-cooperative userfaultfd +=== When the userfaultfd is monitored by an external manager, the manager must be able to track changes in the process virtual memory @@ -172,27 +181,30 @@ the same read(2) protocol as for the page fault notifications. The manager has to explicitly enable these events by setting appropriate bits in uffdio_api.features passed to UFFDIO_API ioctl: -UFFD_FEATURE_EVENT_FORK - enable userfaultfd hooks for fork(). When -this feature is enabled, the userfaultfd context of the parent process -is duplicated into the newly created process. The manager receives -UFFD_EVENT_FORK with file descriptor of the new userfaultfd context in -the uffd_msg.fork. - -UFFD_FEATURE_EVENT_REMAP - enable notifications about mremap() -calls. When the non-cooperative process moves a virtual memory area to -a different location, the manager will receive UFFD_EVENT_REMAP. The -uffd_msg.remap will contain the old and new addresses of the area and -its original length. - -UFFD_FEATURE_EVENT_REMOVE - enable notifications about -madvise(MADV_REMOVE) and madvise(MADV_DONTNEED) calls. The event -UFFD_EVENT_REMOVE will be generated upon these calls to madvise. The -uffd_msg.remove will contain start and end addresses of the removed -area. - -UFFD_FEATURE_EVENT_UNMAP - enable notifications about memory -unmapping. The manager will get UFFD_EVENT_UNMAP with uffd_msg.remove -containing start and end addresses of the unmapped area. +UFFD_FEATURE_EVENT_FORK + enable userfaultfd hooks for fork(). When this feature is + enabled, the userfaultfd context of the parent process is + duplicated into the newly created process. The manager + receives UFFD_EVENT_FORK with file descriptor of the new + userfaultfd context in the uffd_msg.fork. + +UFFD_FEATURE_EVENT_REMAP + enable notifications about mremap() calls. When the + non-cooperative process moves a virtual memory area to a + different location, the manager will receive + UFFD_EVENT_REMAP. The uffd_msg.remap will contain the old and + new addresses of the area and its original length. + +UFFD_FEATURE_EVENT_REMOVE + enable notifications about madvise(MADV_REMOVE) and + madvise(MADV_DONTNEED) calls. The event UFFD_EVENT_REMOVE will + be generated upon these calls to madvise. The uffd_msg.remove + will contain start and end addresses of the removed area. + +UFFD_FEATURE_EVENT_UNMAP + enable notifications about memory unmapping. The manager will + get UFFD_EVENT_UNMAP with uffd_msg.remove containing start and + end addresses of the unmapped area. Although the UFFD_FEATURE_EVENT_REMOVE and UFFD_FEATURE_EVENT_UNMAP are pretty similar, they quite differ in the action expected from the -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 28/32] docs/vm: z3fold.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/z3fold.txt | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Documentation/vm/z3fold.txt b/Documentation/vm/z3fold.txt index 38e4dac..224e3c6 100644 --- a/Documentation/vm/z3fold.txt +++ b/Documentation/vm/z3fold.txt @@ -1,5 +1,8 @@ +.. _z3fold: + +== z3fold --- +== z3fold is a special purpose allocator for storing compressed pages. It is designed to store up to three compressed pages per physical page. @@ -7,6 +10,7 @@ It is a zbud derivative which allows for higher compression ratio keeping the simplicity and determinism of its predecessor. The main differences between z3fold and zbud are: + * unlike zbud, z3fold allows for up to PAGE_SIZE allocations * z3fold can hold up to 3 compressed pages in its page * z3fold doesn't export any API itself and is thus intended to be used -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 26/32] docs/vm: unevictable-lru.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/unevictable-lru.txt | 117 +++ 1 file changed, 49 insertions(+), 68 deletions(-) diff --git a/Documentation/vm/unevictable-lru.txt b/Documentation/vm/unevictable-lru.txt index e147185..fdd84cb 100644 --- a/Documentation/vm/unevictable-lru.txt +++ b/Documentation/vm/unevictable-lru.txt @@ -1,37 +1,13 @@ - == - UNEVICTABLE LRU INFRASTRUCTURE - == - - -CONTENTS - - - (*) The Unevictable LRU - - - The unevictable page list. - - Memory control group interaction. - - Marking address spaces unevictable. - - Detecting Unevictable Pages. - - vmscan's handling of unevictable pages. - - (*) mlock()'d pages. - - - History. - - Basic management. - - mlock()/mlockall() system call handling. - - Filtering special vmas. - - munlock()/munlockall() system call handling. - - Migrating mlocked pages. - - Compacting mlocked pages. - - mmap(MAP_LOCKED) system call handling. - - munmap()/exit()/exec() system call handling. - - try_to_unmap(). - - try_to_munlock() reverse map scan. - - Page reclaim in shrink_*_list(). +.. _unevictable_lru: +== +Unevictable LRU Infrastructure +== - -INTRODUCTION +.. contents:: :local: + + +Introduction This document describes the Linux memory manager's "Unevictable LRU" @@ -46,8 +22,8 @@ details - the "what does it do?" - by reading the code. One hopes that the descriptions below add value by provide the answer to "why does it do that?". -=== -THE UNEVICTABLE LRU + +The Unevictable LRU === The Unevictable LRU facility adds an additional LRU list to track unevictable @@ -66,17 +42,17 @@ completely unresponsive. The unevictable list addresses the following classes of unevictable pages: - (*) Those owned by ramfs. + * Those owned by ramfs. - (*) Those mapped into SHM_LOCK'd shared memory regions. + * Those mapped into SHM_LOCK'd shared memory regions. - (*) Those mapped into VM_LOCKED [mlock()ed] VMAs. + * Those mapped into VM_LOCKED [mlock()ed] VMAs. The infrastructure may also be able to handle other conditions that make pages unevictable, either by definition or by circumstance, in the future. -THE UNEVICTABLE PAGE LIST +The Unevictable Page List - The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list @@ -118,7 +94,7 @@ the unevictable list when one task has the page isolated from the LRU and other tasks are changing the "evictability" state of the page. -MEMORY CONTROL GROUP INTERACTION +Memory Control Group Interaction The unevictable LRU facility interacts with the memory control group [aka @@ -144,7 +120,9 @@ effects: the control group to thrash or to OOM-kill tasks. -MARKING ADDRESS SPACES UNEVICTABLE +.. _mark_addr_space_unevict: + +Marking Address Spaces Unevictable -- For facilities such as ramfs none of the pages attached to the address space @@ -152,15 +130,15 @@ may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE address space flag is provided, and this can be manipulated by a filesystem using a number of wrapper functions: - (*) void mapping_set_unevictable(struct address_space *mapping); + * ``void mapping_set_unevictable(struct address_space *mapping);`` Mark the address space as being completely unevictable. - (*) void mapping_clear_unevictable(struct address_space *mapping); + * ``void mapping_clear_unevictable(struct address_space *mapping);`` Mark the address space as being evictable. - (*) int mapping_unevictable(struct address_space *mapping); + * ``int mapping_unevictable(struct address_space *mapping);`` Query the address space, and return true if it is completely unevictable. @@ -177,12 +155,13 @@ These are currently used in two places in the kernel: ensure they're in memory. -DETECTING UNEVICTABLE PAGES +Detecting Unevictable Pages --- The function page_evictable() in vmscan.c determines whether a page is -evictable or not using the query function outlined above [see section "Marking -address spaces unevictable"] to check the AS_UNEVICTABLE flag. +evictable or not using the query function outlined above [see section +:ref:`Marking address spaces unevictable `] +to check the AS_UNEVICTABLE flag. For address spaces that are so marked after being populated (as SHM regions might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate @@ -202,7 +181,7 @@ flag, PG_mlocked (as wrapped
[PATCH 29/32] docs/vm: zsmalloc.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/zsmalloc.txt | 60 ++- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/Documentation/vm/zsmalloc.txt b/Documentation/vm/zsmalloc.txt index 64ed63c..6e79893 100644 --- a/Documentation/vm/zsmalloc.txt +++ b/Documentation/vm/zsmalloc.txt @@ -1,5 +1,8 @@ +.. _zsmalloc: + + zsmalloc - + This allocator is designed for use with zram. Thus, the allocator is supposed to work well under low memory conditions. In particular, it @@ -31,40 +34,49 @@ be mapped using zs_map_object() to get a usable pointer and subsequently unmapped using zs_unmap_object(). stat - + With CONFIG_ZSMALLOC_STAT, we could see zsmalloc internal information via -/sys/kernel/debug/zsmalloc/. Here is a sample of stat output: +``/sys/kernel/debug/zsmalloc/``. Here is a sample of stat output:: -# cat /sys/kernel/debug/zsmalloc/zram0/classes + # cat /sys/kernel/debug/zsmalloc/zram0/classes class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage -.. -.. +... +... 9 176 01 186129 8 4 10 192 10 2880 2872135 3 11 208 01 819795 42 2 12 224 01 219159 12 4 -.. -.. +... +... + +class + index +size + object size zspage stores +almost_empty + the number of ZS_ALMOST_EMPTY zspages(see below) +almost_full + the number of ZS_ALMOST_FULL zspages(see below) +obj_allocated + the number of objects allocated +obj_used + the number of objects allocated to the user +pages_used + the number of pages allocated for the class +pages_per_zspage + the number of 0-order pages to make a zspage -class: index -size: object size zspage stores -almost_empty: the number of ZS_ALMOST_EMPTY zspages(see below) -almost_full: the number of ZS_ALMOST_FULL zspages(see below) -obj_allocated: the number of objects allocated -obj_used: the number of objects allocated to the user -pages_used: the number of pages allocated for the class -pages_per_zspage: the number of 0-order pages to make a zspage +We assign a zspage to ZS_ALMOST_EMPTY fullness group when n <= N / f, where -We assign a zspage to ZS_ALMOST_EMPTY fullness group when: - n <= N / f, where -n = number of allocated objects -N = total number of objects zspage can store -f = fullness_threshold_frac(ie, 4 at the moment) +* n = number of allocated objects +* N = total number of objects zspage can store +* f = fullness_threshold_frac(ie, 4 at the moment) Similarly, we assign zspage to: - ZS_ALMOST_FULL when n > N / f - ZS_EMPTYwhen n == 0 - ZS_FULL when n == N + +* ZS_ALMOST_FULL when n > N / f +* ZS_EMPTYwhen n == 0 +* ZS_FULL when n == N -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 31/32] docs/vm: rename documentation files to .rst
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/ABI/stable/sysfs-devices-node| 2 +-
.../ABI/testing/sysfs-kernel-mm-hugepages | 2 +-
Documentation/ABI/testing/sysfs-kernel-mm-ksm | 2 +-
Documentation/ABI/testing/sysfs-kernel-slab| 4 +-
Documentation/admin-guide/kernel-parameters.txt| 12 ++---
Documentation/dev-tools/kasan.rst | 2 +-
Documentation/filesystems/proc.txt | 4 +-
Documentation/filesystems/tmpfs.txt| 2 +-
Documentation/sysctl/vm.txt| 6 +--
Documentation/vm/00-INDEX | 58 +++---
Documentation/vm/{active_mm.txt => active_mm.rst} | 0
Documentation/vm/{balance => balance.rst} | 0
.../vm/{cleancache.txt => cleancache.rst} | 0
Documentation/vm/{frontswap.txt => frontswap.rst} | 0
Documentation/vm/{highmem.txt => highmem.rst} | 0
Documentation/vm/{hmm.txt => hmm.rst} | 0
.../{hugetlbfs_reserv.txt => hugetlbfs_reserv.rst} | 0
.../vm/{hugetlbpage.txt => hugetlbpage.rst}| 2 +-
Documentation/vm/{hwpoison.txt => hwpoison.rst}| 2 +-
...le_page_tracking.txt => idle_page_tracking.rst} | 2 +-
Documentation/vm/{ksm.txt => ksm.rst} | 0
.../vm/{mmu_notifier.txt => mmu_notifier.rst} | 0
Documentation/vm/{numa => numa.rst}| 2 +-
...ma_memory_policy.txt => numa_memory_policy.rst} | 0
...commit-accounting => overcommit-accounting.rst} | 0
Documentation/vm/{page_frags => page_frags.rst}| 0
.../vm/{page_migration => page_migration.rst} | 0
.../vm/{page_owner.txt => page_owner.rst} | 0
Documentation/vm/{pagemap.txt => pagemap.rst} | 6 +--
.../{remap_file_pages.txt => remap_file_pages.rst} | 0
Documentation/vm/{slub.txt => slub.rst}| 0
.../vm/{soft-dirty.txt => soft-dirty.rst} | 0
...t_page_table_lock => split_page_table_lock.rst} | 0
Documentation/vm/{swap_numa.txt => swap_numa.rst} | 0
Documentation/vm/{transhuge.txt => transhuge.rst} | 0
.../{unevictable-lru.txt => unevictable-lru.rst} | 0
.../vm/{userfaultfd.txt => userfaultfd.rst}| 0
Documentation/vm/{z3fold.txt => z3fold.rst}| 0
Documentation/vm/{zsmalloc.txt => zsmalloc.rst}| 0
Documentation/vm/{zswap.txt => zswap.rst} | 0
MAINTAINERS| 2 +-
arch/alpha/Kconfig | 2 +-
arch/ia64/Kconfig | 2 +-
arch/mips/Kconfig | 2 +-
arch/powerpc/Kconfig | 2 +-
fs/Kconfig | 2 +-
fs/dax.c | 2 +-
fs/proc/task_mmu.c | 4 +-
include/linux/hmm.h| 2 +-
include/linux/memremap.h | 4 +-
include/linux/mmu_notifier.h | 2 +-
include/linux/sched/mm.h | 4 +-
include/linux/swap.h | 2 +-
mm/Kconfig | 6 +--
mm/cleancache.c| 2 +-
mm/frontswap.c | 2 +-
mm/hmm.c | 2 +-
mm/huge_memory.c | 4 +-
mm/hugetlb.c | 4 +-
mm/ksm.c | 4 +-
mm/mmap.c | 2 +-
mm/rmap.c | 6 +--
mm/util.c | 2 +-
63 files changed, 87 insertions(+), 87 deletions(-)
rename Documentation/vm/{active_mm.txt => active_mm.rst} (100%)
rename Documentation/vm/{balance => balance.rst} (100%)
rename Documentation/vm/{cleancache.txt => cleancache.rst} (100%)
rename Documentation/vm/{frontswap.txt => frontswap.rst} (100%)
rename Documentation/vm/{highmem.txt => highmem.rst} (100%)
rename Documentation/vm/{hmm.txt => hmm.rst} (100%)
rename Documentation/vm/{hugetlbfs_reserv.txt => hugetlbfs_reserv.rst} (100%)
rename Documentation/vm/{hugetlbpage.txt => hugetlbpage.rst} (99%)
rename Documentation/vm/{hwpoison.txt => hwpoison.rst} (99%)
rename Documentation/vm/{idle_page_tracking.txt => idle_page_tracking.rst}
(98%)
rename Documentation/vm/{ksm.txt => ksm.rst} (100%)
rename Documentation/vm/{mmu_notifier.txt => mmu_notifier.rst} (100%)
rename Documentation/vm/{numa => numa.rst} (99%)
rename Documentation/vm/{numa_memory_policy.txt => numa_memory_policy.rst}
(100%)
rename Documentation/vm/{overcommit-accounting => overcommit-accounting.rst}
(100%)
[PATCH 32/32] docs/vm: add index.rst and link MM documentation to top level index
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/index.rst| 3 ++-
Documentation/vm/conf.py | 10 +
Documentation/vm/index.rst | 56 ++
3 files changed, 68 insertions(+), 1 deletion(-)
create mode 100644 Documentation/vm/conf.py
create mode 100644 Documentation/vm/index.rst
diff --git a/Documentation/index.rst b/Documentation/index.rst
index ef5080c..cc4a098 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -45,7 +45,7 @@ the kernel interface as seen by application developers.
.. toctree::
:maxdepth: 2
- userspace-api/index
+ userspace-api/index
Introduction to kernel development
@@ -88,6 +88,7 @@ needed).
sound/index
crypto/index
filesystems/index
+ vm/index
Architecture-specific documentation
---
diff --git a/Documentation/vm/conf.py b/Documentation/vm/conf.py
new file mode 100644
index 000..3b0b601
--- /dev/null
+++ b/Documentation/vm/conf.py
@@ -0,0 +1,10 @@
+# -*- coding: utf-8; mode: python -*-
+
+project = "Linux Memory Management Documentation"
+
+tags.add("subproject")
+
+latex_documents = [
+('index', 'memory-management.tex', project,
+ 'The kernel development community', 'manual'),
+]
diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
new file mode 100644
index 000..6c45142
--- /dev/null
+++ b/Documentation/vm/index.rst
@@ -0,0 +1,56 @@
+=
+Linux Memory Management Documentation
+=
+
+This is a collection of documents about Linux memory management (mm) subsystem.
+
+User guides for MM features
+===
+
+The following documents provide guides for controlling and tuning
+various features of the Linux memory management
+
+.. toctree::
+ :maxdepth: 1
+
+ hugetlbpage
+ idle_page_tracking
+ ksm
+ numa_memory_policy
+ pagemap
+ transhuge
+ soft-dirty
+ swap_numa
+ userfaultfd
+ zswap
+
+Kernel developers MM documentation
+==
+
+The below documents describe MM internals with different level of
+details ranging from notes and mailing list responses to elaborate
+descriptions of data structures and algorithms.
+
+.. toctree::
+ :maxdepth: 1
+
+ active_mm
+ balance
+ cleancache
+ frontswap
+ highmem
+ hmm
+ hwpoison
+ hugetlbfs_reserv
+ mmu_notifier
+ numa
+ overcommit-accounting
+ page_migration
+ page_frags
+ page_owner
+ remap_file_pages
+ slub
+ split_page_table_lock
+ unevictable-lru
+ z3fold
+ zsmalloc
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 19/32] docs/vm: page_owner: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/page_owner.txt | 34 +- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/Documentation/vm/page_owner.txt b/Documentation/vm/page_owner.txt index 143..0ed5ab8 100644 --- a/Documentation/vm/page_owner.txt +++ b/Documentation/vm/page_owner.txt @@ -1,7 +1,11 @@ +.. _page_owner: + +== page owner: Tracking about who allocated each page +== -* Introduction +Introduction + page owner is for the tracking about who allocated each page. It can be used to debug memory leak or to find a memory hogger. @@ -34,13 +38,15 @@ not affect to allocation performance, especially if the static keys jump label patching functionality is available. Following is the kernel's code size change due to this facility. -- Without page owner +- Without page owner:: + textdata bss dec hex filename - 406621493 644 42799a72f mm/page_alloc.o + 40662 1493 644 42799a72f mm/page_alloc.o + +- With page owner:: -- With page owner textdata bss dec hex filename - 408921493 644 43029a815 mm/page_alloc.o + 40892 1493 644 43029a815 mm/page_alloc.o 1427 24 81459 5b3 mm/page_ext.o 2722 50 02772 ad4 mm/page_owner.o @@ -62,21 +68,23 @@ are catched and marked, although they are mostly allocated from struct page extension feature. Anyway, after that, no page is left in un-tracking state. -* Usage +Usage += + +1) Build user-space helper:: -1) Build user-space helper cd tools/vm make page_owner_sort -2) Enable page owner - Add "page_owner=on" to boot cmdline. +2) Enable page owner: add "page_owner=on" to boot cmdline. 3) Do the job what you want to debug -4) Analyze information from page owner +4) Analyze information from page owner:: + cat /sys/kernel/debug/page_owner > page_owner_full.txt grep -v ^PFN page_owner_full.txt > page_owner.txt ./page_owner_sort page_owner.txt sorted_page_owner.txt - See the result about who allocated each page - in the sorted_page_owner.txt. + See the result about who allocated each page + in the ``sorted_page_owner.txt``. -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 14/32] docs/vm: overcommit-accounting: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/overcommit-accounting | 107 ++--- 1 file changed, 57 insertions(+), 50 deletions(-) diff --git a/Documentation/vm/overcommit-accounting b/Documentation/vm/overcommit-accounting index cbfaaa6..0dd54bb 100644 --- a/Documentation/vm/overcommit-accounting +++ b/Documentation/vm/overcommit-accounting @@ -1,80 +1,87 @@ +.. _overcommit_accounting: + += +Overcommit Accounting += + The Linux kernel supports the following overcommit handling modes -0 - Heuristic overcommit handling. Obvious overcommits of - address space are refused. Used for a typical system. It - ensures a seriously wild allocation fails while allowing - overcommit to reduce swap usage. root is allowed to - allocate slightly more memory in this mode. This is the - default. +0 + Heuristic overcommit handling. Obvious overcommits of address + space are refused. Used for a typical system. It ensures a + seriously wild allocation fails while allowing overcommit to + reduce swap usage. root is allowed to allocate slightly more + memory in this mode. This is the default. -1 - Always overcommit. Appropriate for some scientific - applications. Classic example is code using sparse arrays - and just relying on the virtual memory consisting almost - entirely of zero pages. +1 + Always overcommit. Appropriate for some scientific + applications. Classic example is code using sparse arrays and + just relying on the virtual memory consisting almost entirely + of zero pages. -2 - Don't overcommit. The total address space commit - for the system is not permitted to exceed swap + a - configurable amount (default is 50%) of physical RAM. - Depending on the amount you use, in most situations - this means a process will not be killed while accessing - pages but will receive errors on memory allocation as - appropriate. +2 + Don't overcommit. The total address space commit for the + system is not permitted to exceed swap + a configurable amount + (default is 50%) of physical RAM. Depending on the amount you + use, in most situations this means a process will not be + killed while accessing pages but will receive errors on memory + allocation as appropriate. - Useful for applications that want to guarantee their - memory allocations will be available in the future - without having to initialize every page. + Useful for applications that want to guarantee their memory + allocations will be available in the future without having to + initialize every page. -The overcommit policy is set via the sysctl `vm.overcommit_memory'. +The overcommit policy is set via the sysctl ``vm.overcommit_memory``. -The overcommit amount can be set via `vm.overcommit_ratio' (percentage) -or `vm.overcommit_kbytes' (absolute value). +The overcommit amount can be set via ``vm.overcommit_ratio`` (percentage) +or ``vm.overcommit_kbytes`` (absolute value). The current overcommit limit and amount committed are viewable in -/proc/meminfo as CommitLimit and Committed_AS respectively. +``/proc/meminfo`` as CommitLimit and Committed_AS respectively. Gotchas +=== The C language stack growth does an implicit mremap. If you want absolute -guarantees and run close to the edge you MUST mmap your stack for the +guarantees and run close to the edge you MUST mmap your stack for the largest size you think you will need. For typical stack usage this does not matter much but it's a corner case if you really really care -In mode 2 the MAP_NORESERVE flag is ignored. +In mode 2 the MAP_NORESERVE flag is ignored. How It Works - + The overcommit is based on the following rules For a file backed map - SHARED or READ-only - 0 cost (the file is the map not swap) - PRIVATE WRITABLE- size of mapping per instance + | SHARED or READ-only - 0 cost (the file is the map not swap) + | PRIVATE WRITABLE - size of mapping per instance -For an anonymous or /dev/zero map - SHARED - size of mapping - PRIVATE READ-only - 0 cost (but of little use) - PRIVATE WRITABLE- size of mapping per instance +For an anonymous or ``/dev/zero`` map + | SHARED- size of mapping + | PRIVATE READ-only - 0 cost (but of little use) + | PRIVATE WRITABLE - size of mapping per instance Additional accounting - Pages made writable copies b
[PATCH 11/32] docs/vm: ksm.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/ksm.txt | 215 --- 1 file changed, 110 insertions(+), 105 deletions(-) diff --git a/Documentation/vm/ksm.txt b/Documentation/vm/ksm.txt index 6686bd2..87e7eef 100644 --- a/Documentation/vm/ksm.txt +++ b/Documentation/vm/ksm.txt @@ -1,8 +1,11 @@ -How to use the Kernel Samepage Merging feature --- +.. _ksm: + +=== +Kernel Samepage Merging +=== KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, -added to the Linux kernel in 2.6.32. See mm/ksm.c for its implementation, +added to the Linux kernel in 2.6.32. See ``mm/ksm.c`` for its implementation, and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ The KSM daemon ksmd periodically scans those areas of user memory which @@ -51,110 +54,112 @@ Applications should be considerate in their use of MADV_MERGEABLE, restricting its use to areas likely to benefit. KSM's scans may use a lot of processing power: some installations will disable KSM for that reason. -The KSM daemon is controlled by sysfs files in /sys/kernel/mm/ksm/, +The KSM daemon is controlled by sysfs files in ``/sys/kernel/mm/ksm/``, readable by all but writable only by root: -pages_to_scan- how many present pages to scan before ksmd goes to sleep - e.g. "echo 100 > /sys/kernel/mm/ksm/pages_to_scan" - Default: 100 (chosen for demonstration purposes) - -sleep_millisecs - how many milliseconds ksmd should sleep before next scan - e.g. "echo 20 > /sys/kernel/mm/ksm/sleep_millisecs" - Default: 20 (chosen for demonstration purposes) - -merge_across_nodes - specifies if pages from different numa nodes can be merged. - When set to 0, ksm merges only pages which physically - reside in the memory area of same NUMA node. That brings - lower latency to access of shared pages. Systems with more - nodes, at significant NUMA distances, are likely to benefit - from the lower latency of setting 0. Smaller systems, which - need to minimize memory usage, are likely to benefit from - the greater sharing of setting 1 (default). You may wish to - compare how your system performs under each setting, before - deciding on which to use. merge_across_nodes setting can be - changed only when there are no ksm shared pages in system: - set run 2 to unmerge pages first, then to 1 after changing - merge_across_nodes, to remerge according to the new setting. - Default: 1 (merging across nodes as in earlier releases) - -run - set 0 to stop ksmd from running but keep merged pages, - set 1 to run ksmd e.g. "echo 1 > /sys/kernel/mm/ksm/run", - set 2 to stop ksmd and unmerge all pages currently merged, - but leave mergeable areas registered for next run - Default: 0 (must be changed to 1 to activate KSM, - except if CONFIG_SYSFS is disabled) - -use_zero_pages - specifies whether empty pages (i.e. allocated pages - that only contain zeroes) should be treated specially. - When set to 1, empty pages are merged with the kernel - zero page(s) instead of with each other as it would - happen normally. This can improve the performance on - architectures with coloured zero pages, depending on - the workload. Care should be taken when enabling this - setting, as it can potentially degrade the performance - of KSM for some workloads, for example if the checksums - of pages candidate for merging match the checksum of - an empty page. This setting can be changed at any time, - it is only effective for pages merged after the change. - Default: 0 (normal KSM behaviour as in earlier releases) - -max_page_sharing - Maximum sharing allowed for each KSM page. This - enforces a deduplication limit to avoid the virtual - memory rmap lists to grow too large. The minimum - value is 2 as a newly created KSM page will have at - least two sharers. The rmap walk has O(N) - complexity where N is the number of rmap_items - (i.e. virtual mappings) that are sharing the page, - which is in turn capped by max_page_sharing. So - this effectively spread the the linear O(N) -
[PATCH 05/32] docs/vm: highmem.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/highmem.txt | 87 ++-- 1 file changed, 36 insertions(+), 51 deletions(-) diff --git a/Documentation/vm/highmem.txt b/Documentation/vm/highmem.txt index 4324d24..0f69a9f 100644 --- a/Documentation/vm/highmem.txt +++ b/Documentation/vm/highmem.txt @@ -1,25 +1,14 @@ +.. _highmem: - -HIGH MEMORY HANDLING - + +High Memory Handling + By: Peter Zijlstra <a.p.zijls...@chello.nl> -Contents: - - (*) What is high memory? - - (*) Temporary virtual mappings. - - (*) Using kmap_atomic. - - (*) Cost of temporary mappings. - - (*) i386 PAE. +.. contents:: :local: - - -WHAT IS HIGH MEMORY? +What Is High Memory? High memory (highmem) is used when the size of physical memory approaches or @@ -38,7 +27,7 @@ kernel entry/exit. This means the available virtual memory space (4GiB on i386) has to be divided between user and kernel space. The traditional split for architectures using this approach is 3:1, 3GiB for -userspace and the top 1GiB for kernel space: +userspace and the top 1GiB for kernel space:: ++ 0x | Kernel | @@ -58,40 +47,38 @@ and user maps. Some hardware (like some ARMs), however, have limited virtual space when they use mm context tags. -== -TEMPORARY VIRTUAL MAPPINGS +Temporary Virtual Mappings == The kernel contains several ways of creating temporary mappings: - (*) vmap(). This can be used to make a long duration mapping of multiple - physical pages into a contiguous virtual space. It needs global - synchronization to unmap. +* vmap(). This can be used to make a long duration mapping of multiple + physical pages into a contiguous virtual space. It needs global + synchronization to unmap. - (*) kmap(). This permits a short duration mapping of a single page. It needs - global synchronization, but is amortized somewhat. It is also prone to - deadlocks when using in a nested fashion, and so it is not recommended for - new code. +* kmap(). This permits a short duration mapping of a single page. It needs + global synchronization, but is amortized somewhat. It is also prone to + deadlocks when using in a nested fashion, and so it is not recommended for + new code. - (*) kmap_atomic(). This permits a very short duration mapping of a single - page. Since the mapping is restricted to the CPU that issued it, it - performs well, but the issuing task is therefore required to stay on that - CPU until it has finished, lest some other task displace its mappings. +* kmap_atomic(). This permits a very short duration mapping of a single + page. Since the mapping is restricted to the CPU that issued it, it + performs well, but the issuing task is therefore required to stay on that + CPU until it has finished, lest some other task displace its mappings. - kmap_atomic() may also be used by interrupt contexts, since it is does not - sleep and the caller may not sleep until after kunmap_atomic() is called. + kmap_atomic() may also be used by interrupt contexts, since it is does not + sleep and the caller may not sleep until after kunmap_atomic() is called. - It may be assumed that k[un]map_atomic() won't fail. + It may be assumed that k[un]map_atomic() won't fail. -= -USING KMAP_ATOMIC +Using kmap_atomic = When and where to use kmap_atomic() is straightforward. It is used when code wants to access the contents of a page that might be allocated from high memory (see __GFP_HIGHMEM), for example a page in the pagecache. The API has two -functions, and they can be used in a manner similar to the following: +functions, and they can be used in a manner similar to the following:: /* Find the page of interest. */ struct page *page = find_get_page(mapping, offset); @@ -109,7 +96,7 @@ Note that the kunmap_atomic() call takes the result of the kmap_atomic() call not the argument. If you need to map two pages because you want to copy from one page to -another you need to keep the kmap_atomic calls strictly nested, like: +another you need to keep the kmap_atomic calls strictly nested, like:: vaddr1 = kmap_atomic(page1); vaddr2 = kmap_atomic(page2); @@ -120,8 +107,7 @@ another you need to keep the kmap_atomic calls strictly nested, like: kunmap_atomic(vaddr1); -== -COST OF TEMPORARY MAPPINGS +Cost of Temporary Mappings == The cost of creating temporary mappings can be quite high. The arch has to @@ -136,25 +122,24 @@ If CONFIG_MMU is not set, then there can be
[PATCH 06/32] docs/vm: hmm.txt: convert to ReST format
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
Documentation/vm/hmm.txt | 66
1 file changed, 28 insertions(+), 38 deletions(-)
diff --git a/Documentation/vm/hmm.txt b/Documentation/vm/hmm.txt
index 4d3aac9..3fafa33 100644
--- a/Documentation/vm/hmm.txt
+++ b/Documentation/vm/hmm.txt
@@ -1,4 +1,8 @@
+.. hmm:
+
+=
Heterogeneous Memory Management (HMM)
+=
Transparently allow any component of a program to use any memory region of said
program with a device without using device specific memory allocator. This is
@@ -14,19 +18,10 @@ deals with how device memory is represented inside the
kernel. Finaly the last
section present the new migration helper that allow to leverage the device DMA
engine.
+.. contents:: :local:
-1) Problems of using device specific memory allocator:
-2) System bus, device memory characteristics
-3) Share address space and migration
-4) Address space mirroring implementation and API
-5) Represent and manage device memory from core kernel point of view
-6) Migrate to and from device memory
-7) Memory cgroup (memcg) and rss accounting
-
-
-
-1) Problems of using device specific memory allocator:
+Problems of using device specific memory allocator
+==
Device with large amount of on board memory (several giga bytes) like GPU have
historically manage their memory through dedicated driver specific API. This
@@ -68,9 +63,8 @@ only do-able with a share address. It is as well more
reasonable to use a share
address space for all the other patterns.
-
-2) System bus, device memory characteristics
+System bus, device memory characteristics
+=
System bus cripple share address due to few limitations. Most system bus only
allow basic memory access from device to main memory, even cache coherency is
@@ -100,9 +94,8 @@ access any memory memory but we must also permit any memory
to be migrated to
device memory while device is using it (blocking CPU access while it happens).
-
-3) Share address space and migration
+Share address space and migration
+=
HMM intends to provide two main features. First one is to share the address
space by duplication the CPU page table into the device page table so same
@@ -140,14 +133,13 @@ leverage device memory by migrating part of data-set that
is actively use by a
device.
-
-4) Address space mirroring implementation and API
+Address space mirroring implementation and API
+==
Address space mirroring main objective is to allow to duplicate range of CPU
page table into a device page table and HMM helps keeping both synchronize. A
device driver that want to mirror a process address space must start with the
-registration of an hmm_mirror struct:
+registration of an hmm_mirror struct::
int hmm_mirror_register(struct hmm_mirror *mirror,
struct mm_struct *mm);
@@ -156,7 +148,7 @@ registration of an hmm_mirror struct:
The locked variant is to be use when the driver is already holding the mmap_sem
of the mm in write mode. The mirror struct has a set of callback that are use
-to propagate CPU page table:
+to propagate CPU page table::
struct hmm_mirror_ops {
/* sync_cpu_device_pagetables() - synchronize page tables
@@ -187,7 +179,8 @@ be done with the update.
When device driver wants to populate a range of virtual address it can use
-either:
+either::
+
int hmm_vma_get_pfns(struct vm_area_struct *vma,
struct hmm_range *range,
unsigned long start,
@@ -211,7 +204,7 @@ that array correspond to an address in the virtual range.
HMM provide a set of
flags to help driver identify special CPU page table entries.
Locking with the update() callback is the most important aspect the driver must
-respect in order to keep things properly synchronize. The usage pattern is :
+respect in order to keep things properly synchronize. The usage pattern is::
int driver_populate_range(...)
{
@@ -251,9 +244,8 @@ concurrently for multiple devices. Waiting for each device
to report commands
as executed is serialize (there is no point in doing this concurrently).
-
-5) Represent and manage device memory from core kernel point of view
+Represent and manage device memory from core kernel point o
[PATCH] docs/vm: update 00-INDEX
Several files were added to Documentation/vm without updates to 00-INDEX. Fill in the missing documents Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/vm/00-INDEX | 18 ++ 1 file changed, 18 insertions(+) diff --git a/Documentation/vm/00-INDEX b/Documentation/vm/00-INDEX index 11d3d8d..0278f2c 100644 --- a/Documentation/vm/00-INDEX +++ b/Documentation/vm/00-INDEX @@ -10,6 +10,8 @@ frontswap.txt - Outline frontswap, part of the transcendent memory frontend. highmem.txt - Outline of highmem and common issues. +hmm.txt + - Documentation of heterogeneous memory management hugetlbpage.txt - a brief summary of hugetlbpage support in the Linux kernel. hugetlbfs_reserv.txt @@ -20,25 +22,41 @@ idle_page_tracking.txt - description of the idle page tracking feature. ksm.txt - how to use the Kernel Samepage Merging feature. +mmu_notifier.txt + - a note about clearing pte/pmd and mmu notifications numa - information about NUMA specific code in the Linux vm. numa_memory_policy.txt - documentation of concepts and APIs of the 2.6 memory policy support. overcommit-accounting - description of the Linux kernels overcommit handling modes. +page_frags + - description of page fragments allocator page_migration - description of page migration in NUMA systems. pagemap.txt - pagemap, from the userspace perspective +page_owner.txt + - tracking about who allocated each page +remap_file_pages.txt + - a note about remap_file_pages() system call slub.txt - a short users guide for SLUB. soft-dirty.txt - short explanation for soft-dirty PTEs split_page_table_lock - Separate per-table lock to improve scalability of the old page_table_lock. +swap_numa.txt + - automatic binding of swap device to numa node transhuge.txt - Transparent Hugepage Support, alternative way of using hugepages. unevictable-lru.txt - Unevictable LRU infrastructure +userfaultfd.txt + - description of userfaultfd system call +z3fold.txt + - outline of z3fold allocator for storing compressed pages +zsmalloc.txt + - outline of zsmalloc allocator for storing compressed pages zswap.txt - Intro to compressed cache for swap pages -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] mm/nommu: remove description of alloc_vm_area
The alloc_mm_area in nommu is a stub, but it's description states it
allocates kernel address space.
Remove the description to make the code and the documentation agree.
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
mm/nommu.c | 12
1 file changed, 12 deletions(-)
diff --git a/mm/nommu.c b/mm/nommu.c
index ebb6e618dade..4f1cabd07e81 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -457,18 +457,6 @@ void __weak vmalloc_sync_all(void)
{
}
-/**
- * alloc_vm_area - allocate a range of kernel address space
- * @size: size of the area
- *
- * Returns:NULL on failure, vm_struct on success
- *
- * This function reserves a range of kernel address space, and
- * allocates pagetables to map that range. No actual mappings
- * are created. If the kernel address space is not shared
- * between processes, it syncs the pagetable across all
- * processes.
- */
struct vm_struct *alloc_vm_area(size_t size, pte_t **ptes)
{
BUG();
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] mm: kernel-doc: add missing parameter descriptions
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
mm/cma.c| 5 +
mm/compaction.c | 1 +
mm/kmemleak.c | 10 ++
mm/memory_hotplug.c | 6 ++
mm/oom_kill.c | 2 ++
mm/pagewalk.c | 3 +++
mm/rmap.c | 1 +
mm/zsmalloc.c | 2 ++
8 files changed, 30 insertions(+)
diff --git a/mm/cma.c b/mm/cma.c
index 0607729abf3b..0600fc08a9f4 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -165,6 +165,9 @@ core_initcall(cma_init_reserved_areas);
* @base: Base address of the reserved area
* @size: Size of the reserved area (in bytes),
* @order_per_bit: Order of pages represented by one bit on bitmap.
+ * @name: The name of the area. If this parameter is NULL, the name of
+ *the area will be set to "cmaN", where N is a running counter of
+ *used areas.
* @res_cma: Pointer to store the created cma region.
*
* This function creates custom contiguous area from already reserved memory.
@@ -227,6 +230,7 @@ int __init cma_init_reserved_mem(phys_addr_t base,
phys_addr_t size,
* @alignment: Alignment for the CMA area, should be power of 2 or zero
* @order_per_bit: Order of pages represented by one bit on bitmap.
* @fixed: hint about where to place the reserved area
+ * @name: The name of the area. See function cma_init_reserved_mem()
* @res_cma: Pointer to store the created cma region.
*
* This function reserves memory from early allocator. It should be
@@ -390,6 +394,7 @@ static inline void cma_debug_show_areas(struct cma *cma) { }
* @cma: Contiguous memory region for which the allocation is performed.
* @count: Requested number of pages.
* @align: Requested alignment of pages (in PAGE_SIZE order).
+ * @gfp_mask: GFP mask to use during compaction
*
* This function allocates part of contiguous memory on specific
* contiguous memory area.
diff --git a/mm/compaction.c b/mm/compaction.c
index 2c8999d027ab..40966f88c39d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -576,6 +576,7 @@ static unsigned long isolate_freepages_block(struct
compact_control *cc,
/**
* isolate_freepages_range() - isolate free pages.
+ * @cc:Compaction control structure.
* @start_pfn: The first PFN to start isolating.
* @end_pfn: The one-past-last PFN.
*
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index e83987c55a08..7e73961dcf89 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -1187,6 +1187,11 @@ EXPORT_SYMBOL(kmemleak_no_scan);
/**
* kmemleak_alloc_phys - similar to kmemleak_alloc but taking a physical
* address argument
+ * @phys: physical address of the object
+ * @size: size of the object
+ * @min_count: minimum number of references to this object.
+ * See kmemleak_alloc()
+ * @gfp: kmalloc() flags used for kmemleak internal memory allocations
*/
void __ref kmemleak_alloc_phys(phys_addr_t phys, size_t size, int min_count,
gfp_t gfp)
@@ -1199,6 +1204,9 @@ EXPORT_SYMBOL(kmemleak_alloc_phys);
/**
* kmemleak_free_part_phys - similar to kmemleak_free_part but taking a
* physical address argument
+ * @phys: physical address if the beginning or inside an object. This
+ * also represents the start of the range to be freed
+ * @size: size to be unregistered
*/
void __ref kmemleak_free_part_phys(phys_addr_t phys, size_t size)
{
@@ -1210,6 +1218,7 @@ EXPORT_SYMBOL(kmemleak_free_part_phys);
/**
* kmemleak_not_leak_phys - similar to kmemleak_not_leak but taking a physical
* address argument
+ * @phys: physical address of the object
*/
void __ref kmemleak_not_leak_phys(phys_addr_t phys)
{
@@ -1221,6 +1230,7 @@ EXPORT_SYMBOL(kmemleak_not_leak_phys);
/**
* kmemleak_ignore_phys - similar to kmemleak_ignore but taking a physical
* address argument
+ * @phys: physical address of the object
*/
void __ref kmemleak_ignore_phys(phys_addr_t phys)
{
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index b2bd52ff7605..e84d626d7b89 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -559,6 +559,7 @@ static int __remove_section(struct zone *zone, struct
mem_section *ms,
* @zone: zone from which pages need to be removed
* @phys_start_pfn: starting pageframe (must be aligned to start of a section)
* @nr_pages: number of pages to remove (must be multiple of section size)
+ * @altmap: alternative device page map or %NULL if default memmap is used
*
* Generic helper function to remove section mappings and sysfs entries
* for the section of the memory we are removing. Caller needs to make
@@ -1055,6 +1056,7 @@ static void rollback_node_hotadd(int nid, pg_data_t
*pgdat)
/**
* try_online_node - online a node if offlined
+ * @nid: the node ID
*
* called by cpu_up() to online a node without onlined memory.
*/
@@ -1814,6 +1816,7 @@ static int check_and_unmap_cpu_on_node(
[PATCH 0/3] mm: docs: more trivial updates
Hi, Here's another set of (mostly) trivial updates for kernel-doc descriptions in the mm code. Mike Rapoport (3): mm/nommu: remove description of alloc_vm_area mm/swap: remove @cold parameter description for release_pages mm: kernel-doc: add missing parameter descriptions mm/cma.c| 5 + mm/compaction.c | 1 + mm/kmemleak.c | 10 ++ mm/memory_hotplug.c | 6 ++ mm/nommu.c | 12 mm/oom_kill.c | 2 ++ mm/pagewalk.c | 3 +++ mm/rmap.c | 1 + mm/swap.c | 1 - mm/zsmalloc.c | 2 ++ 10 files changed, 30 insertions(+), 13 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] doc-guide: kernel-doc: add comment about formatting verification
Currently there is no automated checking for kernel-doc comments except running 'kernel-doc -v -none '. Mention the possibility to run kernel-doc to verify formatting of the comments in the kernel-doc guide. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- v2: * fix typo * add a sentence about using 'make W=n' Documentation/doc-guide/kernel-doc.rst | 11 +++ 1 file changed, 11 insertions(+) diff --git a/Documentation/doc-guide/kernel-doc.rst b/Documentation/doc-guide/kernel-doc.rst index c6c3297..80383b1 100644 --- a/Documentation/doc-guide/kernel-doc.rst +++ b/Documentation/doc-guide/kernel-doc.rst @@ -44,6 +44,17 @@ that somebody changing the code will also change the documentation. The overview kernel-doc comments may be placed anywhere at the top indentation level. +Running the ``kernel-doc`` tool with increased verbosity and without actual +output generation may be used to verify proper formatting of the +documentation comments. For example:: + + scripts/kernel-doc -v -none drivers/foo/bar.c + +The documentation format is verified by the kernel build when it is +requested to perform extra gcc checks:: + + make W=n + Function documentation -- -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] doc-guide: kernel-doc: add comment about formatting verification
Currently there is no automated checking for kernel-doc comments except running 'kernel-doc -v -none '. Mention the possibility to run kernel-doc to verify formatting of the comments in the kernel-doc guide. Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- Documentation/doc-guide/kernel-doc.rst | 6 ++ 1 file changed, 6 insertions(+) diff --git a/Documentation/doc-guide/kernel-doc.rst b/Documentation/doc-guide/kernel-doc.rst index c6c3297..bb5ed6d 100644 --- a/Documentation/doc-guide/kernel-doc.rst +++ b/Documentation/doc-guide/kernel-doc.rst @@ -44,6 +44,12 @@ that somebody changing the code will also change the documentation. The overview kernel-doc comments may be placed anywhere at the top indentation level. +Running the ``kernel-doc`` tool with increased verbosity and without actual +output generation may be used to verify proper formating of the +documentation comments. For example:: + + scripts/kernel-doc -v -none drivers/foo/bar.c + Function documentation -- -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] scripts: kernel_doc: fixup reporting of function identifiers
When function description includes brackets after the function name as
suggested by Documentation/doc-guide/kernel-doc, the kernel-doc script
omits the function name from "Scanning doc for" report.
Extending match for identifier name with optional brackets fixes this
issue.
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
scripts/kernel-doc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index fee8952037b1..a6a9a8ef116c 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1873,7 +1873,7 @@ sub process_file($) {
}
elsif (/$doc_decl/o) {
$identifier = $1;
- if (/\s*([\w\s]+?)\s*-/) {
+ if (/\s*([\w\s]+?)(\(\))?\s*-/) {
$identifier = $1;
}
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] mm: docs: add blank lines to silence sphinx "Unexpected indentation" errors
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com> --- mm/pagewalk.c | 1 + mm/process_vm_access.c | 2 ++ mm/vmscan.c| 1 + 3 files changed, 4 insertions(+) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 23a3e415ac2c..8d2da5dec1e0 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -265,6 +265,7 @@ static int __walk_page_range(unsigned long start, unsigned long end, * pte_entry(), and/or hugetlb_entry(). If you don't set up for some of these * callbacks, the associated entries/pages are just ignored. * The return values of these callbacks are commonly defined like below: + * * - 0 : succeeded to handle the current entry, and if you don't reach the * end address yet, continue to walk. * - >0 : succeeded to handle the current entry, and return to the caller diff --git a/mm/process_vm_access.c b/mm/process_vm_access.c index 011edefd3c92..1a27c837f004 100644 --- a/mm/process_vm_access.c +++ b/mm/process_vm_access.c @@ -147,6 +147,7 @@ static int process_vm_rw_single_vec(unsigned long addr, * @riovcnt: size of rvec array * @flags: currently unused * @vm_write: 0 if reading from other process, 1 if writing to other process + * * Returns the number of bytes read/written or error code. May * return less bytes than expected if an error occurs during the copying * process. @@ -253,6 +254,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter, * @riovcnt: size of rvec array * @flags: currently unused * @vm_write: 0 if reading from other process, 1 if writing to other process + * * Returns the number of bytes read/written or error code. May * return less bytes than expected if an error occurs during the copying * process. diff --git a/mm/vmscan.c b/mm/vmscan.c index 47d5ced51f2d..8d01b095d97b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1606,6 +1606,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, * found will be decremented. * * Restrictions: + * * (1) Must be called with an elevated refcount on the page. This is a * fundamentnal difference from isolate_lru_pages (which is called * without a stable reference). -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] mm: docs: fixup punctuation
so that kernel-doc will properly recognize the parameter and function
descriptions.
Signed-off-by: Mike Rapoport <r...@linux.vnet.ibm.com>
---
mm/ksm.c| 2 +-
mm/memcontrol.c | 4 ++--
mm/mlock.c | 2 +-
mm/nommu.c | 2 +-
mm/sparse-vmemmap.c | 4 ++--
mm/zpool.c | 44 ++--
6 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index be8f4576f842..64207d936659 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -2309,7 +2309,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct
page **page)
/**
* ksm_do_scan - the ksm scanner main worker function.
- * @scan_npages - number of pages we want to scan before we return.
+ * @scan_npages: number of pages we want to scan before we return.
*/
static void ksm_do_scan(unsigned int scan_npages)
{
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index ac2ffd5e02b9..cddea3ed8e86 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5885,8 +5885,8 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg,
unsigned int nr_pages)
/**
* mem_cgroup_uncharge_skmem - uncharge socket memory
- * @memcg - memcg to uncharge
- * @nr_pages - number of pages to uncharge
+ * @memcg: memcg to uncharge
+ * @nr_pages: number of pages to uncharge
*/
void mem_cgroup_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages)
{
diff --git a/mm/mlock.c b/mm/mlock.c
index 30472d438794..3d2e834a6cb7 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -157,7 +157,7 @@ static void __munlock_isolation_failed(struct page *page)
/**
* munlock_vma_page - munlock a vma page
- * @page - page to be unlocked, either a normal page or THP page head
+ * @page: page to be unlocked, either a normal page or THP page head
*
* returns the size of the page as a page mask (0 for normal page,
* HPAGE_PMD_NR - 1 for THP head page)
diff --git a/mm/nommu.c b/mm/nommu.c
index 17c00d93de2e..52c14127a861 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1843,7 +1843,7 @@ int __access_remote_vm(struct task_struct *tsk, struct
mm_struct *mm,
}
/**
- * @access_remote_vm - access another process' address space
+ * access_remote_vm - access another process' address space
* @mm:the mm_struct of the target address space
* @addr: start address to access
* @buf: source or destination buffer
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 17acf01791fa..015ee4eb79bc 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -108,8 +108,8 @@ static unsigned long __meminit vmem_altmap_nr_free(struct
vmem_altmap *altmap)
/**
* vmem_altmap_alloc - allocate pages from the vmem_altmap reservation
- * @altmap - reserved page pool for the allocation
- * @nr_pfns - size (in pages) of the allocation
+ * @altmap: reserved page pool for the allocation
+ * @nr_pfns: size (in pages) of the allocation
*
* Allocations are aligned to the size of the request
*/
diff --git a/mm/zpool.c b/mm/zpool.c
index fd3ff719c32c..2e6f9c3cebe7 100644
--- a/mm/zpool.c
+++ b/mm/zpool.c
@@ -100,7 +100,7 @@ static void zpool_put_driver(struct zpool_driver *driver)
/**
* zpool_has_pool() - Check if the pool driver is available
- * @type The type of the zpool to check (e.g. zbud, zsmalloc)
+ * @type: The type of the zpool to check (e.g. zbud, zsmalloc)
*
* This checks if the @type pool driver is available. This will try to load
* the requested module, if needed, but there is no guarantee the module will
@@ -135,10 +135,10 @@ EXPORT_SYMBOL(zpool_has_pool);
/**
* zpool_create_pool() - Create a new zpool
- * @type The type of the zpool to create (e.g. zbud, zsmalloc)
- * @name The name of the zpool (e.g. zram0, zswap)
- * @gfpThe GFP flags to use when allocating the pool.
- * @opsThe optional ops callback.
+ * @type: The type of the zpool to create (e.g. zbud, zsmalloc)
+ * @name: The name of the zpool (e.g. zram0, zswap)
+ * @gfp: The GFP flags to use when allocating the pool.
+ * @ops: The optional ops callback.
*
* This creates a new zpool of the specified type. The gfp flags will be
* used when allocating memory, if the implementation supports it. If the
@@ -199,7 +199,7 @@ struct zpool *zpool_create_pool(const char *type, const
char *name, gfp_t gfp,
/**
* zpool_destroy_pool() - Destroy a zpool
- * @pool The zpool to destroy.
+ * @pool: The zpool to destroy.
*
* Implementations must guarantee this to be thread-safe,
* however only when destroying different pools. The same
@@ -222,7 +222,7 @@ void zpool_destroy_pool(struct zpool *zpool)
/**
* zpool_get_type() - Get the type of the zpool
- * @pool The zpool to check
+ * @pool: The zpool to check
*
* This returns the type of the pool.
*
@@ -237,10 +237,10 @@ const char *zpool_get_type(struct zpool *zpool)
/**
* zpool_malloc() - Allocate memory
- *
[PATCH 0/3] mm: docs: trivial fixes
Hi, These are some trvial fixes to the kernel-doc descriptions. Mike Rapoport (3): mm: docs: fixup punctuation mm: docs: fix parameter names mismatch mm: docs: add blank lines to silence sphinx "Unexpected indentation" errors mm/bootmem.c | 2 +- mm/ksm.c | 2 +- mm/maccess.c | 2 +- mm/memcontrol.c| 6 +++--- mm/mlock.c | 2 +- mm/nommu.c | 2 +- mm/pagewalk.c | 1 + mm/process_vm_access.c | 4 +++- mm/sparse-vmemmap.c| 4 ++-- mm/swap.c | 4 ++-- mm/vmscan.c| 1 + mm/z3fold.c| 4 ++-- mm/zbud.c | 4 ++-- mm/zpool.c | 46 +++--- 14 files changed, 44 insertions(+), 40 deletions(-) -- 2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
