Re: [PATCH 1/5] mm: reuse only-pte-mapped KSM page in do_wp_page()
On Thu, Apr 1, 2021 at 12:38 PM Greg KH wrote: > > On Thu, Apr 01, 2021 at 11:17:37AM -0700, Suren Baghdasaryan wrote: > > From: Kirill Tkhai > > > > Add an optimization for KSM pages almost in the same way that we have > > for ordinary anonymous pages. If there is a write fault in a page, > > which is mapped to an only pte, and it is not related to swap cache; the > > page may be reused without copying its content. > > > > [ Note that we do not consider PageSwapCache() pages at least for now, > > since we don't want to complicate __get_ksm_page(), which has nice > > optimization based on this (for the migration case). Currenly it is > > spinning on PageSwapCache() pages, waiting for when they have > > unfreezed counters (i.e., for the migration finish). But we don't want > > to make it also spinning on swap cache pages, which we try to reuse, > > since there is not a very high probability to reuse them. So, for now > > we do not consider PageSwapCache() pages at all. ] > > > > So in reuse_ksm_page() we check for 1) PageSwapCache() and 2) > > page_stable_node(), to skip a page, which KSM is currently trying to > > link to stable tree. Then we do page_ref_freeze() to prohibit KSM to > > merge one more page into the page, we are reusing. After that, nobody > > can refer to the reusing page: KSM skips !PageSwapCache() pages with > > zero refcount; and the protection against of all other participants is > > the same as for reused ordinary anon pages pte lock, page lock and > > mmap_sem. > > > > [a...@linux-foundation.org: replace BUG_ON()s with WARN_ON()s] > > Link: > > http://lkml.kernel.org/r/154471491016.31352.1168978849911555609.stgit@localhost.localdomain > > Signed-off-by: Kirill Tkhai > > Reviewed-by: Yang Shi > > Cc: "Kirill A. Shutemov" > > Cc: Hugh Dickins > > Cc: Andrea Arcangeli > > Cc: Christian Koenig > > Cc: Claudio Imbrenda > > Cc: Rik van Riel > > Cc: Huang Ying > > Cc: Minchan Kim > > Cc: Kirill Tkhai > > Signed-off-by: Andrew Morton > > Signed-off-by: Linus Torvalds > > --- > > include/linux/ksm.h | 7 +++ > > mm/ksm.c| 30 -- > > mm/memory.c | 16 ++-- > > 3 files changed, 49 insertions(+), 4 deletions(-) > > You forgot to put the git commit id of the upstream commit in here > somewhere so we can properly reference it and track it. > > When/if you resend this, please add it to all of the commits. Will do. Thanks! > > thanks, > > greg k-h
Re: [PATCH 1/5] mm: reuse only-pte-mapped KSM page in do_wp_page()
On Thu, Apr 01, 2021 at 11:17:37AM -0700, Suren Baghdasaryan wrote: > From: Kirill Tkhai > > Add an optimization for KSM pages almost in the same way that we have > for ordinary anonymous pages. If there is a write fault in a page, > which is mapped to an only pte, and it is not related to swap cache; the > page may be reused without copying its content. > > [ Note that we do not consider PageSwapCache() pages at least for now, > since we don't want to complicate __get_ksm_page(), which has nice > optimization based on this (for the migration case). Currenly it is > spinning on PageSwapCache() pages, waiting for when they have > unfreezed counters (i.e., for the migration finish). But we don't want > to make it also spinning on swap cache pages, which we try to reuse, > since there is not a very high probability to reuse them. So, for now > we do not consider PageSwapCache() pages at all. ] > > So in reuse_ksm_page() we check for 1) PageSwapCache() and 2) > page_stable_node(), to skip a page, which KSM is currently trying to > link to stable tree. Then we do page_ref_freeze() to prohibit KSM to > merge one more page into the page, we are reusing. After that, nobody > can refer to the reusing page: KSM skips !PageSwapCache() pages with > zero refcount; and the protection against of all other participants is > the same as for reused ordinary anon pages pte lock, page lock and > mmap_sem. > > [a...@linux-foundation.org: replace BUG_ON()s with WARN_ON()s] > Link: > http://lkml.kernel.org/r/154471491016.31352.1168978849911555609.stgit@localhost.localdomain > Signed-off-by: Kirill Tkhai > Reviewed-by: Yang Shi > Cc: "Kirill A. Shutemov" > Cc: Hugh Dickins > Cc: Andrea Arcangeli > Cc: Christian Koenig > Cc: Claudio Imbrenda > Cc: Rik van Riel > Cc: Huang Ying > Cc: Minchan Kim > Cc: Kirill Tkhai > Signed-off-by: Andrew Morton > Signed-off-by: Linus Torvalds > --- > include/linux/ksm.h | 7 +++ > mm/ksm.c| 30 -- > mm/memory.c | 16 ++-- > 3 files changed, 49 insertions(+), 4 deletions(-) You forgot to put the git commit id of the upstream commit in here somewhere so we can properly reference it and track it. When/if you resend this, please add it to all of the commits. thanks, greg k-h
[PATCH 1/5] mm: reuse only-pte-mapped KSM page in do_wp_page()
From: Kirill Tkhai
Add an optimization for KSM pages almost in the same way that we have
for ordinary anonymous pages. If there is a write fault in a page,
which is mapped to an only pte, and it is not related to swap cache; the
page may be reused without copying its content.
[ Note that we do not consider PageSwapCache() pages at least for now,
since we don't want to complicate __get_ksm_page(), which has nice
optimization based on this (for the migration case). Currenly it is
spinning on PageSwapCache() pages, waiting for when they have
unfreezed counters (i.e., for the migration finish). But we don't want
to make it also spinning on swap cache pages, which we try to reuse,
since there is not a very high probability to reuse them. So, for now
we do not consider PageSwapCache() pages at all. ]
So in reuse_ksm_page() we check for 1) PageSwapCache() and 2)
page_stable_node(), to skip a page, which KSM is currently trying to
link to stable tree. Then we do page_ref_freeze() to prohibit KSM to
merge one more page into the page, we are reusing. After that, nobody
can refer to the reusing page: KSM skips !PageSwapCache() pages with
zero refcount; and the protection against of all other participants is
the same as for reused ordinary anon pages pte lock, page lock and
mmap_sem.
[a...@linux-foundation.org: replace BUG_ON()s with WARN_ON()s]
Link:
http://lkml.kernel.org/r/154471491016.31352.1168978849911555609.stgit@localhost.localdomain
Signed-off-by: Kirill Tkhai
Reviewed-by: Yang Shi
Cc: "Kirill A. Shutemov"
Cc: Hugh Dickins
Cc: Andrea Arcangeli
Cc: Christian Koenig
Cc: Claudio Imbrenda
Cc: Rik van Riel
Cc: Huang Ying
Cc: Minchan Kim
Cc: Kirill Tkhai
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
---
include/linux/ksm.h | 7 +++
mm/ksm.c| 30 --
mm/memory.c | 16 ++--
3 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/include/linux/ksm.h b/include/linux/ksm.h
index 161e8164abcf..e48b1e453ff5 100644
--- a/include/linux/ksm.h
+++ b/include/linux/ksm.h
@@ -53,6 +53,8 @@ struct page *ksm_might_need_to_copy(struct page *page,
void rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc);
void ksm_migrate_page(struct page *newpage, struct page *oldpage);
+bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma, unsigned long address);
#else /* !CONFIG_KSM */
@@ -86,6 +88,11 @@ static inline void rmap_walk_ksm(struct page *page,
static inline void ksm_migrate_page(struct page *newpage, struct page *oldpage)
{
}
+static inline bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma, unsigned long address)
+{
+ return false;
+}
#endif /* CONFIG_MMU */
#endif /* !CONFIG_KSM */
diff --git a/mm/ksm.c b/mm/ksm.c
index d021bcf94c41..c4e95ca65d62 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -705,8 +705,9 @@ static struct page *get_ksm_page(struct stable_node
*stable_node, bool lock_it)
* case this node is no longer referenced, and should be freed;
* however, it might mean that the page is under page_ref_freeze().
* The __remove_mapping() case is easy, again the node is now stale;
-* but if page is swapcache in migrate_page_move_mapping(), it might
-* still be our page, in which case it's essential to keep the node.
+* the same is in reuse_ksm_page() case; but if page is swapcache
+* in migrate_page_move_mapping(), it might still be our page,
+* in which case it's essential to keep the node.
*/
while (!get_page_unless_zero(page)) {
/*
@@ -2648,6 +2649,31 @@ void rmap_walk_ksm(struct page *page, struct
rmap_walk_control *rwc)
goto again;
}
+bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma,
+ unsigned long address)
+{
+#ifdef CONFIG_DEBUG_VM
+ if (WARN_ON(is_zero_pfn(page_to_pfn(page))) ||
+ WARN_ON(!page_mapped(page)) ||
+ WARN_ON(!PageLocked(page))) {
+ dump_page(page, "reuse_ksm_page");
+ return false;
+ }
+#endif
+
+ if (PageSwapCache(page) || !page_stable_node(page))
+ return false;
+ /* Prohibit parallel get_ksm_page() */
+ if (!page_ref_freeze(page, 1))
+ return false;
+
+ page_move_anon_rmap(page, vma);
+ page->index = linear_page_index(vma, address);
+ page_ref_unfreeze(page, 1);
+
+ return true;
+}
#ifdef CONFIG_MIGRATION
void ksm_migrate_page(struct page *newpage, struct page *oldpage)
{
diff --git a/mm/memory.c b/mm/memory.c
index c1a05c2484b0..3874acce1472 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2846,8 +2846,11 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
* Take out anonymous pages first, anonymous shared vmas are
* not dirty accountable.
[PATCH 1/5] mm: reuse only-pte-mapped KSM page in do_wp_page()
From: Kirill Tkhai
Add an optimization for KSM pages almost in the same way that we have
for ordinary anonymous pages. If there is a write fault in a page,
which is mapped to an only pte, and it is not related to swap cache; the
page may be reused without copying its content.
[ Note that we do not consider PageSwapCache() pages at least for now,
since we don't want to complicate __get_ksm_page(), which has nice
optimization based on this (for the migration case). Currenly it is
spinning on PageSwapCache() pages, waiting for when they have
unfreezed counters (i.e., for the migration finish). But we don't want
to make it also spinning on swap cache pages, which we try to reuse,
since there is not a very high probability to reuse them. So, for now
we do not consider PageSwapCache() pages at all. ]
So in reuse_ksm_page() we check for 1) PageSwapCache() and 2)
page_stable_node(), to skip a page, which KSM is currently trying to
link to stable tree. Then we do page_ref_freeze() to prohibit KSM to
merge one more page into the page, we are reusing. After that, nobody
can refer to the reusing page: KSM skips !PageSwapCache() pages with
zero refcount; and the protection against of all other participants is
the same as for reused ordinary anon pages pte lock, page lock and
mmap_sem.
[a...@linux-foundation.org: replace BUG_ON()s with WARN_ON()s]
Link:
http://lkml.kernel.org/r/154471491016.31352.1168978849911555609.stgit@localhost.localdomain
Signed-off-by: Kirill Tkhai
Reviewed-by: Yang Shi
Cc: "Kirill A. Shutemov"
Cc: Hugh Dickins
Cc: Andrea Arcangeli
Cc: Christian Koenig
Cc: Claudio Imbrenda
Cc: Rik van Riel
Cc: Huang Ying
Cc: Minchan Kim
Cc: Kirill Tkhai
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
---
include/linux/ksm.h | 7 +++
mm/ksm.c| 30 --
mm/memory.c | 16 ++--
3 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/include/linux/ksm.h b/include/linux/ksm.h
index 44368b19b27e..def48a2d87aa 100644
--- a/include/linux/ksm.h
+++ b/include/linux/ksm.h
@@ -64,6 +64,8 @@ struct page *ksm_might_need_to_copy(struct page *page,
void rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc);
void ksm_migrate_page(struct page *newpage, struct page *oldpage);
+bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma, unsigned long address);
#else /* !CONFIG_KSM */
@@ -103,6 +105,11 @@ static inline void rmap_walk_ksm(struct page *page,
static inline void ksm_migrate_page(struct page *newpage, struct page *oldpage)
{
}
+static inline bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma, unsigned long address)
+{
+ return false;
+}
#endif /* CONFIG_MMU */
#endif /* !CONFIG_KSM */
diff --git a/mm/ksm.c b/mm/ksm.c
index 65d4bf52f543..62419735ee9c 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -695,8 +695,9 @@ static struct page *get_ksm_page(struct stable_node
*stable_node, bool lock_it)
* case this node is no longer referenced, and should be freed;
* however, it might mean that the page is under page_freeze_refs().
* The __remove_mapping() case is easy, again the node is now stale;
-* but if page is swapcache in migrate_page_move_mapping(), it might
-* still be our page, in which case it's essential to keep the node.
+* the same is in reuse_ksm_page() case; but if page is swapcache
+* in migrate_page_move_mapping(), it might still be our page,
+* in which case it's essential to keep the node.
*/
while (!get_page_unless_zero(page)) {
/*
@@ -2609,6 +2610,31 @@ void rmap_walk_ksm(struct page *page, struct
rmap_walk_control *rwc)
goto again;
}
+bool reuse_ksm_page(struct page *page,
+ struct vm_area_struct *vma,
+ unsigned long address)
+{
+#ifdef CONFIG_DEBUG_VM
+ if (WARN_ON(is_zero_pfn(page_to_pfn(page))) ||
+ WARN_ON(!page_mapped(page)) ||
+ WARN_ON(!PageLocked(page))) {
+ dump_page(page, "reuse_ksm_page");
+ return false;
+ }
+#endif
+
+ if (PageSwapCache(page) || !page_stable_node(page))
+ return false;
+ /* Prohibit parallel get_ksm_page() */
+ if (!page_ref_freeze(page, 1))
+ return false;
+
+ page_move_anon_rmap(page, vma);
+ page->index = linear_page_index(vma, address);
+ page_ref_unfreeze(page, 1);
+
+ return true;
+}
#ifdef CONFIG_MIGRATION
void ksm_migrate_page(struct page *newpage, struct page *oldpage)
{
diff --git a/mm/memory.c b/mm/memory.c
index 21a0bbb9c21f..6920bfb3f89c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2831,8 +2831,11 @@ static int do_wp_page(struct vm_fault *vmf)
* Take out anonymous pages first, anonymous shared vmas are
* not dirty accountable.
