Re: [PATCH v2 21/34] arm64: Convert various functions to use ptdescs
Hi Vishal,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master next-20230428]
[cannot apply to s390/features powerpc/next powerpc/fixes geert-m68k/for-next
geert-m68k/for-linus v6.3]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url:
https://github.com/intel-lab-lkp/linux/commits/Vishal-Moola-Oracle/mm-Add-PAGE_TYPE_OP-folio-functions/20230502-033042
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
mm-everything
patch link:
https://lore.kernel.org/r/20230501192829.17086-22-vishal.moola%40gmail.com
patch subject: [PATCH v2 21/34] arm64: Convert various functions to use ptdescs
config: arm64-randconfig-r023-20230430
(https://download.01.org/0day-ci/archive/20230502/202305021038.c9jfvdsv-...@intel.com/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project
b1465cd49efcbc114a75220b153f5a055ce7911f)
reproduce (this is a W=1 build):
wget
https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O
~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
#
https://github.com/intel-lab-lkp/linux/commit/8e9481b63b5773d7c914836dcd7fbec2449902bc
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review
Vishal-Moola-Oracle/mm-Add-PAGE_TYPE_OP-folio-functions/20230502-033042
git checkout 8e9481b63b5773d7c914836dcd7fbec2449902bc
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1
O=build_dir ARCH=arm64 olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1
O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot
| Link:
https://lore.kernel.org/oe-kbuild-all/202305021038.c9jfvdsv-...@intel.com/
All errors (new ones prefixed by >>):
>> arch/arm64/mm/mmu.c:440:10: error: invalid argument type 'void' to unary
>> expression
BUG_ON(!ptdesc_pte_dtor(ptdesc));
^~~~
include/asm-generic/bug.h:71:45: note: expanded from macro 'BUG_ON'
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
^
include/linux/compiler.h:78:42: note: expanded from macro 'unlikely'
# define unlikely(x)__builtin_expect(!!(x), 0)
^
arch/arm64/mm/mmu.c:442:10: error: invalid argument type 'void' to unary
expression
BUG_ON(!ptdesc_pte_dtor(ptdesc));
^~~~
include/asm-generic/bug.h:71:45: note: expanded from macro 'BUG_ON'
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
^
include/linux/compiler.h:78:42: note: expanded from macro 'unlikely'
# define unlikely(x)__builtin_expect(!!(x), 0)
^
2 errors generated.
vim +/void +440 arch/arm64/mm/mmu.c
425
426 static phys_addr_t pgd_pgtable_alloc(int shift)
427 {
428 phys_addr_t pa = __pgd_pgtable_alloc(shift);
429 struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa));
430
431 /*
432 * Call proper page table ctor in case later we need to
433 * call core mm functions like apply_to_page_range() on
434 * this pre-allocated page table.
435 *
436 * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is
437 * folded, and if so ptdesc_pte_dtor() becomes nop.
438 */
439 if (shift == PAGE_SHIFT)
> 440 BUG_ON(!ptdesc_pte_dtor(ptdesc));
441 else if (shift == PMD_SHIFT)
442 BUG_ON(!ptdesc_pte_dtor(ptdesc));
443
444 return pa;
445 }
446
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
Re: [PATCH v2 21/34] arm64: Convert various functions to use ptdescs
Hi Vishal,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master next-20230428]
[cannot apply to s390/features powerpc/next powerpc/fixes geert-m68k/for-next
geert-m68k/for-linus v6.3]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url:
https://github.com/intel-lab-lkp/linux/commits/Vishal-Moola-Oracle/mm-Add-PAGE_TYPE_OP-folio-functions/20230502-033042
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
mm-everything
patch link:
https://lore.kernel.org/r/20230501192829.17086-22-vishal.moola%40gmail.com
patch subject: [PATCH v2 21/34] arm64: Convert various functions to use ptdescs
config: arm64-allyesconfig
(https://download.01.org/0day-ci/archive/20230502/202305020914.ogrwceg1-...@intel.com/config)
compiler: aarch64-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget
https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O
~/bin/make.cross
chmod +x ~/bin/make.cross
#
https://github.com/intel-lab-lkp/linux/commit/8e9481b63b5773d7c914836dcd7fbec2449902bc
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review
Vishal-Moola-Oracle/mm-Add-PAGE_TYPE_OP-folio-functions/20230502-033042
git checkout 8e9481b63b5773d7c914836dcd7fbec2449902bc
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1
O=build_dir ARCH=arm64 olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1
O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot
| Link:
https://lore.kernel.org/oe-kbuild-all/202305020914.ogrwceg1-...@intel.com/
All errors (new ones prefixed by >>):
In file included from include/linux/build_bug.h:5,
from include/linux/bits.h:21,
from include/linux/bitops.h:6,
from arch/arm64/include/asm/cache.h:39,
from include/linux/cache.h:6,
from arch/arm64/mm/mmu.c:9:
arch/arm64/mm/mmu.c: In function 'pgd_pgtable_alloc':
>> arch/arm64/mm/mmu.c:440:24: error: invalid use of void expression
440 | BUG_ON(!ptdesc_pte_dtor(ptdesc));
|^
include/linux/compiler.h:78:45: note: in definition of macro 'unlikely'
78 | # define unlikely(x)__builtin_expect(!!(x), 0)
| ^
arch/arm64/mm/mmu.c:440:17: note: in expansion of macro 'BUG_ON'
440 | BUG_ON(!ptdesc_pte_dtor(ptdesc));
| ^~
arch/arm64/mm/mmu.c:442:24: error: invalid use of void expression
442 | BUG_ON(!ptdesc_pte_dtor(ptdesc));
|^
include/linux/compiler.h:78:45: note: in definition of macro 'unlikely'
78 | # define unlikely(x)__builtin_expect(!!(x), 0)
| ^
arch/arm64/mm/mmu.c:442:17: note: in expansion of macro 'BUG_ON'
442 | BUG_ON(!ptdesc_pte_dtor(ptdesc));
| ^~
vim +440 arch/arm64/mm/mmu.c
425
426 static phys_addr_t pgd_pgtable_alloc(int shift)
427 {
428 phys_addr_t pa = __pgd_pgtable_alloc(shift);
429 struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa));
430
431 /*
432 * Call proper page table ctor in case later we need to
433 * call core mm functions like apply_to_page_range() on
434 * this pre-allocated page table.
435 *
436 * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is
437 * folded, and if so ptdesc_pte_dtor() becomes nop.
438 */
439 if (shift == PAGE_SHIFT)
> 440 BUG_ON(!ptdesc_pte_dtor(ptdesc));
441 else if (shift == PMD_SHIFT)
442 BUG_ON(!ptdesc_pte_dtor(ptdesc));
443
444 return pa;
445 }
446
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
Re: [PATCH v2 29/34] riscv: Convert alloc_{pmd, pte}_late() to use ptdescs
On Mon, 01 May 2023 12:28:24 PDT (-0700), vishal.mo...@gmail.com wrote:
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/riscv/include/asm/pgalloc.h | 8
arch/riscv/mm/init.c | 16 ++--
2 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
index 59dc12b5b7e8..cb5536403bd8 100644
--- a/arch/riscv/include/asm/pgalloc.h
+++ b/arch/riscv/include/asm/pgalloc.h
@@ -153,10 +153,10 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
#endif /* __PAGETABLE_PMD_FOLDED */
-#define __pte_free_tlb(tlb, pte, buf) \
-do {\
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), pte);\
+#define __pte_free_tlb(tlb, pte, buf) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
} while (0)
#endif /* CONFIG_MMU */
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index eb8173a91ce3..8f1982664687 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -353,12 +353,10 @@ static inline phys_addr_t __init
alloc_pte_fixmap(uintptr_t va)
static phys_addr_t __init alloc_pte_late(uintptr_t va)
{
- unsigned long vaddr;
-
- vaddr = __get_free_page(GFP_KERNEL);
- BUG_ON(!vaddr || !pgtable_pte_page_ctor(virt_to_page(vaddr)));
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- return __pa(vaddr);
+ BUG_ON(!ptdesc || !ptdesc_pte_ctor(ptdesc));
+ return __pa((pte_t *)ptdesc_address(ptdesc));
}
static void __init create_pte_mapping(pte_t *ptep,
@@ -436,12 +434,10 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va)
static phys_addr_t __init alloc_pmd_late(uintptr_t va)
{
- unsigned long vaddr;
-
- vaddr = __get_free_page(GFP_KERNEL);
- BUG_ON(!vaddr || !pgtable_pmd_page_ctor(virt_to_page(vaddr)));
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- return __pa(vaddr);
+ BUG_ON(!ptdesc || !ptdesc_pmd_ctor(ptdesc));
+ return __pa((pmd_t *)ptdesc_address(ptdesc));
}
static void __init create_pmd_mapping(pmd_t *pmdp,
Acked-by: Palmer Dabbelt
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
Hi Conor
On Mon, 1 May 2023 at 19:41, Conor Dooley wrote:
>
> Hey Ricardo,
>
> On Mon, May 01, 2023 at 02:38:22PM +0200, Ricardo Ribalda wrote:
> > If PGO is enabled, the purgatory ends up with multiple .text sections.
> > This is not supported by kexec and crashes the system.
> >
> > Cc: sta...@vger.kernel.org
> > Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
> > Signed-off-by: Ricardo Ribalda
> > ---
> > arch/riscv/purgatory/Makefile | 5 +
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
> > index 5730797a6b40..cf3a44121a90 100644
> > --- a/arch/riscv/purgatory/Makefile
> > +++ b/arch/riscv/purgatory/Makefile
> > @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> > CFLAGS_string.o := -D__DISABLE_EXPORTS
> > CFLAGS_ctype.o := -D__DISABLE_EXPORTS
> >
> > +# When profile optimization is enabled, llvm emits two different
> > overlapping
> > +# text sections, which is not supported by kexec. Remove profile
> > optimization
> > +# flags.
> > +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> > -fprofile-use=%,$(KBUILD_CFLAGS))
>
> With the caveat of not being au fait with the workings of either PGO or
> of purgatory, how come you modify KBUILD_CFLAGS here rather than the
> purgatory specific PURGATORY_CFLAGS that are used later in the file?
Definitely, not a Makefile expert here, but when I tried this:
@@ -35,6 +40,7 @@ PURGATORY_CFLAGS_REMOVE := -mcmodel=kernel
PURGATORY_CFLAGS := -mcmodel=large -ffreestanding
-fno-zero-initialized-in-bss -g0
PURGATORY_CFLAGS += $(DISABLE_STACKLEAK_PLUGIN) -DDISABLE_BRANCH_PROFILING
PURGATORY_CFLAGS += -fno-stack-protector
+PURGATORY_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))
It did not work.
Fixes: bde971a83bbf ("KVM: arm64: nvhe: Fix build with profile optimization")
does this approach, so this is what I tried and worked.
Thanks!
>
> Cheers,
> Conor.
>
> > +
> > # When linking purgatory.ro with -r unresolved symbols are not checked,
> > # also link a purgatory.chk binary without -r to check for unresolved
> > symbols.
> > PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
> >
> > --
> > 2.40.1.495.gc816e09b53d-goog
> >
> >
> > ___
> > linux-riscv mailing list
> > linux-ri...@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv
--
Ricardo Ribalda
[PATCH v2 34/34] mm: Remove pgtable_{pmd, pte}_page_{ctor, dtor}() wrappers
These functions are no longer necessary. Remove them and cleanup
Documentation referencing them.
Signed-off-by: Vishal Moola (Oracle)
---
Documentation/mm/split_page_table_lock.rst| 12 +--
.../zh_CN/mm/split_page_table_lock.rst| 14 ++---
include/linux/mm.h| 20 ---
3 files changed, 13 insertions(+), 33 deletions(-)
diff --git a/Documentation/mm/split_page_table_lock.rst
b/Documentation/mm/split_page_table_lock.rst
index 50ee0dfc95be..b3c612183135 100644
--- a/Documentation/mm/split_page_table_lock.rst
+++ b/Documentation/mm/split_page_table_lock.rst
@@ -53,7 +53,7 @@ Support of split page table lock by an architecture
===
There's no need in special enabling of PTE split page table lock: everything
-required is done by pgtable_pte_page_ctor() and pgtable_pte_page_dtor(), which
+required is done by ptdesc_pte_ctor() and ptdesc_pte_dtor(), which
must be called on PTE table allocation / freeing.
Make sure the architecture doesn't use slab allocator for page table
@@ -63,8 +63,8 @@ This field shares storage with page->ptl.
PMD split lock only makes sense if you have more than two page table
levels.
-PMD split lock enabling requires pgtable_pmd_page_ctor() call on PMD table
-allocation and pgtable_pmd_page_dtor() on freeing.
+PMD split lock enabling requires ptdesc_pmd_ctor() call on PMD table
+allocation and ptdesc_pmd_dtor() on freeing.
Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and
pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing
@@ -72,7 +72,7 @@ paths: i.e X86_PAE preallocate few PMDs on pgd_alloc().
With everything in place you can set CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK.
-NOTE: pgtable_pte_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must
+NOTE: ptdesc_pte_ctor() and ptdesc_pmd_ctor() can fail -- it must
be handled properly.
page->ptl
@@ -92,7 +92,7 @@ trick:
split lock with enabled DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC, but costs
one more cache line for indirect access;
-The spinlock_t allocated in pgtable_pte_page_ctor() for PTE table and in
-pgtable_pmd_page_ctor() for PMD table.
+The spinlock_t allocated in ptdesc_pte_ctor() for PTE table and in
+ptdesc_pmd_ctor() for PMD table.
Please, never access page->ptl directly -- use appropriate helper.
diff --git a/Documentation/translations/zh_CN/mm/split_page_table_lock.rst
b/Documentation/translations/zh_CN/mm/split_page_table_lock.rst
index 4fb7aa666037..a3323eb9dc40 100644
--- a/Documentation/translations/zh_CN/mm/split_page_table_lock.rst
+++ b/Documentation/translations/zh_CN/mm/split_page_table_lock.rst
@@ -56,16 +56,16 @@ Hugetlb特定的辅助函数:
架构对分页表锁的支持
-没有必要特别启用PTE分页表锁：所有需要的东西都由pgtable_pte_page_ctor()
-和pgtable_pte_page_dtor()完成，它们必须在PTE表分配/释放时被调用。
+没有必要特别启用PTE分页表锁：所有需要的东西都由ptdesc_pte_ctor()
+和ptdesc_pte_dtor()完成，它们必须在PTE表分配/释放时被调用。
确保架构不使用slab分配器来分配页表：slab使用page->slab_cache来分配其页
面。这个区域与page->ptl共享存储。
PMD分页锁只有在你有两个以上的页表级别时才有意义。
-启用PMD分页锁需要在PMD表分配时调用pgtable_pmd_page_ctor()，在释放时调
-用pgtable_pmd_page_dtor()。
+启用PMD分页锁需要在PMD表分配时调用ptdesc_pmd_ctor()，在释放时调
+用ptdesc_pmd_dtor()。
分配通常发生在pmd_alloc_one()中，释放发生在pmd_free()和pmd_free_tlb()
中，但要确保覆盖所有的PMD表分配/释放路径：即X86_PAE在pgd_alloc()中预先
@@ -73,7 +73,7 @@ PMD分页锁只有在你有两个以上的页表级别时才有意义。
一切就绪后，你可以设置CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK。
-注意：pgtable_pte_page_ctor()和pgtable_pmd_page_ctor()可能失败--必
+注意：ptdesc_pte_ctor()和ptdesc_pmd_ctor()可能失败--必
须正确处理。
page->ptl
@@ -90,7 +90,7 @@ page->ptl用于访问分割页表锁，其中'page'是包含该表的页面struc
的指针并动态分配它。这允许在启用DEBUG_SPINLOCK或DEBUG_LOCK_ALLOC的
情况下使用分页锁，但由于间接访问而多花了一个缓存行。
-PTE表的spinlock_t分配在pgtable_pte_page_ctor()中，PMD表的spinlock_t
-分配在pgtable_pmd_page_ctor()中。
+PTE表的spinlock_t分配在ptdesc_pte_ctor()中，PMD表的spinlock_t
+分配在ptdesc_pmd_ctor()中。
请不要直接访问page->ptl - -使用适当的辅助函数。
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dc61aeca9077..dfa3e202099a 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2858,11 +2858,6 @@ static inline bool ptdesc_pte_ctor(struct ptdesc *ptdesc)
return true;
}
-static inline bool pgtable_pte_page_ctor(struct page *page)
-{
- return ptdesc_pte_ctor(page_ptdesc(page));
-}
-
static inline void ptdesc_pte_dtor(struct ptdesc *ptdesc)
{
struct folio *folio = ptdesc_folio(ptdesc);
@@ -2872,11 +2867,6 @@ static inline void ptdesc_pte_dtor(struct ptdesc *ptdesc)
lruvec_stat_sub_folio(folio, NR_PAGETABLE);
}
-static inline void pgtable_pte_page_dtor(struct page *page)
-{
- ptdesc_pte_dtor(page_ptdesc(page));
-}
-
#define pte_offset_map_lock(mm, pmd, address, ptlp)\
({ \
spinlock_t *__ptl = pte_lockptr(mm, pmd); \
@@ -2967,11 +2957,6 @@ static inline bool ptdesc_pmd_ctor(struct ptdesc *ptdesc)
return true;
}
-static inline bool pgtable_pmd_page_ctor(struct page *page)
-{
-
[PATCH v2 33/34] um: Convert {pmd, pte}_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents. Also cleans up some spacing issues.
Signed-off-by: Vishal Moola (Oracle)
---
arch/um/include/asm/pgalloc.h | 18 +-
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
index 8ec7cd46dd96..760b029505c1 100644
--- a/arch/um/include/asm/pgalloc.h
+++ b/arch/um/include/asm/pgalloc.h
@@ -25,19 +25,19 @@
*/
extern pgd_t *pgd_alloc(struct mm_struct *);
-#define __pte_free_tlb(tlb,pte, address) \
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb),(pte)); \
+#define __pte_free_tlb(tlb, pte, address) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \
} while (0)
#ifdef CONFIG_3_LEVEL_PGTABLES
-#define __pmd_free_tlb(tlb, pmd, address) \
-do { \
- pgtable_pmd_page_dtor(virt_to_page(pmd)); \
- tlb_remove_page((tlb),virt_to_page(pmd)); \
-} while (0)\
+#define __pmd_free_tlb(tlb, pmd, address) \
+do { \
+ ptdesc_pmd_dtor(virt_to_ptdesc(pmd)); \
+ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \
+} while (0)
#endif
--
2.39.2
[PATCH v2 32/34] sparc: Convert pgtable_pte_page_{ctor, dtor}() to ptdesc equivalents
Part of the conversions to replace pgtable pte constructor/destructors with
ptdesc equivalents.
Signed-off-by: Vishal Moola (Oracle)
---
arch/sparc/mm/srmmu.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c
index 13f027afc875..964938aa7b88 100644
--- a/arch/sparc/mm/srmmu.c
+++ b/arch/sparc/mm/srmmu.c
@@ -355,7 +355,8 @@ pgtable_t pte_alloc_one(struct mm_struct *mm)
return NULL;
page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT);
spin_lock(>page_table_lock);
- if (page_ref_inc_return(page) == 2 && !pgtable_pte_page_ctor(page)) {
+ if (page_ref_inc_return(page) == 2 &&
+ !ptdesc_pte_ctor(page_ptdesc(page))) {
page_ref_dec(page);
ptep = NULL;
}
@@ -371,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep)
page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT);
spin_lock(>page_table_lock);
if (page_ref_dec_return(page) == 1)
- pgtable_pte_page_dtor(page);
+ ptdesc_pte_dtor(page_ptdesc(page));
spin_unlock(>page_table_lock);
srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE);
--
2.39.2
[PATCH v2 31/34] sparc64: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Signed-off-by: Vishal Moola (Oracle)
---
arch/sparc/mm/init_64.c | 17 +
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 04f9db0c3111..eedb3e03b1fe 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2893,14 +2893,15 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
pgtable_t pte_alloc_one(struct mm_struct *mm)
{
- struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO);
- if (!page)
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL | __GFP_ZERO, 0);
+
+ if (!ptdesc)
return NULL;
- if (!pgtable_pte_page_ctor(page)) {
- __free_page(page);
+ if (!ptdesc_pte_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- return (pte_t *) page_address(page);
+ return (pte_t *) ptdesc_address(ptdesc);
}
void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
@@ -2910,10 +2911,10 @@ void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
static void __pte_free(pgtable_t pte)
{
- struct page *page = virt_to_page(pte);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pte);
- pgtable_pte_page_dtor(page);
- __free_page(page);
+ ptdesc_pte_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
void pte_free(struct mm_struct *mm, pgtable_t pte)
--
2.39.2
[PATCH v2 29/34] riscv: Convert alloc_{pmd, pte}_late() to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/riscv/include/asm/pgalloc.h | 8
arch/riscv/mm/init.c | 16 ++--
2 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
index 59dc12b5b7e8..cb5536403bd8 100644
--- a/arch/riscv/include/asm/pgalloc.h
+++ b/arch/riscv/include/asm/pgalloc.h
@@ -153,10 +153,10 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
#endif /* __PAGETABLE_PMD_FOLDED */
-#define __pte_free_tlb(tlb, pte, buf) \
-do {\
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), pte);\
+#define __pte_free_tlb(tlb, pte, buf) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
} while (0)
#endif /* CONFIG_MMU */
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index eb8173a91ce3..8f1982664687 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -353,12 +353,10 @@ static inline phys_addr_t __init
alloc_pte_fixmap(uintptr_t va)
static phys_addr_t __init alloc_pte_late(uintptr_t va)
{
- unsigned long vaddr;
-
- vaddr = __get_free_page(GFP_KERNEL);
- BUG_ON(!vaddr || !pgtable_pte_page_ctor(virt_to_page(vaddr)));
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- return __pa(vaddr);
+ BUG_ON(!ptdesc || !ptdesc_pte_ctor(ptdesc));
+ return __pa((pte_t *)ptdesc_address(ptdesc));
}
static void __init create_pte_mapping(pte_t *ptep,
@@ -436,12 +434,10 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va)
static phys_addr_t __init alloc_pmd_late(uintptr_t va)
{
- unsigned long vaddr;
-
- vaddr = __get_free_page(GFP_KERNEL);
- BUG_ON(!vaddr || !pgtable_pmd_page_ctor(virt_to_page(vaddr)));
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- return __pa(vaddr);
+ BUG_ON(!ptdesc || !ptdesc_pmd_ctor(ptdesc));
+ return __pa((pmd_t *)ptdesc_address(ptdesc));
}
static void __init create_pmd_mapping(pmd_t *pmdp,
--
2.39.2
[PATCH v2 30/34] sh: Convert pte_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents. Also cleans up some spacing issues.
Signed-off-by: Vishal Moola (Oracle)
---
arch/sh/include/asm/pgalloc.h | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h
index a9e98233c4d4..ce2ba99dbd84 100644
--- a/arch/sh/include/asm/pgalloc.h
+++ b/arch/sh/include/asm/pgalloc.h
@@ -2,6 +2,7 @@
#ifndef __ASM_SH_PGALLOC_H
#define __ASM_SH_PGALLOC_H
+#include
#include
#define __HAVE_ARCH_PMD_ALLOC_ONE
@@ -31,10 +32,10 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t
*pmd,
set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
}
-#define __pte_free_tlb(tlb,pte,addr) \
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), (pte)); \
+#define __pte_free_tlb(tlb, pte, addr) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \
} while (0)
#endif /* __ASM_SH_PGALLOC_H */
--
2.39.2
[PATCH v2 28/34] openrisc: Convert __pte_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents.
Signed-off-by: Vishal Moola (Oracle)
---
arch/openrisc/include/asm/pgalloc.h | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/openrisc/include/asm/pgalloc.h
b/arch/openrisc/include/asm/pgalloc.h
index b7b2b8d16fad..14e641686281 100644
--- a/arch/openrisc/include/asm/pgalloc.h
+++ b/arch/openrisc/include/asm/pgalloc.h
@@ -66,10 +66,10 @@ extern inline pgd_t *pgd_alloc(struct mm_struct *mm)
extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
-#define __pte_free_tlb(tlb, pte, addr) \
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), (pte)); \
+#define __pte_free_tlb(tlb, pte, addr) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \
} while (0)
#endif
--
2.39.2
[PATCH v2 27/34] nios2: Convert __pte_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents.
Signed-off-by: Vishal Moola (Oracle)
---
arch/nios2/include/asm/pgalloc.h | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h
index ecd1657bb2ce..ed868f4c0ca9 100644
--- a/arch/nios2/include/asm/pgalloc.h
+++ b/arch/nios2/include/asm/pgalloc.h
@@ -28,10 +28,10 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t
*pmd,
extern pgd_t *pgd_alloc(struct mm_struct *mm);
-#define __pte_free_tlb(tlb, pte, addr) \
- do {\
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), (pte)); \
+#define __pte_free_tlb(tlb, pte, addr) \
+ do {\
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \
} while (0)
#endif /* _ASM_NIOS2_PGALLOC_H */
--
2.39.2
[PATCH v2 26/34] mips: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/mips/include/asm/pgalloc.h | 31 +--
arch/mips/mm/pgtable.c | 7 ---
2 files changed, 21 insertions(+), 17 deletions(-)
diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h
index f72e737dda21..7f7cc3140b27 100644
--- a/arch/mips/include/asm/pgalloc.h
+++ b/arch/mips/include/asm/pgalloc.h
@@ -51,13 +51,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
- free_pages((unsigned long)pgd, PGD_TABLE_ORDER);
+ ptdesc_free(virt_to_ptdesc(pgd));
}
-#define __pte_free_tlb(tlb,pte,address)\
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), pte);\
+#define __pte_free_tlb(tlb, pte, address) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
} while (0)
#ifndef __PAGETABLE_PMD_FOLDED
@@ -65,18 +65,18 @@ do {
\
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
{
pmd_t *pmd;
- struct page *pg;
+ struct ptdesc *ptdesc;
- pg = alloc_pages(GFP_KERNEL_ACCOUNT, PMD_TABLE_ORDER);
- if (!pg)
+ ptdesc = ptdesc_alloc(GFP_KERNEL_ACCOUNT, PMD_TABLE_ORDER);
+ if (!ptdesc)
return NULL;
- if (!pgtable_pmd_page_ctor(pg)) {
- __free_pages(pg, PMD_TABLE_ORDER);
+ if (!ptdesc_pmd_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- pmd = (pmd_t *)page_address(pg);
+ pmd = (pmd_t *)ptdesc_address(ptdesc);
pmd_init(pmd);
return pmd;
}
@@ -90,10 +90,13 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm,
unsigned long address)
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address)
{
pud_t *pud;
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, PUD_TABLE_ORDER);
- pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_TABLE_ORDER);
- if (pud)
- pud_init(pud);
+ if (!ptdesc)
+ return NULL;
+ pud = (pud_t *)ptdesc_address(ptdesc);
+
+ pud_init(pud);
return pud;
}
diff --git a/arch/mips/mm/pgtable.c b/arch/mips/mm/pgtable.c
index b13314be5d0e..d626db9ac224 100644
--- a/arch/mips/mm/pgtable.c
+++ b/arch/mips/mm/pgtable.c
@@ -10,10 +10,11 @@
pgd_t *pgd_alloc(struct mm_struct *mm)
{
- pgd_t *ret, *init;
+ pgd_t *init, *ret = NULL;
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, PGD_TABLE_ORDER);
- ret = (pgd_t *) __get_free_pages(GFP_KERNEL, PGD_TABLE_ORDER);
- if (ret) {
+ if (ptdesc) {
+ ret = (pgd_t *) ptdesc_address(ptdesc);
init = pgd_offset(_mm, 0UL);
pgd_init(ret);
memcpy(ret + USER_PTRS_PER_PGD, init + USER_PTRS_PER_PGD,
--
2.39.2
[PATCH v2 25/34] m68k: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/m68k/include/asm/mcf_pgalloc.h | 41 ++--
arch/m68k/include/asm/sun3_pgalloc.h | 8 +++---
arch/m68k/mm/motorola.c | 4 +--
3 files changed, 27 insertions(+), 26 deletions(-)
diff --git a/arch/m68k/include/asm/mcf_pgalloc.h
b/arch/m68k/include/asm/mcf_pgalloc.h
index 5c2c0a864524..b0e909e23e14 100644
--- a/arch/m68k/include/asm/mcf_pgalloc.h
+++ b/arch/m68k/include/asm/mcf_pgalloc.h
@@ -7,20 +7,19 @@
extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
- free_page((unsigned long) pte);
+ ptdesc_free(virt_to_ptdesc(pte));
}
extern const char bad_pmd_string[];
extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
- unsigned long page = __get_free_page(GFP_DMA);
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_DMA | __GFP_ZERO, 0);
- if (!page)
+ if (!ptdesc)
return NULL;
- memset((void *)page, 0, PAGE_SIZE);
- return (pte_t *) (page);
+ return (pte_t *) (ptdesc_address(ptdesc));
}
extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)
@@ -35,36 +34,36 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned
long address)
static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable,
unsigned long address)
{
- struct page *page = virt_to_page(pgtable);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pgtable);
- pgtable_pte_page_dtor(page);
- __free_page(page);
+ ptdesc_pte_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
- struct page *page = alloc_pages(GFP_DMA, 0);
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_DMA, 0);
pte_t *pte;
- if (!page)
+ if (!ptdesc)
return NULL;
- if (!pgtable_pte_page_ctor(page)) {
- __free_page(page);
+ if (!ptdesc_pte_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- pte = page_address(page);
- clear_page(pte);
+ pte = ptdesc_address(ptdesc);
+ ptdesc_clear(pte);
return pte;
}
static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable)
{
- struct page *page = virt_to_page(pgtable);
+ struct ptdesc *ptdesc = virt_to_ptdesc(ptdesc);
- pgtable_pte_page_dtor(page);
- __free_page(page);
+ ptdesc_pte_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
/*
@@ -75,16 +74,18 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t
pgtable)
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
- free_page((unsigned long) pgd);
+ ptdesc_free(virt_to_ptdesc(pgd));
}
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
pgd_t *new_pgd;
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_DMA | GFP_NOWARN, 0);
- new_pgd = (pgd_t *)__get_free_page(GFP_DMA | __GFP_NOWARN);
- if (!new_pgd)
+ if (!ptdesc)
return NULL;
+ new_pgd = (pgd_t *) ptdesc_address(ptdesc);
+
memcpy(new_pgd, swapper_pg_dir, PTRS_PER_PGD * sizeof(pgd_t));
memset(new_pgd, 0, PAGE_OFFSET >> PGDIR_SHIFT);
return new_pgd;
diff --git a/arch/m68k/include/asm/sun3_pgalloc.h
b/arch/m68k/include/asm/sun3_pgalloc.h
index 198036aff519..013d375fc239 100644
--- a/arch/m68k/include/asm/sun3_pgalloc.h
+++ b/arch/m68k/include/asm/sun3_pgalloc.h
@@ -17,10 +17,10 @@
extern const char bad_pmd_string[];
-#define __pte_free_tlb(tlb,pte,addr) \
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), pte);\
+#define __pte_free_tlb(tlb, pte, addr) \
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
} while (0)
static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t
*pte)
diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c
index 911301224078..f7adb86b37fb 100644
--- a/arch/m68k/mm/motorola.c
+++ b/arch/m68k/mm/motorola.c
@@ -161,7 +161,7 @@ void *get_pointer_table(int type)
* m68k doesn't have SPLIT_PTE_PTLOCKS for not having
* SMP.
*/
- pgtable_pte_page_ctor(virt_to_page(page));
+ ptdesc_pte_ctor(virt_to_ptdesc(page));
[PATCH v2 24/34] loongarch: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/loongarch/include/asm/pgalloc.h | 27 +++
arch/loongarch/mm/pgtable.c | 7 ---
2 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/arch/loongarch/include/asm/pgalloc.h
b/arch/loongarch/include/asm/pgalloc.h
index af1d1e4a6965..1fe074f85b6b 100644
--- a/arch/loongarch/include/asm/pgalloc.h
+++ b/arch/loongarch/include/asm/pgalloc.h
@@ -45,9 +45,9 @@ extern void pagetable_init(void);
extern pgd_t *pgd_alloc(struct mm_struct *mm);
#define __pte_free_tlb(tlb, pte, address) \
-do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page((tlb), pte);\
+do { \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
} while (0)
#ifndef __PAGETABLE_PMD_FOLDED
@@ -55,18 +55,18 @@ do {
\
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
{
pmd_t *pmd;
- struct page *pg;
+ struct ptdesc *ptdesc;
- pg = alloc_page(GFP_KERNEL_ACCOUNT);
- if (!pg)
+ ptdesc = ptdesc_alloc(GFP_KERNEL_ACCOUNT, 0);
+ if (!ptdesc)
return NULL;
- if (!pgtable_pmd_page_ctor(pg)) {
- __free_page(pg);
+ if (!ptdesc_pmd_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- pmd = (pmd_t *)page_address(pg);
+ pmd = (pmd_t *)ptdesc_address(ptdesc);
pmd_init(pmd);
return pmd;
}
@@ -80,10 +80,13 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm,
unsigned long address)
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address)
{
pud_t *pud;
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- pud = (pud_t *) __get_free_page(GFP_KERNEL);
- if (pud)
- pud_init(pud);
+ if (!ptdesc)
+ return NULL;
+ pud = (pud_t *)ptdesc_address(ptdesc);
+
+ pud_init(pud);
return pud;
}
diff --git a/arch/loongarch/mm/pgtable.c b/arch/loongarch/mm/pgtable.c
index 36a6dc0148ae..ff07b8f1ef30 100644
--- a/arch/loongarch/mm/pgtable.c
+++ b/arch/loongarch/mm/pgtable.c
@@ -11,10 +11,11 @@
pgd_t *pgd_alloc(struct mm_struct *mm)
{
- pgd_t *ret, *init;
+ pgd_t *init, *ret = NULL;
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
- ret = (pgd_t *) __get_free_page(GFP_KERNEL);
- if (ret) {
+ if (ptdesc) {
+ ret = (pgd_t *)ptdesc_address(ptdesc);
init = pgd_offset(_mm, 0UL);
pgd_init(ret);
memcpy(ret + USER_PTRS_PER_PGD, init + USER_PTRS_PER_PGD,
--
2.39.2
[PATCH v2 23/34] hexagon: Convert __pte_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents.
Signed-off-by: Vishal Moola (Oracle)
---
arch/hexagon/include/asm/pgalloc.h | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/hexagon/include/asm/pgalloc.h
b/arch/hexagon/include/asm/pgalloc.h
index f0c47e6a7427..0f8432430e68 100644
--- a/arch/hexagon/include/asm/pgalloc.h
+++ b/arch/hexagon/include/asm/pgalloc.h
@@ -87,10 +87,10 @@ static inline void pmd_populate_kernel(struct mm_struct
*mm, pmd_t *pmd,
max_kernel_seg = pmdindex;
}
-#define __pte_free_tlb(tlb, pte, addr) \
-do { \
- pgtable_pte_page_dtor((pte)); \
- tlb_remove_page((tlb), (pte)); \
+#define __pte_free_tlb(tlb, pte, addr) \
+do { \
+ ptdesc_pte_dtor((page_ptdesc(pte)));\
+ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \
} while (0)
#endif
--
2.39.2
[PATCH v2 22/34] csky: Convert __pte_free_tlb() to use ptdescs
Part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents.
Signed-off-by: Vishal Moola (Oracle)
---
arch/csky/include/asm/pgalloc.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h
index 7d57e5da0914..af26f1191b43 100644
--- a/arch/csky/include/asm/pgalloc.h
+++ b/arch/csky/include/asm/pgalloc.h
@@ -63,8 +63,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
#define __pte_free_tlb(tlb, pte, address) \
do { \
- pgtable_pte_page_dtor(pte); \
- tlb_remove_page(tlb, pte); \
+ ptdesc_pte_dtor(page_ptdesc(pte)); \
+ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \
} while (0)
extern void pagetable_init(void);
--
2.39.2
[PATCH v2 21/34] arm64: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Signed-off-by: Vishal Moola (Oracle)
---
arch/arm64/include/asm/tlb.h | 14 --
arch/arm64/mm/mmu.c | 7 ---
2 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h
index c995d1f4594f..6cb70c247e30 100644
--- a/arch/arm64/include/asm/tlb.h
+++ b/arch/arm64/include/asm/tlb.h
@@ -75,18 +75,20 @@ static inline void tlb_flush(struct mmu_gather *tlb)
static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
unsigned long addr)
{
- pgtable_pte_page_dtor(pte);
- tlb_remove_table(tlb, pte);
+ struct ptdesc *ptdesc = page_ptdesc(pte);
+
+ ptdesc_pte_dtor(ptdesc);
+ tlb_remove_ptdesc(tlb, ptdesc);
}
#if CONFIG_PGTABLE_LEVELS > 2
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long addr)
{
- struct page *page = virt_to_page(pmdp);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pmdp);
- pgtable_pmd_page_dtor(page);
- tlb_remove_table(tlb, page);
+ ptdesc_pmd_dtor(ptdesc);
+ tlb_remove_ptdesc(tlb, ptdesc);
}
#endif
@@ -94,7 +96,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb,
pmd_t *pmdp,
static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
unsigned long addr)
{
- tlb_remove_table(tlb, virt_to_page(pudp));
+ tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp));
}
#endif
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index af6bc8403ee4..5ba005fd607e 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -426,6 +426,7 @@ static phys_addr_t __pgd_pgtable_alloc(int shift)
static phys_addr_t pgd_pgtable_alloc(int shift)
{
phys_addr_t pa = __pgd_pgtable_alloc(shift);
+ struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa));
/*
* Call proper page table ctor in case later we need to
@@ -433,12 +434,12 @@ static phys_addr_t pgd_pgtable_alloc(int shift)
* this pre-allocated page table.
*
* We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is
-* folded, and if so pgtable_pmd_page_ctor() becomes nop.
+* folded, and if so ptdesc_pte_dtor() becomes nop.
*/
if (shift == PAGE_SHIFT)
- BUG_ON(!pgtable_pte_page_ctor(phys_to_page(pa)));
+ BUG_ON(!ptdesc_pte_dtor(ptdesc));
else if (shift == PMD_SHIFT)
- BUG_ON(!pgtable_pmd_page_ctor(phys_to_page(pa)));
+ BUG_ON(!ptdesc_pte_dtor(ptdesc));
return pa;
}
--
2.39.2
[PATCH v2 20/34] arm: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
late_alloc() also uses the __get_free_pages() helper function. Convert
this to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/arm/include/asm/tlb.h | 12 +++-
arch/arm/mm/mmu.c | 6 +++---
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h
index b8cbe03ad260..9ab8a6929d35 100644
--- a/arch/arm/include/asm/tlb.h
+++ b/arch/arm/include/asm/tlb.h
@@ -39,7 +39,9 @@ static inline void __tlb_remove_table(void *_table)
static inline void
__pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr)
{
- pgtable_pte_page_dtor(pte);
+ struct ptdesc *ptdesc = page_ptdesc(pte);
+
+ ptdesc_pte_dtor(ptdesc);
#ifndef CONFIG_ARM_LPAE
/*
@@ -50,17 +52,17 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
unsigned long addr)
__tlb_adjust_range(tlb, addr - PAGE_SIZE, 2 * PAGE_SIZE);
#endif
- tlb_remove_table(tlb, pte);
+ tlb_remove_ptdesc(tlb, ptdesc);
}
static inline void
__pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr)
{
#ifdef CONFIG_ARM_LPAE
- struct page *page = virt_to_page(pmdp);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pmdp);
- pgtable_pmd_page_dtor(page);
- tlb_remove_table(tlb, page);
+ ptdesc_pmd_dtor(ptdesc);
+ tlb_remove_ptdesc(tlb, ptdesc);
#endif
}
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index 463fc2a8448f..7add505bd797 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -737,11 +737,11 @@ static void __init *early_alloc(unsigned long sz)
static void *__init late_alloc(unsigned long sz)
{
- void *ptr = (void *)__get_free_pages(GFP_PGTABLE_KERNEL, get_order(sz));
+ void *ptdesc = ptdesc_alloc(GFP_PGTABLE_KERNEL, get_order(sz));
- if (!ptr || !pgtable_pte_page_ctor(virt_to_page(ptr)))
+ if (!ptdesc || !ptdesc_pte_ctor(ptdesc))
BUG();
- return ptr;
+ return ptdesc;
}
static pte_t * __init arm_pte_alloc(pmd_t *pmd, unsigned long addr,
--
2.39.2
[PATCH v2 19/34] pgalloc: Convert various functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
include/asm-generic/pgalloc.h | 62 +--
1 file changed, 37 insertions(+), 25 deletions(-)
diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h
index a7cf825befae..7d4a1f5d3c17 100644
--- a/include/asm-generic/pgalloc.h
+++ b/include/asm-generic/pgalloc.h
@@ -18,7 +18,11 @@
*/
static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm)
{
- return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL);
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_PGTABLE_KERNEL, 0);
+
+ if (!ptdesc)
+ return NULL;
+ return (pte_t *)ptdesc_address(ptdesc);
}
#ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL
@@ -41,7 +45,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct
*mm)
*/
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
- free_page((unsigned long)pte);
+ ptdesc_free(virt_to_ptdesc(pte));
}
/**
@@ -49,7 +53,7 @@ static inline void pte_free_kernel(struct mm_struct *mm,
pte_t *pte)
* @mm: the mm_struct of the current context
* @gfp: GFP flags to use for the allocation
*
- * Allocates a page and runs the pgtable_pte_page_ctor().
+ * Allocates a ptdesc and runs the ptdesc_pte_ctor().
*
* This function is intended for architectures that need
* anything beyond simple page allocation or must have custom GFP flags.
@@ -58,17 +62,17 @@ static inline void pte_free_kernel(struct mm_struct *mm,
pte_t *pte)
*/
static inline pgtable_t __pte_alloc_one(struct mm_struct *mm, gfp_t gfp)
{
- struct page *pte;
+ struct ptdesc *ptdesc;
- pte = alloc_page(gfp);
- if (!pte)
+ ptdesc = ptdesc_alloc(gfp, 0);
+ if (!ptdesc)
return NULL;
- if (!pgtable_pte_page_ctor(pte)) {
- __free_page(pte);
+ if (!ptdesc_pte_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- return pte;
+ return ptdesc_page(ptdesc);
}
#ifndef __HAVE_ARCH_PTE_ALLOC_ONE
@@ -76,7 +80,7 @@ static inline pgtable_t __pte_alloc_one(struct mm_struct *mm,
gfp_t gfp)
* pte_alloc_one - allocate a page for PTE-level user page table
* @mm: the mm_struct of the current context
*
- * Allocates a page and runs the pgtable_pte_page_ctor().
+ * Allocates a ptdesc and runs the ptdesc_pte_ctor().
*
* Return: `struct page` initialized as page table or %NULL on error
*/
@@ -98,8 +102,10 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
*/
static inline void pte_free(struct mm_struct *mm, struct page *pte_page)
{
- pgtable_pte_page_dtor(pte_page);
- __free_page(pte_page);
+ struct ptdesc *ptdesc = page_ptdesc(pte_page);
+
+ ptdesc_pte_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
@@ -110,7 +116,7 @@ static inline void pte_free(struct mm_struct *mm, struct
page *pte_page)
* pmd_alloc_one - allocate a page for PMD-level page table
* @mm: the mm_struct of the current context
*
- * Allocates a page and runs the pgtable_pmd_page_ctor().
+ * Allocates a ptdesc and runs the ptdesc_pmd_ctor().
* Allocations use %GFP_PGTABLE_USER in user context and
* %GFP_PGTABLE_KERNEL in kernel context.
*
@@ -118,28 +124,30 @@ static inline void pte_free(struct mm_struct *mm, struct
page *pte_page)
*/
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- struct page *page;
+ struct ptdesc *ptdesc;
gfp_t gfp = GFP_PGTABLE_USER;
if (mm == _mm)
gfp = GFP_PGTABLE_KERNEL;
- page = alloc_page(gfp);
- if (!page)
+ ptdesc = ptdesc_alloc(gfp, 0);
+ if (!ptdesc)
return NULL;
- if (!pgtable_pmd_page_ctor(page)) {
- __free_page(page);
+ if (!ptdesc_pmd_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- return (pmd_t *)page_address(page);
+ return (pmd_t *)ptdesc_address(ptdesc);
}
#endif
#ifndef __HAVE_ARCH_PMD_FREE
static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
+ struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
+
BUG_ON((unsigned long)pmd & (PAGE_SIZE-1));
- pgtable_pmd_page_dtor(virt_to_page(pmd));
- free_page((unsigned long)pmd);
+ ptdesc_pmd_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
#endif
@@ -149,11 +157,15 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t
*pmd)
static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- gfp_t gfp = GFP_PGTABLE_USER;
+ gfp_t gfp = GFP_PGTABLE_USER | __GFP_ZERO;
+ struct
[PATCH v2 18/34] mm: Remove page table members from struct page
The page table members are now split out into their own ptdesc struct.
Remove them from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm_types.h | 14 --
include/linux/pgtable.h | 3 ---
2 files changed, 17 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 6161fe1ae5b8..31ffa1be21d0 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -141,20 +141,6 @@ struct page {
struct {/* Tail pages of compound page */
unsigned long compound_head;/* Bit zero is set */
};
- struct {/* Page table pages */
- unsigned long _pt_pad_1;/* compound_head */
- pgtable_t pmd_huge_pte; /* protected by page->ptl */
- unsigned long _pt_s390_gaddr; /* mapping */
- union {
- struct mm_struct *pt_mm; /* x86 pgds only */
- atomic_t pt_frag_refcount; /* powerpc */
- };
-#if ALLOC_SPLIT_PTLOCKS
- spinlock_t *ptl;
-#else
- spinlock_t ptl;
-#endif
- };
struct {/* ZONE_DEVICE pages */
/** @pgmap: Points to the hosting device page map. */
struct dev_pagemap *pgmap;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index b067ac10f3dd..90fa73a896db 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1034,10 +1034,7 @@ struct ptdesc {
TABLE_MATCH(flags, __page_flags);
TABLE_MATCH(compound_head, pt_list);
TABLE_MATCH(compound_head, _pt_pad_1);
-TABLE_MATCH(pmd_huge_pte, pmd_huge_pte);
TABLE_MATCH(mapping, _pt_s390_gaddr);
-TABLE_MATCH(pt_mm, pt_mm);
-TABLE_MATCH(ptl, ptl);
#undef TABLE_MATCH
static_assert(sizeof(struct ptdesc) <= sizeof(struct page));
--
2.39.2
[PATCH v2 17/34] s390: Convert various pgalloc functions to use ptdescs
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/s390/include/asm/pgalloc.h | 4 +-
arch/s390/include/asm/tlb.h | 4 +-
arch/s390/mm/pgalloc.c | 108
3 files changed, 59 insertions(+), 57 deletions(-)
diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h
index 17eb618f1348..9841481560ae 100644
--- a/arch/s390/include/asm/pgalloc.h
+++ b/arch/s390/include/asm/pgalloc.h
@@ -86,7 +86,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm,
unsigned long vmaddr)
if (!table)
return NULL;
crst_table_init(table, _SEGMENT_ENTRY_EMPTY);
- if (!pgtable_pmd_page_ctor(virt_to_page(table))) {
+ if (!ptdesc_pmd_ctor(virt_to_ptdesc(table))) {
crst_table_free(mm, table);
return NULL;
}
@@ -97,7 +97,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
if (mm_pmd_folded(mm))
return;
- pgtable_pmd_page_dtor(virt_to_page(pmd));
+ ptdesc_pmd_dtor(virt_to_ptdesc(pmd));
crst_table_free(mm, (unsigned long *) pmd);
}
diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h
index b91f4a9b044c..1388c819b467 100644
--- a/arch/s390/include/asm/tlb.h
+++ b/arch/s390/include/asm/tlb.h
@@ -89,12 +89,12 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb,
pmd_t *pmd,
{
if (mm_pmd_folded(tlb->mm))
return;
- pgtable_pmd_page_dtor(virt_to_page(pmd));
+ ptdesc_pmd_dtor(virt_to_ptdesc(pmd));
__tlb_adjust_range(tlb, address, PAGE_SIZE);
tlb->mm->context.flush_mm = 1;
tlb->freed_tables = 1;
tlb->cleared_puds = 1;
- tlb_remove_table(tlb, pmd);
+ tlb_remove_ptdesc(tlb, pmd);
}
/*
diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c
index 6b99932abc66..e740b4c76665 100644
--- a/arch/s390/mm/pgalloc.c
+++ b/arch/s390/mm/pgalloc.c
@@ -43,17 +43,17 @@ __initcall(page_table_register_sysctl);
unsigned long *crst_table_alloc(struct mm_struct *mm)
{
- struct page *page = alloc_pages(GFP_KERNEL, CRST_ALLOC_ORDER);
+ struct ptdesc *ptdesc = ptdesc_alloc(GFP_KERNEL, CRST_ALLOC_ORDER);
- if (!page)
+ if (!ptdesc)
return NULL;
- arch_set_page_dat(page, CRST_ALLOC_ORDER);
- return (unsigned long *) page_to_virt(page);
+ arch_set_page_dat(ptdesc_page(ptdesc), CRST_ALLOC_ORDER);
+ return (unsigned long *) ptdesc_to_virt(ptdesc);
}
void crst_table_free(struct mm_struct *mm, unsigned long *table)
{
- free_pages((unsigned long)table, CRST_ALLOC_ORDER);
+ ptdesc_free(virt_to_ptdesc(table));
}
static void __crst_table_upgrade(void *arg)
@@ -140,21 +140,21 @@ static inline unsigned int atomic_xor_bits(atomic_t *v,
unsigned int bits)
struct page *page_table_alloc_pgste(struct mm_struct *mm)
{
- struct page *page;
+ struct ptdesc *ptdesc;
u64 *table;
- page = alloc_page(GFP_KERNEL);
- if (page) {
- table = (u64 *)page_to_virt(page);
+ ptdesc = ptdesc_alloc(GFP_KERNEL, 0);
+ if (ptdesc) {
+ table = (u64 *)ptdesc_to_virt(ptdesc);
memset64(table, _PAGE_INVALID, PTRS_PER_PTE);
memset64(table + PTRS_PER_PTE, 0, PTRS_PER_PTE);
}
- return page;
+ return ptdesc_page(ptdesc);
}
void page_table_free_pgste(struct page *page)
{
- __free_page(page);
+ ptdesc_free(page_ptdesc(page));
}
#endif /* CONFIG_PGSTE */
@@ -230,7 +230,7 @@ void page_table_free_pgste(struct page *page)
unsigned long *page_table_alloc(struct mm_struct *mm)
{
unsigned long *table;
- struct page *page;
+ struct ptdesc *ptdesc;
unsigned int mask, bit;
/* Try to get a fragment of a 4K page as a 2K page table */
@@ -238,9 +238,9 @@ unsigned long *page_table_alloc(struct mm_struct *mm)
table = NULL;
spin_lock_bh(>context.lock);
if (!list_empty(>context.pgtable_list)) {
- page = list_first_entry(>context.pgtable_list,
- struct page, lru);
- mask = atomic_read(>pt_frag_refcount);
+ ptdesc = list_first_entry(>context.pgtable_list,
+ struct ptdesc, pt_list);
+ mask = atomic_read(>pt_frag_refcount);
/*
* The pending removal bits must also be checked.
* Failure to do so might lead to an impossible
@@
[PATCH v2 16/34] s390: Convert various gmap functions to use ptdescs
In order to split struct ptdesc from struct page, convert various
functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/s390/mm/gmap.c | 230
1 file changed, 128 insertions(+), 102 deletions(-)
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index a9e8b1805894..e833a7e81fbd 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -34,7 +34,7 @@
static struct gmap *gmap_alloc(unsigned long limit)
{
struct gmap *gmap;
- struct page *page;
+ struct ptdesc *ptdesc;
unsigned long *table;
unsigned long etype, atype;
@@ -67,12 +67,12 @@ static struct gmap *gmap_alloc(unsigned long limit)
spin_lock_init(>guest_table_lock);
spin_lock_init(>shadow_lock);
refcount_set(>ref_count, 1);
- page = alloc_pages(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER);
- if (!page)
+ ptdesc = ptdesc_alloc(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER);
+ if (!ptdesc)
goto out_free;
- page->_pt_s390_gaddr = 0;
- list_add(>lru, >crst_list);
- table = page_to_virt(page);
+ ptdesc->_pt_s390_gaddr = 0;
+ list_add(>pt_list, >crst_list);
+ table = ptdesc_to_virt(ptdesc);
crst_table_init(table, etype);
gmap->table = table;
gmap->asce = atype | _ASCE_TABLE_LENGTH |
@@ -181,25 +181,25 @@ static void gmap_rmap_radix_tree_free(struct
radix_tree_root *root)
*/
static void gmap_free(struct gmap *gmap)
{
- struct page *page, *next;
+ struct ptdesc *ptdesc, *next;
/* Flush tlb of all gmaps (if not already done for shadows) */
if (!(gmap_is_shadow(gmap) && gmap->removed))
gmap_flush_tlb(gmap);
/* Free all segment & region tables. */
- list_for_each_entry_safe(page, next, >crst_list, lru) {
- page->_pt_s390_gaddr = 0;
- __free_pages(page, CRST_ALLOC_ORDER);
+ list_for_each_entry_safe(ptdesc, next, >crst_list, pt_list) {
+ ptdesc->_pt_s390_gaddr = 0;
+ ptdesc_free(ptdesc);
}
gmap_radix_tree_free(>guest_to_host);
gmap_radix_tree_free(>host_to_guest);
/* Free additional data for a shadow gmap */
if (gmap_is_shadow(gmap)) {
- /* Free all page tables. */
- list_for_each_entry_safe(page, next, >pt_list, lru) {
- page->_pt_s390_gaddr = 0;
- page_table_free_pgste(page);
+ /* Free all ptdesc tables. */
+ list_for_each_entry_safe(ptdesc, next, >pt_list, pt_list)
{
+ ptdesc->_pt_s390_gaddr = 0;
+ page_table_free_pgste(ptdesc_page(ptdesc));
}
gmap_rmap_radix_tree_free(>host_to_rmap);
/* Release reference to the parent */
@@ -308,27 +308,27 @@ EXPORT_SYMBOL_GPL(gmap_get_enabled);
static int gmap_alloc_table(struct gmap *gmap, unsigned long *table,
unsigned long init, unsigned long gaddr)
{
- struct page *page;
+ struct ptdesc *ptdesc;
unsigned long *new;
/* since we dont free the gmap table until gmap_free we can unlock */
- page = alloc_pages(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER);
- if (!page)
+ ptdesc = ptdesc_alloc(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER);
+ if (!ptdesc)
return -ENOMEM;
- new = page_to_virt(page);
+ new = ptdesc_to_virt(ptdesc);
crst_table_init(new, init);
spin_lock(>guest_table_lock);
if (*table & _REGION_ENTRY_INVALID) {
- list_add(>lru, >crst_list);
+ list_add(>pt_list, >crst_list);
*table = __pa(new) | _REGION_ENTRY_LENGTH |
(*table & _REGION_ENTRY_TYPE_MASK);
- page->_pt_s390_gaddr = gaddr;
- page = NULL;
+ ptdesc->_pt_s390_gaddr = gaddr;
+ ptdesc = NULL;
}
spin_unlock(>guest_table_lock);
- if (page) {
- page->_pt_s390_gaddr = 0;
- __free_pages(page, CRST_ALLOC_ORDER);
+ if (ptdesc) {
+ ptdesc->_pt_s390_gaddr = 0;
+ ptdesc_free(ptdesc);
}
return 0;
}
@@ -341,15 +341,15 @@ static int gmap_alloc_table(struct gmap *gmap, unsigned
long *table,
*/
static unsigned long __gmap_segment_gaddr(unsigned long *entry)
{
- struct page *page;
+ struct ptdesc *ptdesc;
unsigned long offset, mask;
offset = (unsigned long) entry / sizeof(unsigned long);
offset = (offset & (PTRS_PER_PMD - 1)) * PMD_SIZE;
mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1);
- page = virt_to_page((void *)((unsigned long) entry & mask));
+
[PATCH v2 15/34] x86: Convert various functions to use ptdescs
In order to split struct ptdesc from struct page, convert various
functions to use ptdescs.
Some of the functions use the *get*page*() helper functions. Convert
these to use ptdesc_alloc() and ptdesc_address() instead to help
standardize page tables further.
Signed-off-by: Vishal Moola (Oracle)
---
arch/x86/mm/pgtable.c | 46 +--
1 file changed, 27 insertions(+), 19 deletions(-)
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index afab0bc7862b..9b6f81c8eb32 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -52,7 +52,7 @@ early_param("userpte", setup_userpte);
void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
{
- pgtable_pte_page_dtor(pte);
+ ptdesc_pte_dtor(page_ptdesc(pte));
paravirt_release_pte(page_to_pfn(pte));
paravirt_tlb_remove_table(tlb, pte);
}
@@ -60,7 +60,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
#if CONFIG_PGTABLE_LEVELS > 2
void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
{
- struct page *page = virt_to_page(pmd);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT);
/*
* NOTE! For PAE, any changes to the top page-directory-pointer-table
@@ -69,8 +69,8 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
#ifdef CONFIG_X86_PAE
tlb->need_flush_all = 1;
#endif
- pgtable_pmd_page_dtor(page);
- paravirt_tlb_remove_table(tlb, page);
+ ptdesc_pmd_dtor(ptdesc);
+ paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc));
}
#if CONFIG_PGTABLE_LEVELS > 3
@@ -92,16 +92,16 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
static inline void pgd_list_add(pgd_t *pgd)
{
- struct page *page = virt_to_page(pgd);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pgd);
- list_add(>lru, _list);
+ list_add(>pt_list, _list);
}
static inline void pgd_list_del(pgd_t *pgd)
{
- struct page *page = virt_to_page(pgd);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pgd);
- list_del(>lru);
+ list_del(>pt_list);
}
#define UNSHARED_PTRS_PER_PGD \
@@ -112,12 +112,12 @@ static inline void pgd_list_del(pgd_t *pgd)
static void pgd_set_mm(pgd_t *pgd, struct mm_struct *mm)
{
- virt_to_page(pgd)->pt_mm = mm;
+ virt_to_ptdesc(pgd)->pt_mm = mm;
}
struct mm_struct *pgd_page_get_mm(struct page *page)
{
- return page->pt_mm;
+ return page_ptdesc(page)->pt_mm;
}
static void pgd_ctor(struct mm_struct *mm, pgd_t *pgd)
@@ -213,11 +213,14 @@ void pud_populate(struct mm_struct *mm, pud_t *pudp,
pmd_t *pmd)
static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)
{
int i;
+ struct ptdesc *ptdesc;
for (i = 0; i < count; i++)
if (pmds[i]) {
- pgtable_pmd_page_dtor(virt_to_page(pmds[i]));
- free_page((unsigned long)pmds[i]);
+ ptdesc = virt_to_ptdesc(pmds[i]);
+
+ ptdesc_pmd_dtor(ptdesc);
+ ptdesc_free(ptdesc);
mm_dec_nr_pmds(mm);
}
}
@@ -232,16 +235,21 @@ static int preallocate_pmds(struct mm_struct *mm, pmd_t
*pmds[], int count)
gfp &= ~__GFP_ACCOUNT;
for (i = 0; i < count; i++) {
- pmd_t *pmd = (pmd_t *)__get_free_page(gfp);
- if (!pmd)
+ pmd_t *pmd = NULL;
+ struct ptdesc *ptdesc = ptdesc_alloc(gfp, 0);
+
+ if (!ptdesc)
failed = true;
- if (pmd && !pgtable_pmd_page_ctor(virt_to_page(pmd))) {
- free_page((unsigned long)pmd);
- pmd = NULL;
+ if (ptdesc && !ptdesc_pmd_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
+ ptdesc = NULL;
failed = true;
}
- if (pmd)
+ if (ptdesc) {
mm_inc_nr_pmds(mm);
+ pmd = (pmd_t *)ptdesc_address(ptdesc);
+ }
+
pmds[i] = pmd;
}
@@ -838,7 +846,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)
free_page((unsigned long)pmd_sv);
- pgtable_pmd_page_dtor(virt_to_page(pmd));
+ ptdesc_pmd_dtor(virt_to_ptdesc(pmd));
free_page((unsigned long)pmd);
return 1;
--
2.39.2
[PATCH v2 14/34] powerpc: Convert various functions to use ptdescs
In order to split struct ptdesc from struct page, convert various
functions to use ptdescs.
Signed-off-by: Vishal Moola (Oracle)
---
arch/powerpc/mm/book3s64/mmu_context.c | 10 +++---
arch/powerpc/mm/book3s64/pgtable.c | 32 +-
arch/powerpc/mm/pgtable-frag.c | 46 +-
3 files changed, 44 insertions(+), 44 deletions(-)
diff --git a/arch/powerpc/mm/book3s64/mmu_context.c
b/arch/powerpc/mm/book3s64/mmu_context.c
index c766e4c26e42..b22ad2839897 100644
--- a/arch/powerpc/mm/book3s64/mmu_context.c
+++ b/arch/powerpc/mm/book3s64/mmu_context.c
@@ -246,15 +246,15 @@ static void destroy_contexts(mm_context_t *ctx)
static void pmd_frag_destroy(void *pmd_frag)
{
int count;
- struct page *page;
+ struct ptdesc *ptdesc;
- page = virt_to_page(pmd_frag);
+ ptdesc = virt_to_ptdesc(pmd_frag);
/* drop all the pending references */
count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT;
/* We allow PTE_FRAG_NR fragments from a PTE page */
- if (atomic_sub_and_test(PMD_FRAG_NR - count, >pt_frag_refcount)) {
- pgtable_pmd_page_dtor(page);
- __free_page(page);
+ if (atomic_sub_and_test(PMD_FRAG_NR - count,
>pt_frag_refcount)) {
+ ptdesc_pmd_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
}
diff --git a/arch/powerpc/mm/book3s64/pgtable.c
b/arch/powerpc/mm/book3s64/pgtable.c
index 85c84e89e3ea..da46e3efc66c 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -306,22 +306,22 @@ static pmd_t *get_pmd_from_cache(struct mm_struct *mm)
static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm)
{
void *ret = NULL;
- struct page *page;
+ struct ptdesc *ptdesc;
gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO;
if (mm == _mm)
gfp &= ~__GFP_ACCOUNT;
- page = alloc_page(gfp);
- if (!page)
+ ptdesc = ptdesc_alloc(gfp, 0);
+ if (!ptdesc)
return NULL;
- if (!pgtable_pmd_page_ctor(page)) {
- __free_pages(page, 0);
+ if (!ptdesc_pmd_ctor(ptdesc)) {
+ ptdesc_free(ptdesc);
return NULL;
}
- atomic_set(>pt_frag_refcount, 1);
+ atomic_set(>pt_frag_refcount, 1);
- ret = page_address(page);
+ ret = ptdesc_address(ptdesc);
/*
* if we support only one fragment just return the
* allocated page.
@@ -331,12 +331,12 @@ static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm)
spin_lock(>page_table_lock);
/*
-* If we find pgtable_page set, we return
+* If we find ptdesc_page set, we return
* the allocated page with single fragment
* count.
*/
if (likely(!mm->context.pmd_frag)) {
- atomic_set(>pt_frag_refcount, PMD_FRAG_NR);
+ atomic_set(>pt_frag_refcount, PMD_FRAG_NR);
mm->context.pmd_frag = ret + PMD_FRAG_SIZE;
}
spin_unlock(>page_table_lock);
@@ -357,15 +357,15 @@ pmd_t *pmd_fragment_alloc(struct mm_struct *mm, unsigned
long vmaddr)
void pmd_fragment_free(unsigned long *pmd)
{
- struct page *page = virt_to_page(pmd);
+ struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
- if (PageReserved(page))
- return free_reserved_page(page);
+ if (ptdesc_is_reserved(ptdesc))
+ return free_reserved_ptdesc(ptdesc);
- BUG_ON(atomic_read(>pt_frag_refcount) <= 0);
- if (atomic_dec_and_test(>pt_frag_refcount)) {
- pgtable_pmd_page_dtor(page);
- __free_page(page);
+ BUG_ON(atomic_read(>pt_frag_refcount) <= 0);
+ if (atomic_dec_and_test(>pt_frag_refcount)) {
+ ptdesc_pmd_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
}
diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c
index 20652daa1d7e..b53e18fab74a 100644
--- a/arch/powerpc/mm/pgtable-frag.c
+++ b/arch/powerpc/mm/pgtable-frag.c
@@ -18,15 +18,15 @@
void pte_frag_destroy(void *pte_frag)
{
int count;
- struct page *page;
+ struct ptdesc *ptdesc;
- page = virt_to_page(pte_frag);
+ ptdesc = virt_to_ptdesc(pte_frag);
/* drop all the pending references */
count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT;
/* We allow PTE_FRAG_NR fragments from a PTE page */
- if (atomic_sub_and_test(PTE_FRAG_NR - count, >pt_frag_refcount)) {
- pgtable_pte_page_dtor(page);
- __free_page(page);
+ if (atomic_sub_and_test(PTE_FRAG_NR - count,
>pt_frag_refcount)) {
+ ptdesc_pte_dtor(ptdesc);
+ ptdesc_free(ptdesc);
}
}
@@ -55,25 +55,25 @@ static pte_t *get_pte_from_cache(struct mm_struct *mm)
static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)
{
[PATCH v2 13/34] mm: Create ptdesc equivalents for pgtable_{pte,pmd}_page_{ctor,dtor}
Creates ptdesc_pte_ctor(), ptdesc_pmd_ctor(), ptdesc_pte_dtor(), and
ptdesc_pmd_dtor() and make the original pgtable constructor/destructors
wrappers.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 56 ++
1 file changed, 42 insertions(+), 14 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 58c911341a33..dc61aeca9077 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2847,20 +2847,34 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) {
return true; }
static inline void ptlock_free(struct ptdesc *ptdesc) {}
#endif /* USE_SPLIT_PTE_PTLOCKS */
-static inline bool pgtable_pte_page_ctor(struct page *page)
+static inline bool ptdesc_pte_ctor(struct ptdesc *ptdesc)
{
- if (!ptlock_init(page_ptdesc(page)))
+ struct folio *folio = ptdesc_folio(ptdesc);
+
+ if (!ptlock_init(ptdesc))
return false;
- __SetPageTable(page);
- inc_lruvec_page_state(page, NR_PAGETABLE);
+ __folio_set_table(folio);
+ lruvec_stat_add_folio(folio, NR_PAGETABLE);
return true;
}
+static inline bool pgtable_pte_page_ctor(struct page *page)
+{
+ return ptdesc_pte_ctor(page_ptdesc(page));
+}
+
+static inline void ptdesc_pte_dtor(struct ptdesc *ptdesc)
+{
+ struct folio *folio = ptdesc_folio(ptdesc);
+
+ ptlock_free(ptdesc);
+ __folio_clear_table(folio);
+ lruvec_stat_sub_folio(folio, NR_PAGETABLE);
+}
+
static inline void pgtable_pte_page_dtor(struct page *page)
{
- ptlock_free(page_ptdesc(page));
- __ClearPageTable(page);
- dec_lruvec_page_state(page, NR_PAGETABLE);
+ ptdesc_pte_dtor(page_ptdesc(page));
}
#define pte_offset_map_lock(mm, pmd, address, ptlp)\
@@ -2942,20 +2956,34 @@ static inline spinlock_t *pmd_lock(struct mm_struct
*mm, pmd_t *pmd)
return ptl;
}
-static inline bool pgtable_pmd_page_ctor(struct page *page)
+static inline bool ptdesc_pmd_ctor(struct ptdesc *ptdesc)
{
- if (!pmd_ptlock_init(page_ptdesc(page)))
+ struct folio *folio = ptdesc_folio(ptdesc);
+
+ if (!pmd_ptlock_init(ptdesc))
return false;
- __SetPageTable(page);
- inc_lruvec_page_state(page, NR_PAGETABLE);
+ __folio_set_table(folio);
+ lruvec_stat_add_folio(folio, NR_PAGETABLE);
return true;
}
+static inline bool pgtable_pmd_page_ctor(struct page *page)
+{
+ return ptdesc_pmd_ctor(page_ptdesc(page));
+}
+
+static inline void ptdesc_pmd_dtor(struct ptdesc *ptdesc)
+{
+ struct folio *folio = ptdesc_folio(ptdesc);
+
+ pmd_ptlock_free(ptdesc);
+ __folio_clear_table(folio);
+ lruvec_stat_sub_folio(folio, NR_PAGETABLE);
+}
+
static inline void pgtable_pmd_page_dtor(struct page *page)
{
- pmd_ptlock_free(page_ptdesc(page));
- __ClearPageTable(page);
- dec_lruvec_page_state(page, NR_PAGETABLE);
+ ptdesc_pmd_dtor(page_ptdesc(page));
}
/*
--
2.39.2
[PATCH v2 12/34] mm: Convert ptlock_free() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 10 +-
mm/memory.c| 4 ++--
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a2a1bca84ada..58c911341a33 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2787,7 +2787,7 @@ static inline void ptdesc_clear(void *x)
#if ALLOC_SPLIT_PTLOCKS
void __init ptlock_cache_init(void);
bool ptlock_alloc(struct ptdesc *ptdesc);
-extern void ptlock_free(struct page *page);
+void ptlock_free(struct ptdesc *ptdesc);
static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc)
{
@@ -2803,7 +2803,7 @@ static inline bool ptlock_alloc(struct ptdesc *ptdesc)
return true;
}
-static inline void ptlock_free(struct page *page)
+static inline void ptlock_free(struct ptdesc *ptdesc)
{
}
@@ -2844,7 +2844,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct
*mm, pmd_t *pmd)
}
static inline void ptlock_cache_init(void) {}
static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; }
-static inline void ptlock_free(struct page *page) {}
+static inline void ptlock_free(struct ptdesc *ptdesc) {}
#endif /* USE_SPLIT_PTE_PTLOCKS */
static inline bool pgtable_pte_page_ctor(struct page *page)
@@ -2858,7 +2858,7 @@ static inline bool pgtable_pte_page_ctor(struct page
*page)
static inline void pgtable_pte_page_dtor(struct page *page)
{
- ptlock_free(page);
+ ptlock_free(page_ptdesc(page));
__ClearPageTable(page);
dec_lruvec_page_state(page, NR_PAGETABLE);
}
@@ -2916,7 +2916,7 @@ static inline void pmd_ptlock_free(struct ptdesc *ptdesc)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc));
#endif
- ptlock_free(ptdesc_page(ptdesc));
+ ptlock_free(ptdesc);
}
#define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte)
diff --git a/mm/memory.c b/mm/memory.c
index ba0dd1b2d616..7a0b36560e28 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5950,8 +5950,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc)
return true;
}
-void ptlock_free(struct page *page)
+void ptlock_free(struct ptdesc *ptdesc)
{
- kmem_cache_free(page_ptl_cachep, page->ptl);
+ kmem_cache_free(page_ptl_cachep, ptdesc->ptl);
}
#endif
--
2.39.2
[PATCH v2 11/34] mm: Convert pmd_ptlock_free() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index bbd44f43e375..a2a1bca84ada 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2911,12 +2911,12 @@ static inline bool pmd_ptlock_init(struct ptdesc
*ptdesc)
return ptlock_init(ptdesc);
}
-static inline void pmd_ptlock_free(struct page *page)
+static inline void pmd_ptlock_free(struct ptdesc *ptdesc)
{
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- VM_BUG_ON_PAGE(page->pmd_huge_pte, page);
+ VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc));
#endif
- ptlock_free(page);
+ ptlock_free(ptdesc_page(ptdesc));
}
#define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte)
@@ -2929,7 +2929,7 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct
*mm, pmd_t *pmd)
}
static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; }
-static inline void pmd_ptlock_free(struct page *page) {}
+static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {}
#define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte)
@@ -2953,7 +2953,7 @@ static inline bool pgtable_pmd_page_ctor(struct page
*page)
static inline void pgtable_pmd_page_dtor(struct page *page)
{
- pmd_ptlock_free(page);
+ pmd_ptlock_free(page_ptdesc(page));
__ClearPageTable(page);
dec_lruvec_page_state(page, NR_PAGETABLE);
}
--
2.39.2
[PATCH v2 10/34] mm: Convert ptlock_init() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 044c9f874b47..bbd44f43e375 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2818,7 +2818,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct
*mm, pmd_t *pmd)
return ptlock_ptr(page_ptdesc(pmd_page(*pmd)));
}
-static inline bool ptlock_init(struct page *page)
+static inline bool ptlock_init(struct ptdesc *ptdesc)
{
/*
* prep_new_page() initialize page->private (and therefore page->ptl)
@@ -2827,10 +2827,10 @@ static inline bool ptlock_init(struct page *page)
* It can happen if arch try to use slab for page table allocation:
* slab code uses page->slab_cache, which share storage with page->ptl.
*/
- VM_BUG_ON_PAGE(*(unsigned long *)>ptl, page);
- if (!ptlock_alloc(page_ptdesc(page)))
+ VM_BUG_ON_PAGE(*(unsigned long *)>ptl, ptdesc_page(ptdesc));
+ if (!ptlock_alloc(ptdesc))
return false;
- spin_lock_init(ptlock_ptr(page_ptdesc(page)));
+ spin_lock_init(ptlock_ptr(ptdesc));
return true;
}
@@ -2843,13 +2843,13 @@ static inline spinlock_t *pte_lockptr(struct mm_struct
*mm, pmd_t *pmd)
return >page_table_lock;
}
static inline void ptlock_cache_init(void) {}
-static inline bool ptlock_init(struct page *page) { return true; }
+static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; }
static inline void ptlock_free(struct page *page) {}
#endif /* USE_SPLIT_PTE_PTLOCKS */
static inline bool pgtable_pte_page_ctor(struct page *page)
{
- if (!ptlock_init(page))
+ if (!ptlock_init(page_ptdesc(page)))
return false;
__SetPageTable(page);
inc_lruvec_page_state(page, NR_PAGETABLE);
@@ -2908,7 +2908,7 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
ptdesc->pmd_huge_pte = NULL;
#endif
- return ptlock_init(ptdesc_page(ptdesc));
+ return ptlock_init(ptdesc);
}
static inline void pmd_ptlock_free(struct page *page)
--
2.39.2
[PATCH v2 09/34] mm: Convert pmd_ptlock_init() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 49fdc1199bd4..044c9f874b47 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2903,12 +2903,12 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct
*mm, pmd_t *pmd)
return ptlock_ptr(pmd_ptdesc(pmd));
}
-static inline bool pmd_ptlock_init(struct page *page)
+static inline bool pmd_ptlock_init(struct ptdesc *ptdesc)
{
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- page->pmd_huge_pte = NULL;
+ ptdesc->pmd_huge_pte = NULL;
#endif
- return ptlock_init(page);
+ return ptlock_init(ptdesc_page(ptdesc));
}
static inline void pmd_ptlock_free(struct page *page)
@@ -2928,7 +2928,7 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct
*mm, pmd_t *pmd)
return >page_table_lock;
}
-static inline bool pmd_ptlock_init(struct page *page) { return true; }
+static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; }
static inline void pmd_ptlock_free(struct page *page) {}
#define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte)
@@ -2944,7 +2944,7 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm,
pmd_t *pmd)
static inline bool pgtable_pmd_page_ctor(struct page *page)
{
- if (!pmd_ptlock_init(page))
+ if (!pmd_ptlock_init(page_ptdesc(page)))
return false;
__SetPageTable(page);
inc_lruvec_page_state(page, NR_PAGETABLE);
--
2.39.2
[PATCH v2 08/34] mm: Convert ptlock_ptr() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
arch/x86/xen/mmu_pv.c | 2 +-
include/linux/mm.h| 14 +++---
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c
index fdc91deece7e..a1c9f8dcbb5a 100644
--- a/arch/x86/xen/mmu_pv.c
+++ b/arch/x86/xen/mmu_pv.c
@@ -651,7 +651,7 @@ static spinlock_t *xen_pte_lock(struct page *page, struct
mm_struct *mm)
spinlock_t *ptl = NULL;
#if USE_SPLIT_PTE_PTLOCKS
- ptl = ptlock_ptr(page);
+ ptl = ptlock_ptr(page_ptdesc(page));
spin_lock_nest_lock(ptl, >page_table_lock);
#endif
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 565da5f39376..49fdc1199bd4 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2789,9 +2789,9 @@ void __init ptlock_cache_init(void);
bool ptlock_alloc(struct ptdesc *ptdesc);
extern void ptlock_free(struct page *page);
-static inline spinlock_t *ptlock_ptr(struct page *page)
+static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc)
{
- return page->ptl;
+ return ptdesc->ptl;
}
#else /* ALLOC_SPLIT_PTLOCKS */
static inline void ptlock_cache_init(void)
@@ -2807,15 +2807,15 @@ static inline void ptlock_free(struct page *page)
{
}
-static inline spinlock_t *ptlock_ptr(struct page *page)
+static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc)
{
- return >ptl;
+ return >ptl;
}
#endif /* ALLOC_SPLIT_PTLOCKS */
static inline spinlock_t *pte_lockptr(struct mm_struct *mm, pmd_t *pmd)
{
- return ptlock_ptr(pmd_page(*pmd));
+ return ptlock_ptr(page_ptdesc(pmd_page(*pmd)));
}
static inline bool ptlock_init(struct page *page)
@@ -2830,7 +2830,7 @@ static inline bool ptlock_init(struct page *page)
VM_BUG_ON_PAGE(*(unsigned long *)>ptl, page);
if (!ptlock_alloc(page_ptdesc(page)))
return false;
- spin_lock_init(ptlock_ptr(page));
+ spin_lock_init(ptlock_ptr(page_ptdesc(page)));
return true;
}
@@ -2900,7 +2900,7 @@ static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd)
static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd)
{
- return ptlock_ptr(ptdesc_page(pmd_ptdesc(pmd)));
+ return ptlock_ptr(pmd_ptdesc(pmd));
}
static inline bool pmd_ptlock_init(struct page *page)
--
2.39.2
[PATCH v2 07/34] mm: Convert ptlock_alloc() to use ptdescs
This removes some direct accesses to struct page, working towards
splitting out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 6 +++---
mm/memory.c| 4 ++--
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 62c1635a9d44..565da5f39376 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2786,7 +2786,7 @@ static inline void ptdesc_clear(void *x)
#if USE_SPLIT_PTE_PTLOCKS
#if ALLOC_SPLIT_PTLOCKS
void __init ptlock_cache_init(void);
-extern bool ptlock_alloc(struct page *page);
+bool ptlock_alloc(struct ptdesc *ptdesc);
extern void ptlock_free(struct page *page);
static inline spinlock_t *ptlock_ptr(struct page *page)
@@ -2798,7 +2798,7 @@ static inline void ptlock_cache_init(void)
{
}
-static inline bool ptlock_alloc(struct page *page)
+static inline bool ptlock_alloc(struct ptdesc *ptdesc)
{
return true;
}
@@ -2828,7 +2828,7 @@ static inline bool ptlock_init(struct page *page)
* slab code uses page->slab_cache, which share storage with page->ptl.
*/
VM_BUG_ON_PAGE(*(unsigned long *)>ptl, page);
- if (!ptlock_alloc(page))
+ if (!ptlock_alloc(page_ptdesc(page)))
return false;
spin_lock_init(ptlock_ptr(page));
return true;
diff --git a/mm/memory.c b/mm/memory.c
index 5e2c6b1fc00e..ba0dd1b2d616 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5939,14 +5939,14 @@ void __init ptlock_cache_init(void)
SLAB_PANIC, NULL);
}
-bool ptlock_alloc(struct page *page)
+bool ptlock_alloc(struct ptdesc *ptdesc)
{
spinlock_t *ptl;
ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL);
if (!ptl)
return false;
- page->ptl = ptl;
+ ptdesc->ptl = ptl;
return true;
}
--
2.39.2
[PATCH v2 06/34] mm: Convert pmd_pgtable_page() to pmd_ptdesc()
Converts pmd_pgtable_page() to pmd_ptdesc() and all its callers. This
removes some direct accesses to struct page, working towards splitting
out struct ptdesc from struct page.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/mm.h | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 258f3b730359..62c1635a9d44 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2892,15 +2892,15 @@ static inline void pgtable_pte_page_dtor(struct page
*page)
#if USE_SPLIT_PMD_PTLOCKS
-static inline struct page *pmd_pgtable_page(pmd_t *pmd)
+static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd)
{
unsigned long mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1);
- return virt_to_page((void *)((unsigned long) pmd & mask));
+ return virt_to_ptdesc((void *)((unsigned long) pmd & mask));
}
static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd)
{
- return ptlock_ptr(pmd_pgtable_page(pmd));
+ return ptlock_ptr(ptdesc_page(pmd_ptdesc(pmd)));
}
static inline bool pmd_ptlock_init(struct page *page)
@@ -2919,7 +2919,7 @@ static inline void pmd_ptlock_free(struct page *page)
ptlock_free(page);
}
-#define pmd_huge_pte(mm, pmd) (pmd_pgtable_page(pmd)->pmd_huge_pte)
+#define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte)
#else
--
2.39.2
[PATCH v2 05/34] mm: add utility functions for ptdesc
Introduce utility functions setting the foundation for ptdescs. These
will also assist in the splitting out of ptdesc from struct page.
ptdesc_alloc() is defined to allocate new ptdesc pages as compound
pages. This is to standardize ptdescs by allowing for one allocation
and one free function, in contrast to 2 allocation and 2 free functions.
Signed-off-by: Vishal Moola (Oracle)
---
include/asm-generic/tlb.h | 11 ++
include/linux/mm.h| 44 +++
include/linux/pgtable.h | 12 +++
3 files changed, 67 insertions(+)
diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
index b46617207c93..6bade9e0e799 100644
--- a/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -481,6 +481,17 @@ static inline void tlb_remove_page(struct mmu_gather *tlb,
struct page *page)
return tlb_remove_page_size(tlb, page, PAGE_SIZE);
}
+static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt)
+{
+ tlb_remove_table(tlb, pt);
+}
+
+/* Like tlb_remove_ptdesc, but for page-like page directories. */
+static inline void tlb_remove_page_ptdesc(struct mmu_gather *tlb, struct
ptdesc *pt)
+{
+ tlb_remove_page(tlb, ptdesc_page(pt));
+}
+
static inline void tlb_change_page_size(struct mmu_gather *tlb,
unsigned int page_size)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index b18848ae7e22..258f3b730359 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2744,6 +2744,45 @@ static inline pmd_t *pmd_alloc(struct mm_struct *mm,
pud_t *pud, unsigned long a
}
#endif /* CONFIG_MMU */
+static inline struct ptdesc *virt_to_ptdesc(const void *x)
+{
+ return page_ptdesc(virt_to_head_page(x));
+}
+
+static inline void *ptdesc_to_virt(const struct ptdesc *pt)
+{
+ return page_to_virt(ptdesc_page(pt));
+}
+
+static inline void *ptdesc_address(const struct ptdesc *pt)
+{
+ return folio_address(ptdesc_folio(pt));
+}
+
+static inline bool ptdesc_is_reserved(struct ptdesc *pt)
+{
+ return folio_test_reserved(ptdesc_folio(pt));
+}
+
+static inline struct ptdesc *ptdesc_alloc(gfp_t gfp, unsigned int order)
+{
+ struct page *page = alloc_pages(gfp | __GFP_COMP, order);
+
+ return page_ptdesc(page);
+}
+
+static inline void ptdesc_free(struct ptdesc *pt)
+{
+ struct page *page = ptdesc_page(pt);
+
+ __free_pages(page, compound_order(page));
+}
+
+static inline void ptdesc_clear(void *x)
+{
+ clear_page(x);
+}
+
#if USE_SPLIT_PTE_PTLOCKS
#if ALLOC_SPLIT_PTLOCKS
void __init ptlock_cache_init(void);
@@ -2970,6 +3009,11 @@ static inline void mark_page_reserved(struct page *page)
adjust_managed_page_count(page, -1);
}
+static inline void free_reserved_ptdesc(struct ptdesc *pt)
+{
+ free_reserved_page(ptdesc_page(pt));
+}
+
/*
* Default method to free all the __init memory into the buddy system.
* The freed pages will be poisoned with pattern "poison" if it's within
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 5e0f51308724..b067ac10f3dd 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1041,6 +1041,18 @@ TABLE_MATCH(ptl, ptl);
#undef TABLE_MATCH
static_assert(sizeof(struct ptdesc) <= sizeof(struct page));
+#define ptdesc_page(pt)(_Generic((pt),
\
+ const struct ptdesc *: (const struct page *)(pt), \
+ struct ptdesc *:(struct page *)(pt)))
+
+#define ptdesc_folio(pt) (_Generic((pt), \
+ const struct ptdesc *: (const struct folio *)(pt), \
+ struct ptdesc *:(struct folio *)(pt)))
+
+#define page_ptdesc(p) (_Generic((p), \
+ const struct page *:(const struct ptdesc *)(p), \
+ struct page *: (struct ptdesc *)(p)))
+
/*
* No-op macros that just return the current protection value. Defined here
* because these macros can be used even if CONFIG_MMU is not defined.
--
2.39.2
[PATCH v2 04/34] pgtable: Create struct ptdesc
Currently, page table information is stored within struct page. As part
of simplifying struct page, create struct ptdesc for page table
information.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/pgtable.h | 52 +
1 file changed, 52 insertions(+)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 023918666dd4..5e0f51308724 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -989,6 +989,58 @@ static inline void ptep_modify_prot_commit(struct
vm_area_struct *vma,
#endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */
#endif /* CONFIG_MMU */
+
+/**
+ * struct ptdesc - Memory descriptor for page tables.
+ * @__page_flags: Same as page flags. Unused for page tables.
+ * @pt_list: List of used page tables. Used for s390 and x86.
+ * @_pt_pad_1: Padding that aliases with page's compound head.
+ * @pmd_huge_pte: Protected by ptdesc->ptl, used for THPs.
+ * @_pt_s390_gaddr: Aliases with page's mapping. Used for s390 gmap only.
+ * @pt_mm: Used for x86 pgds.
+ * @pt_frag_refcount: For fragmented page table tracking. Powerpc and s390
only.
+ * @ptl: Lock for the page table.
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues.
+ */
+struct ptdesc {
+ unsigned long __page_flags;
+
+ union {
+ struct list_head pt_list;
+ struct {
+ unsigned long _pt_pad_1;
+ pgtable_t pmd_huge_pte;
+ };
+ };
+ unsigned long _pt_s390_gaddr;
+
+ union {
+ struct mm_struct *pt_mm;
+ atomic_t pt_frag_refcount;
+ unsigned long index;
+ };
+
+#if ALLOC_SPLIT_PTLOCKS
+ spinlock_t *ptl;
+#else
+ spinlock_t ptl;
+#endif
+};
+
+#define TABLE_MATCH(pg, pt)\
+ static_assert(offsetof(struct page, pg) == offsetof(struct ptdesc, pt))
+TABLE_MATCH(flags, __page_flags);
+TABLE_MATCH(compound_head, pt_list);
+TABLE_MATCH(compound_head, _pt_pad_1);
+TABLE_MATCH(pmd_huge_pte, pmd_huge_pte);
+TABLE_MATCH(mapping, _pt_s390_gaddr);
+TABLE_MATCH(pt_mm, pt_mm);
+TABLE_MATCH(ptl, ptl);
+#undef TABLE_MATCH
+static_assert(sizeof(struct ptdesc) <= sizeof(struct page));
+
/*
* No-op macros that just return the current protection value. Defined here
* because these macros can be used even if CONFIG_MMU is not defined.
--
2.39.2
[PATCH v2 03/34] s390: Use pt_frag_refcount for pagetables
s390 currently uses _refcount to identify fragmented page tables.
The page table struct already has a member pt_frag_refcount used by
powerpc, so have s390 use that instead of the _refcount field as well.
This improves the safety for _refcount and the page table tracking.
This also allows us to simplify the tracking since we can once again use
the lower byte of pt_frag_refcount instead of the upper byte of _refcount.
Signed-off-by: Vishal Moola (Oracle)
---
arch/s390/mm/pgalloc.c | 38 +++---
1 file changed, 15 insertions(+), 23 deletions(-)
diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c
index 66ab68db9842..6b99932abc66 100644
--- a/arch/s390/mm/pgalloc.c
+++ b/arch/s390/mm/pgalloc.c
@@ -182,20 +182,17 @@ void page_table_free_pgste(struct page *page)
* As follows from the above, no unallocated or fully allocated parent
* pages are contained in mm_context_t::pgtable_list.
*
- * The upper byte (bits 24-31) of the parent page _refcount is used
+ * The lower byte (bits 0-7) of the parent page pt_frag_refcount is used
* for tracking contained 2KB-pgtables and has the following format:
*
* PP AA
- * 01234567upper byte (bits 24-31) of struct page::_refcount
+ * 01234567upper byte (bits 0-7) of struct page::pt_frag_refcount
* || ||
* || |+--- upper 2KB-pgtable is allocated
* || + lower 2KB-pgtable is allocated
* |+--- upper 2KB-pgtable is pending for removal
* + lower 2KB-pgtable is pending for removal
*
- * (See commit 620b4e903179 ("s390: use _refcount for pgtables") on why
- * using _refcount is possible).
- *
* When 2KB-pgtable is allocated the corresponding AA bit is set to 1.
* The parent page is either:
* - added to mm_context_t::pgtable_list in case the second half of the
@@ -243,11 +240,12 @@ unsigned long *page_table_alloc(struct mm_struct *mm)
if (!list_empty(>context.pgtable_list)) {
page = list_first_entry(>context.pgtable_list,
struct page, lru);
- mask = atomic_read(>_refcount) >> 24;
+ mask = atomic_read(>pt_frag_refcount);
/*
* The pending removal bits must also be checked.
* Failure to do so might lead to an impossible
-* value of (i.e 0x13 or 0x23) written to _refcount.
+* value of (i.e 0x13 or 0x23) written to
+* pt_frag_refcount.
* Such values violate the assumption that pending and
* allocation bits are mutually exclusive, and the rest
* of the code unrails as result. That could lead to
@@ -259,8 +257,8 @@ unsigned long *page_table_alloc(struct mm_struct *mm)
bit = mask & 1; /* =1 -> second 2K */
if (bit)
table += PTRS_PER_PTE;
- atomic_xor_bits(>_refcount,
- 0x01U << (bit + 24));
+ atomic_xor_bits(>pt_frag_refcount,
+ 0x01U << bit);
list_del(>lru);
}
}
@@ -281,12 +279,12 @@ unsigned long *page_table_alloc(struct mm_struct *mm)
table = (unsigned long *) page_to_virt(page);
if (mm_alloc_pgste(mm)) {
/* Return 4K page table with PGSTEs */
- atomic_xor_bits(>_refcount, 0x03U << 24);
+ atomic_xor_bits(>pt_frag_refcount, 0x03U);
memset64((u64 *)table, _PAGE_INVALID, PTRS_PER_PTE);
memset64((u64 *)table + PTRS_PER_PTE, 0, PTRS_PER_PTE);
} else {
/* Return the first 2K fragment of the page */
- atomic_xor_bits(>_refcount, 0x01U << 24);
+ atomic_xor_bits(>pt_frag_refcount, 0x01U);
memset64((u64 *)table, _PAGE_INVALID, 2 * PTRS_PER_PTE);
spin_lock_bh(>context.lock);
list_add(>lru, >context.pgtable_list);
@@ -323,22 +321,19 @@ void page_table_free(struct mm_struct *mm, unsigned long
*table)
* will happen outside of the critical section from this
* function or from __tlb_remove_table()
*/
- mask = atomic_xor_bits(>_refcount, 0x11U << (bit + 24));
- mask >>= 24;
+ mask = atomic_xor_bits(>pt_frag_refcount, 0x11U << bit);
if (mask & 0x03U)
list_add(>lru, >context.pgtable_list);
else
list_del(>lru);
spin_unlock_bh(>context.lock);
- mask = atomic_xor_bits(>_refcount, 0x10U << (bit + 24));
-
[PATCH v2 02/34] s390: Use _pt_s390_gaddr for gmap address tracking
s390 uses page->index to keep track of page tables for the guest address
space. In an attempt to consolidate the usage of page fields in s390,
replace _pt_pad_2 with _pt_s390_gaddr to replace page->index in gmap.
This will help with the splitting of struct ptdesc from struct page, as
well as allow s390 to use _pt_frag_refcount for fragmented page table
tracking.
Since page->_pt_s390_gaddr aliases with mapping, ensure its set to NULL
before freeing the pages as well.
This also reverts commit 7e25de77bc5ea ("s390/mm: use pmd_pgtable_page()
helper in __gmap_segment_gaddr()") which had s390 use
pmd_pgtable_page() to get a gmap page table, as pmd_pgtable_page()
should be used for more generic process page tables.
Signed-off-by: Vishal Moola (Oracle)
---
arch/s390/mm/gmap.c | 56 +++-
include/linux/mm_types.h | 2 +-
2 files changed, 39 insertions(+), 19 deletions(-)
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index dfe905c7bd8e..a9e8b1805894 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -70,7 +70,7 @@ static struct gmap *gmap_alloc(unsigned long limit)
page = alloc_pages(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER);
if (!page)
goto out_free;
- page->index = 0;
+ page->_pt_s390_gaddr = 0;
list_add(>lru, >crst_list);
table = page_to_virt(page);
crst_table_init(table, etype);
@@ -187,16 +187,20 @@ static void gmap_free(struct gmap *gmap)
if (!(gmap_is_shadow(gmap) && gmap->removed))
gmap_flush_tlb(gmap);
/* Free all segment & region tables. */
- list_for_each_entry_safe(page, next, >crst_list, lru)
+ list_for_each_entry_safe(page, next, >crst_list, lru) {
+ page->_pt_s390_gaddr = 0;
__free_pages(page, CRST_ALLOC_ORDER);
+ }
gmap_radix_tree_free(>guest_to_host);
gmap_radix_tree_free(>host_to_guest);
/* Free additional data for a shadow gmap */
if (gmap_is_shadow(gmap)) {
/* Free all page tables. */
- list_for_each_entry_safe(page, next, >pt_list, lru)
+ list_for_each_entry_safe(page, next, >pt_list, lru) {
+ page->_pt_s390_gaddr = 0;
page_table_free_pgste(page);
+ }
gmap_rmap_radix_tree_free(>host_to_rmap);
/* Release reference to the parent */
gmap_put(gmap->parent);
@@ -318,12 +322,14 @@ static int gmap_alloc_table(struct gmap *gmap, unsigned
long *table,
list_add(>lru, >crst_list);
*table = __pa(new) | _REGION_ENTRY_LENGTH |
(*table & _REGION_ENTRY_TYPE_MASK);
- page->index = gaddr;
+ page->_pt_s390_gaddr = gaddr;
page = NULL;
}
spin_unlock(>guest_table_lock);
- if (page)
+ if (page) {
+ page->_pt_s390_gaddr = 0;
__free_pages(page, CRST_ALLOC_ORDER);
+ }
return 0;
}
@@ -336,12 +342,14 @@ static int gmap_alloc_table(struct gmap *gmap, unsigned
long *table,
static unsigned long __gmap_segment_gaddr(unsigned long *entry)
{
struct page *page;
- unsigned long offset;
+ unsigned long offset, mask;
offset = (unsigned long) entry / sizeof(unsigned long);
offset = (offset & (PTRS_PER_PMD - 1)) * PMD_SIZE;
- page = pmd_pgtable_page((pmd_t *) entry);
- return page->index + offset;
+ mask = ~(PTRS_PER_PMD * sizeof(pmd_t) - 1);
+ page = virt_to_page((void *)((unsigned long) entry & mask));
+
+ return page->_pt_s390_gaddr + offset;
}
/**
@@ -1351,6 +1359,7 @@ static void gmap_unshadow_pgt(struct gmap *sg, unsigned
long raddr)
/* Free page table */
page = phys_to_page(pgt);
list_del(>lru);
+ page->_pt_s390_gaddr = 0;
page_table_free_pgste(page);
}
@@ -1379,6 +1388,7 @@ static void __gmap_unshadow_sgt(struct gmap *sg, unsigned
long raddr,
/* Free page table */
page = phys_to_page(pgt);
list_del(>lru);
+ page->_pt_s390_gaddr = 0;
page_table_free_pgste(page);
}
}
@@ -1409,6 +1419,7 @@ static void gmap_unshadow_sgt(struct gmap *sg, unsigned
long raddr)
/* Free segment table */
page = phys_to_page(sgt);
list_del(>lru);
+ page->_pt_s390_gaddr = 0;
__free_pages(page, CRST_ALLOC_ORDER);
}
@@ -1437,6 +1448,7 @@ static void __gmap_unshadow_r3t(struct gmap *sg, unsigned
long raddr,
/* Free segment table */
page = phys_to_page(sgt);
list_del(>lru);
+ page->_pt_s390_gaddr = 0;
__free_pages(page, CRST_ALLOC_ORDER);
}
}
@@ -1467,6 +1479,7 @@ static void gmap_unshadow_r3t(struct gmap *sg, unsigned
long raddr)
/* Free region 3 table */
[PATCH v2 01/34] mm: Add PAGE_TYPE_OP folio functions
No folio equivalents for page type operations have been defined, so
define them for later folio conversions.
Also changes the Page##uname macros to take in const struct page* since
we only read the memory here.
Signed-off-by: Vishal Moola (Oracle)
---
include/linux/page-flags.h | 20 ++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 1c68d67b832f..607b495d1b57 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -902,6 +902,8 @@ static inline bool is_page_hwpoison(struct page *page)
#define PageType(page, flag) \
((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE)
+#define folio_test_type(folio, flag) \
+ ((folio->page.page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE)
static inline int page_type_has_type(unsigned int page_type)
{
@@ -914,20 +916,34 @@ static inline int page_has_type(struct page *page)
}
#define PAGE_TYPE_OPS(uname, lname)\
-static __always_inline int Page##uname(struct page *page) \
+static __always_inline int Page##uname(const struct page *page)
\
{ \
return PageType(page, PG_##lname); \
} \
+static __always_inline int folio_test_##lname(const struct folio *folio)\
+{ \
+ return folio_test_type(folio, PG_##lname); \
+} \
static __always_inline void __SetPage##uname(struct page *page)
\
{ \
VM_BUG_ON_PAGE(!PageType(page, 0), page); \
page->page_type &= ~PG_##lname; \
} \
+static __always_inline void __folio_set_##lname(struct folio *folio) \
+{ \
+ VM_BUG_ON_FOLIO(!folio_test_type(folio, 0), folio); \
+ folio->page.page_type &= ~PG_##lname; \
+} \
static __always_inline void __ClearPage##uname(struct page *page) \
{ \
VM_BUG_ON_PAGE(!Page##uname(page), page); \
page->page_type |= PG_##lname; \
-}
+} \
+static __always_inline void __folio_clear_##lname(struct folio *folio) \
+{ \
+ VM_BUG_ON_FOLIO(!folio_test_##lname(folio), folio); \
+ folio->page.page_type |= PG_##lname;\
+} \
/*
* PageBuddy() indicates that the page is free and in the buddy system
--
2.39.2
[PATCH v2 00/34] Split ptdesc from struct page
The MM subsystem is trying to shrink struct page. This patchset
introduces a memory descriptor for page table tracking - struct ptdesc.
This patchset introduces ptdesc, splits ptdesc from struct page, and
converts many callers of page table constructor/destructors to use ptdescs.
Ptdesc is a foundation to further standardize page tables, and eventually
allow for dynamic allocation of page tables independent of struct page.
However, the use of pages for page table tracking is quite deeply
ingrained and varied across archictectures, so there is still a lot of
work to be done before that can happen.
This is rebased on next-20230428.
v2:
Fix a lot of compiler warning/errors
Moved definition of ptdesc to outside CONFIG_MMU
Revert commit 7e25de77bc5ea which had gmap use pmd_pgtable_page()
Allow functions to preserve const-ness where applicable
Define folio equivalents for PAGE_TYPE_OPS page functions
Vishal Moola (Oracle) (34):
mm: Add PAGE_TYPE_OP folio functions
s390: Use _pt_s390_gaddr for gmap address tracking
s390: Use pt_frag_refcount for pagetables
pgtable: Create struct ptdesc
mm: add utility functions for ptdesc
mm: Convert pmd_pgtable_page() to pmd_ptdesc()
mm: Convert ptlock_alloc() to use ptdescs
mm: Convert ptlock_ptr() to use ptdescs
mm: Convert pmd_ptlock_init() to use ptdescs
mm: Convert ptlock_init() to use ptdescs
mm: Convert pmd_ptlock_free() to use ptdescs
mm: Convert ptlock_free() to use ptdescs
mm: Create ptdesc equivalents for pgtable_{pte,pmd}_page_{ctor,dtor}
powerpc: Convert various functions to use ptdescs
x86: Convert various functions to use ptdescs
s390: Convert various gmap functions to use ptdescs
s390: Convert various pgalloc functions to use ptdescs
mm: Remove page table members from struct page
pgalloc: Convert various functions to use ptdescs
arm: Convert various functions to use ptdescs
arm64: Convert various functions to use ptdescs
csky: Convert __pte_free_tlb() to use ptdescs
hexagon: Convert __pte_free_tlb() to use ptdescs
loongarch: Convert various functions to use ptdescs
m68k: Convert various functions to use ptdescs
mips: Convert various functions to use ptdescs
nios2: Convert __pte_free_tlb() to use ptdescs
openrisc: Convert __pte_free_tlb() to use ptdescs
riscv: Convert alloc_{pmd, pte}_late() to use ptdescs
sh: Convert pte_free_tlb() to use ptdescs
sparc64: Convert various functions to use ptdescs
sparc: Convert pgtable_pte_page_{ctor, dtor}() to ptdesc equivalents
um: Convert {pmd, pte}_free_tlb() to use ptdescs
mm: Remove pgtable_{pmd, pte}_page_{ctor, dtor}() wrappers
Documentation/mm/split_page_table_lock.rst| 12 +-
.../zh_CN/mm/split_page_table_lock.rst| 14 +-
arch/arm/include/asm/tlb.h| 12 +-
arch/arm/mm/mmu.c | 6 +-
arch/arm64/include/asm/tlb.h | 14 +-
arch/arm64/mm/mmu.c | 7 +-
arch/csky/include/asm/pgalloc.h | 4 +-
arch/hexagon/include/asm/pgalloc.h| 8 +-
arch/loongarch/include/asm/pgalloc.h | 27 ++-
arch/loongarch/mm/pgtable.c | 7 +-
arch/m68k/include/asm/mcf_pgalloc.h | 41 ++--
arch/m68k/include/asm/sun3_pgalloc.h | 8 +-
arch/m68k/mm/motorola.c | 4 +-
arch/mips/include/asm/pgalloc.h | 31 +--
arch/mips/mm/pgtable.c| 7 +-
arch/nios2/include/asm/pgalloc.h | 8 +-
arch/openrisc/include/asm/pgalloc.h | 8 +-
arch/powerpc/mm/book3s64/mmu_context.c| 10 +-
arch/powerpc/mm/book3s64/pgtable.c| 32 +--
arch/powerpc/mm/pgtable-frag.c| 46 ++--
arch/riscv/include/asm/pgalloc.h | 8 +-
arch/riscv/mm/init.c | 16 +-
arch/s390/include/asm/pgalloc.h | 4 +-
arch/s390/include/asm/tlb.h | 4 +-
arch/s390/mm/gmap.c | 222 +++---
arch/s390/mm/pgalloc.c| 126 +-
arch/sh/include/asm/pgalloc.h | 9 +-
arch/sparc/mm/init_64.c | 17 +-
arch/sparc/mm/srmmu.c | 5 +-
arch/um/include/asm/pgalloc.h | 18 +-
arch/x86/mm/pgtable.c | 46 ++--
arch/x86/xen/mmu_pv.c | 2 +-
include/asm-generic/pgalloc.h | 62 +++--
include/asm-generic/tlb.h | 11 +
include/linux/mm.h| 138 +++
include/linux/mm_types.h | 14 --
include/linux/page-flags.h| 20 +-
include/linux/pgtable.h | 61 +
mm/memory.c | 8 +-
39 files changed, 648 insertions(+), 449 deletions(-)
--
2.39.2
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
Hey Ricardo,
On Mon, May 01, 2023 at 02:38:22PM +0200, Ricardo Ribalda wrote:
> If PGO is enabled, the purgatory ends up with multiple .text sections.
> This is not supported by kexec and crashes the system.
>
> Cc: sta...@vger.kernel.org
> Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
> Signed-off-by: Ricardo Ribalda
> ---
> arch/riscv/purgatory/Makefile | 5 +
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
> index 5730797a6b40..cf3a44121a90 100644
> --- a/arch/riscv/purgatory/Makefile
> +++ b/arch/riscv/purgatory/Makefile
> @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> CFLAGS_string.o := -D__DISABLE_EXPORTS
> CFLAGS_ctype.o := -D__DISABLE_EXPORTS
>
> +# When profile optimization is enabled, llvm emits two different overlapping
> +# text sections, which is not supported by kexec. Remove profile optimization
> +# flags.
> +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> -fprofile-use=%,$(KBUILD_CFLAGS))
With the caveat of not being au fait with the workings of either PGO or
of purgatory, how come you modify KBUILD_CFLAGS here rather than the
purgatory specific PURGATORY_CFLAGS that are used later in the file?
Cheers,
Conor.
> +
> # When linking purgatory.ro with -r unresolved symbols are not checked,
> # also link a purgatory.chk binary without -r to check for unresolved
> symbols.
> PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
>
> --
> 2.40.1.495.gc816e09b53d-goog
>
>
> ___
> linux-riscv mailing list
> linux-ri...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
signature.asc
Description: PGP signature
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
Hi Conor
Fixed on my branch
https://git.kernel.org/pub/scm/linux/kernel/git/ribalda/linux.git/commit/?h=b4/kexec_clang16=1e9cda9fa638cc72581986f60b490cc069a38f75
Will submit a new version after a while :)
Thanks!
On Mon, 1 May 2023 at 19:28, Conor Dooley wrote:
>
> On Mon, May 01, 2023 at 07:18:12PM +0200, Ricardo Ribalda wrote:
> > On Mon, 1 May 2023 at 18:19, Nick Desaulniers
> > wrote:
> > >
> > > On Mon, May 1, 2023 at 5:39 AM Ricardo Ribalda
> > > wrote:
> > > >
> > > > If PGO is enabled, the purgatory ends up with multiple .text sections.
> > > > This is not supported by kexec and crashes the system.
> > > >
> > > > Cc: sta...@vger.kernel.org
> > > > Fixes: 930457057abe ("kernel/kexec_file.c: split up
> > > > __kexec_load_puragory")
> > > > Signed-off-by: Ricardo Ribalda
> > >
> > > Hi Ricardo,
> > > Thanks for the series. Does this patch 4/4 need a new online commit
> > > description? It's not adding a linker script (maybe an earlier version
> > > was).
>
> > Thanks for catching this. It should have said
> >
> > risc/purgatory: Remove profile optimization flags
> ^^
> Perhaps with the omitted v added too?
>
> Also while playing the $subject nitpicking game, is it not called
> "profile**-guided** optimisation" (and ditto in the comments)?
>
> Cheers,
> Conor.
>
> > Will fix it on my local branch in case there is a next version of the
> > series. Otherwise, please the maintainer fix the subject.
>
> > > > ---
> > > > arch/riscv/purgatory/Makefile | 5 +
> > > > 1 file changed, 5 insertions(+)
> > > >
> > > > diff --git a/arch/riscv/purgatory/Makefile
> > > > b/arch/riscv/purgatory/Makefile
> > > > index 5730797a6b40..cf3a44121a90 100644
> > > > --- a/arch/riscv/purgatory/Makefile
> > > > +++ b/arch/riscv/purgatory/Makefile
> > > > @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> > > > CFLAGS_string.o := -D__DISABLE_EXPORTS
> > > > CFLAGS_ctype.o := -D__DISABLE_EXPORTS
> > > >
> > > > +# When profile optimization is enabled, llvm emits two different
> > > > overlapping
> > > > +# text sections, which is not supported by kexec. Remove profile
> > > > optimization
> > > > +# flags.
> > > > +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> > > > -fprofile-use=%,$(KBUILD_CFLAGS))
> > > > +
> > > > # When linking purgatory.ro with -r unresolved symbols are not checked,
> > > > # also link a purgatory.chk binary without -r to check for unresolved
> > > > symbols.
> > > > PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
> > > >
> > > > --
> > > > 2.40.1.495.gc816e09b53d-goog
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > > ~Nick Desaulniers
> >
> >
> >
> > --
> > Ricardo Ribalda
> >
> > ___
> > linux-riscv mailing list
> > linux-ri...@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv
--
Ricardo Ribalda
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
On Mon, May 01, 2023 at 07:18:12PM +0200, Ricardo Ribalda wrote:
> On Mon, 1 May 2023 at 18:19, Nick Desaulniers wrote:
> >
> > On Mon, May 1, 2023 at 5:39 AM Ricardo Ribalda wrote:
> > >
> > > If PGO is enabled, the purgatory ends up with multiple .text sections.
> > > This is not supported by kexec and crashes the system.
> > >
> > > Cc: sta...@vger.kernel.org
> > > Fixes: 930457057abe ("kernel/kexec_file.c: split up
> > > __kexec_load_puragory")
> > > Signed-off-by: Ricardo Ribalda
> >
> > Hi Ricardo,
> > Thanks for the series. Does this patch 4/4 need a new online commit
> > description? It's not adding a linker script (maybe an earlier version
> > was).
> Thanks for catching this. It should have said
>
> risc/purgatory: Remove profile optimization flags
^^
Perhaps with the omitted v added too?
Also while playing the $subject nitpicking game, is it not called
"profile**-guided** optimisation" (and ditto in the comments)?
Cheers,
Conor.
> Will fix it on my local branch in case there is a next version of the
> series. Otherwise, please the maintainer fix the subject.
> > > ---
> > > arch/riscv/purgatory/Makefile | 5 +
> > > 1 file changed, 5 insertions(+)
> > >
> > > diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
> > > index 5730797a6b40..cf3a44121a90 100644
> > > --- a/arch/riscv/purgatory/Makefile
> > > +++ b/arch/riscv/purgatory/Makefile
> > > @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> > > CFLAGS_string.o := -D__DISABLE_EXPORTS
> > > CFLAGS_ctype.o := -D__DISABLE_EXPORTS
> > >
> > > +# When profile optimization is enabled, llvm emits two different
> > > overlapping
> > > +# text sections, which is not supported by kexec. Remove profile
> > > optimization
> > > +# flags.
> > > +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> > > -fprofile-use=%,$(KBUILD_CFLAGS))
> > > +
> > > # When linking purgatory.ro with -r unresolved symbols are not checked,
> > > # also link a purgatory.chk binary without -r to check for unresolved
> > > symbols.
> > > PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
> > >
> > > --
> > > 2.40.1.495.gc816e09b53d-goog
> > >
> >
> >
> > --
> > Thanks,
> > ~Nick Desaulniers
>
>
>
> --
> Ricardo Ribalda
>
> ___
> linux-riscv mailing list
> linux-ri...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv
signature.asc
Description: PGP signature
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
Hi Nick
Thanks for catching this. It should have said
risc/purgatory: Remove profile optimization flags
Will fix it on my local branch in case there is a next version of the
series. Otherwise, please the maintainer fix the subject.
Thanks!
On Mon, 1 May 2023 at 18:19, Nick Desaulniers wrote:
>
> On Mon, May 1, 2023 at 5:39 AM Ricardo Ribalda wrote:
> >
> > If PGO is enabled, the purgatory ends up with multiple .text sections.
> > This is not supported by kexec and crashes the system.
> >
> > Cc: sta...@vger.kernel.org
> > Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
> > Signed-off-by: Ricardo Ribalda
>
> Hi Ricardo,
> Thanks for the series. Does this patch 4/4 need a new online commit
> description? It's not adding a linker script (maybe an earlier version
> was).
>
> > ---
> > arch/riscv/purgatory/Makefile | 5 +
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
> > index 5730797a6b40..cf3a44121a90 100644
> > --- a/arch/riscv/purgatory/Makefile
> > +++ b/arch/riscv/purgatory/Makefile
> > @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> > CFLAGS_string.o := -D__DISABLE_EXPORTS
> > CFLAGS_ctype.o := -D__DISABLE_EXPORTS
> >
> > +# When profile optimization is enabled, llvm emits two different
> > overlapping
> > +# text sections, which is not supported by kexec. Remove profile
> > optimization
> > +# flags.
> > +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> > -fprofile-use=%,$(KBUILD_CFLAGS))
> > +
> > # When linking purgatory.ro with -r unresolved symbols are not checked,
> > # also link a purgatory.chk binary without -r to check for unresolved
> > symbols.
> > PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
> >
> > --
> > 2.40.1.495.gc816e09b53d-goog
> >
>
>
> --
> Thanks,
> ~Nick Desaulniers
--
Ricardo Ribalda
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
On Mon, May 1, 2023 at 5:39 AM Ricardo Ribalda wrote:
>
> If PGO is enabled, the purgatory ends up with multiple .text sections.
> This is not supported by kexec and crashes the system.
>
> Cc: sta...@vger.kernel.org
> Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
> Signed-off-by: Ricardo Ribalda
Hi Ricardo,
Thanks for the series. Does this patch 4/4 need a new online commit
description? It's not adding a linker script (maybe an earlier version
was).
> ---
> arch/riscv/purgatory/Makefile | 5 +
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
> index 5730797a6b40..cf3a44121a90 100644
> --- a/arch/riscv/purgatory/Makefile
> +++ b/arch/riscv/purgatory/Makefile
> @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
> CFLAGS_string.o := -D__DISABLE_EXPORTS
> CFLAGS_ctype.o := -D__DISABLE_EXPORTS
>
> +# When profile optimization is enabled, llvm emits two different overlapping
> +# text sections, which is not supported by kexec. Remove profile optimization
> +# flags.
> +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> -fprofile-use=%,$(KBUILD_CFLAGS))
> +
> # When linking purgatory.ro with -r unresolved symbols are not checked,
> # also link a purgatory.chk binary without -r to check for unresolved
> symbols.
> PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
>
> --
> 2.40.1.495.gc816e09b53d-goog
>
--
Thanks,
~Nick Desaulniers
Re: [PATCH v14 00/15] phy: Add support for Lynx 10G SerDes
On 4/29/23 13:24, Vladimir Oltean wrote: > On Wed, Apr 26, 2023 at 10:50:17AM -0400, Sean Anderson wrote: >> > I need to catch up with 14 rounds of patches from you and with the >> > discussions that took place on each version, and understand how you >> > responded to feedback like "don't remove PHY interrupts without finding >> > out why they don't work" >> >> All I can say is that >> >> - It doesn't work on my board >> - The traces are on the bottom of the PCB >> - The signal goes through an FPGA which (unlike the LS1046ARDB) is >> closed-source > > I don't understand the distinction you are making here. Are the sources > for QIXIS bit streams public for any Layerscape board? Correct. The sources for the LS1046ARDB QIXIS are available for download. >> - The alternative is polling once a second (not terribly intensive) > > It makes a difference to performance (forwarded packets per second), believe > it or not. I don't. Please elaborate how link status latency from the phy affects performance. >> >> I think it's very reasonable to make this change. Anyway, it's in a separate >> patch so that it can be applied independently. > > Perhaps better phrased: "discussed separately"... > >> > Even if the SERDES and PLL drivers "work for you" in the current form, >> > I doubt the usefulness of a PLL driver if you have to disconnect the >> > SoC's reset request signal on the board to not be stuck in a reboot loop. >> >> I would like to emphasize that this has *nothing to do with this driver*. >> This behavior is part of the boot ROM (or something like it) and occurs >> before >> any user code has ever executed. The problem of course is that certain RCWs >> expect the reference clocks to be in certain (incompatible) configurations, >> and will fail the boot without a lock. I think this is rather silly (since >> you only need PLL lock when you actually want to use the serdes), but that's >> how it is. And of course, this is only necessary because I was unable to get >> major reconfiguration to work. In an ideal world, you could always boot with >> the same RCW (with PLL config matching the board) and choose the major >> protocol >> at runtime. > > Could you please tell me what are the reference clock frequencies that > your board provides at boot time to the 2 PLLs, and which SERDES > protocol out of those 2 (1133 and ) boots correctly (no RESET_REQ > hacks necessary) with those refclks? I will try to get a LS1046A-QDS > where I boot from the same refclk + SERDES protocol configuration as > you, and use PBI commands in the RCW to reconfigure the lanes (PLL > selection and protocol registers) for the other mode, while keeping the > FRATE_SEL of the PLLs unmodified. From table 31-1 in the RM, the PLL mapping for 1133 is 2211, and the PLL mapping for is . As a consequence, for 1133, PLL 2 must be 156.25 MHz and PLL 1 must be either 100 or 125 MHz. And for , PLL 2 must be either 100 or 125 MHz, and PLL 1 should be shut down (as it is unused). This conflict for PLL 2 means that the same reference clock configuration cannot work for both 1133 and . In one of the configurations, SRDS_RST_RR will be set in RSTRQSR1. On our board, reference clock 1 is 156.25 MHz, and reference clock 2 is 125 MHz. Therefore, will fail to boot. Unfortunately, this reset request occurs before any user-configurable code has run (except the RCW), so it is not possible to fix this issue with e.g. PBI. --Sean not
[PATCH v2 1/1] PCI: layerscape: Add the endpoint linkup notifier support
Layerscape has PME interrupt, which can be used as linkup notifier.
Set CFG_READY bit when linkup detected.
Signed-off-by: Xiaowei Bao
Signed-off-by: Frank Li
---
Change from v1 to v2
- pme -> PME
- irq -> IRQ
- update dev_info message according to Bjorn's suggestion
- remove '.' at error message
.../pci/controller/dwc/pci-layerscape-ep.c| 104 +-
1 file changed, 103 insertions(+), 1 deletion(-)
diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index c640db60edc6..e974fbe3b6d8 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -18,6 +18,20 @@
#include "pcie-designware.h"
+#define PEX_PF0_CONFIG 0xC0014
+#define PEX_PF0_CFG_READY BIT(0)
+
+/* PEX PFa PCIE PME and message interrupt registers*/
+#define PEX_PF0_PME_MES_DR 0xC0020
+#define PEX_PF0_PME_MES_DR_LUD BIT(7)
+#define PEX_PF0_PME_MES_DR_LDD BIT(9)
+#define PEX_PF0_PME_MES_DR_HRD BIT(10)
+
+#define PEX_PF0_PME_MES_IER0xC0028
+#define PEX_PF0_PME_MES_IER_LUDIE BIT(7)
+#define PEX_PF0_PME_MES_IER_LDDIE BIT(9)
+#define PEX_PF0_PME_MES_IER_HRDIE BIT(10)
+
#define to_ls_pcie_ep(x) dev_get_drvdata((x)->dev)
struct ls_pcie_ep_drvdata {
@@ -30,8 +44,88 @@ struct ls_pcie_ep {
struct dw_pcie *pci;
struct pci_epc_features *ls_epc;
const struct ls_pcie_ep_drvdata *drvdata;
+ boolbig_endian;
+ int irq;
};
+static u32 ls_lut_readl(struct ls_pcie_ep *pcie, u32 offset)
+{
+ struct dw_pcie *pci = pcie->pci;
+
+ if (pcie->big_endian)
+ return ioread32be(pci->dbi_base + offset);
+ else
+ return ioread32(pci->dbi_base + offset);
+}
+
+static void ls_lut_writel(struct ls_pcie_ep *pcie, u32 offset,
+ u32 value)
+{
+ struct dw_pcie *pci = pcie->pci;
+
+ if (pcie->big_endian)
+ iowrite32be(value, pci->dbi_base + offset);
+ else
+ iowrite32(value, pci->dbi_base + offset);
+}
+
+static irqreturn_t ls_pcie_ep_event_handler(int irq, void *dev_id)
+{
+ struct ls_pcie_ep *pcie = (struct ls_pcie_ep *)dev_id;
+ struct dw_pcie *pci = pcie->pci;
+ u32 val, cfg;
+
+ val = ls_lut_readl(pcie, PEX_PF0_PME_MES_DR);
+ if (!val)
+ return IRQ_NONE;
+
+ if (val & PEX_PF0_PME_MES_DR_LUD) {
+ cfg = ls_lut_readl(pcie, PEX_PF0_CONFIG);
+ cfg |= PEX_PF0_CFG_READY;
+ ls_lut_writel(pcie, PEX_PF0_CONFIG, cfg);
+ dw_pcie_ep_linkup(>ep);
+
+ dev_info(pci->dev, "Link up\n");
+ } else if (val & PEX_PF0_PME_MES_DR_LDD) {
+ dev_info(pci->dev, "Link down\n");
+ } else if (val & PEX_PF0_PME_MES_DR_HRD) {
+ dev_info(pci->dev, "Hot reset\n");
+ }
+
+ ls_lut_writel(pcie, PEX_PF0_PME_MES_DR, val);
+
+ return IRQ_HANDLED;
+}
+
+static int ls_pcie_ep_interrupt_init(struct ls_pcie_ep *pcie,
+struct platform_device *pdev)
+{
+ u32 val;
+ int ret;
+
+ pcie->irq = platform_get_irq_byname(pdev, "pme");
+ if (pcie->irq < 0) {
+ dev_err(>dev, "Can't get 'pme' IRQ\n");
+ return pcie->irq;
+ }
+
+ ret = devm_request_irq(>dev, pcie->irq,
+ ls_pcie_ep_event_handler, IRQF_SHARED,
+ pdev->name, pcie);
+ if (ret) {
+ dev_err(>dev, "Can't register PCIe IRQ\n");
+ return ret;
+ }
+
+ /* Enable interrupts */
+ val = ls_lut_readl(pcie, PEX_PF0_PME_MES_IER);
+ val |= PEX_PF0_PME_MES_IER_LDDIE | PEX_PF0_PME_MES_IER_HRDIE |
+ PEX_PF0_PME_MES_IER_LUDIE;
+ ls_lut_writel(pcie, PEX_PF0_PME_MES_IER, val);
+
+ return 0;
+}
+
static const struct pci_epc_features*
ls_pcie_ep_get_features(struct dw_pcie_ep *ep)
{
@@ -125,6 +219,7 @@ static int __init ls_pcie_ep_probe(struct platform_device
*pdev)
struct ls_pcie_ep *pcie;
struct pci_epc_features *ls_epc;
struct resource *dbi_base;
+ int ret;
pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
if (!pcie)
@@ -144,6 +239,7 @@ static int __init ls_pcie_ep_probe(struct platform_device
*pdev)
pci->ops = pcie->drvdata->dw_pcie_ops;
ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4);
+ ls_epc->linkup_notifier = true;
pcie->pci = pci;
pcie->ls_epc = ls_epc;
@@ -155,9 +251,15 @@ static int __init ls_pcie_ep_probe(struct platform_device
*pdev)
pci->ep.ops = _pcie_ep_ops;
+ pcie->big_endian = of_property_read_bool(dev->of_node, "big-endian");
+
platform_set_drvdata(pdev, pcie);
Re: [PATCH v6 4/4] risc/purgatory: Add linker script
On Mon, 01 May 2023 05:38:22 PDT (-0700), riba...@chromium.org wrote:
If PGO is enabled, the purgatory ends up with multiple .text sections.
This is not supported by kexec and crashes the system.
Cc: sta...@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda
---
arch/riscv/purgatory/Makefile | 5 +
1 file changed, 5 insertions(+)
diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
index 5730797a6b40..cf3a44121a90 100644
--- a/arch/riscv/purgatory/Makefile
+++ b/arch/riscv/purgatory/Makefile
@@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
CFLAGS_string.o := -D__DISABLE_EXPORTS
CFLAGS_ctype.o := -D__DISABLE_EXPORTS
+# When profile optimization is enabled, llvm emits two different overlapping
+# text sections, which is not supported by kexec. Remove profile optimization
+# flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))
+
# When linking purgatory.ro with -r unresolved symbols are not checked,
# also link a purgatory.chk binary without -r to check for unresolved symbols.
PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
Acked-by: Palmer Dabbelt
[PATCH v6 4/4] risc/purgatory: Add linker script
If PGO is enabled, the purgatory ends up with multiple .text sections.
This is not supported by kexec and crashes the system.
Cc: sta...@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda
---
arch/riscv/purgatory/Makefile | 5 +
1 file changed, 5 insertions(+)
diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
index 5730797a6b40..cf3a44121a90 100644
--- a/arch/riscv/purgatory/Makefile
+++ b/arch/riscv/purgatory/Makefile
@@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
CFLAGS_string.o := -D__DISABLE_EXPORTS
CFLAGS_ctype.o := -D__DISABLE_EXPORTS
+# When profile optimization is enabled, llvm emits two different overlapping
+# text sections, which is not supported by kexec. Remove profile optimization
+# flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))
+
# When linking purgatory.ro with -r unresolved symbols are not checked,
# also link a purgatory.chk binary without -r to check for unresolved symbols.
PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
--
2.40.1.495.gc816e09b53d-goog
[PATCH v6 3/4] powerpc/purgatory: Remove profile optimization flags
If PGO is enabled, the purgatory ends up with multiple .text sections.
This is not supported by kexec and crashes the system.
Cc: sta...@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda
---
arch/powerpc/purgatory/Makefile | 5 +
1 file changed, 5 insertions(+)
diff --git a/arch/powerpc/purgatory/Makefile b/arch/powerpc/purgatory/Makefile
index 6f5e2727963c..5efb164330b2 100644
--- a/arch/powerpc/purgatory/Makefile
+++ b/arch/powerpc/purgatory/Makefile
@@ -5,6 +5,11 @@ KCSAN_SANITIZE := n
targets += trampoline_$(BITS).o purgatory.ro
+# When profile optimization is enabled, llvm emits two different overlapping
+# text sections, which is not supported by kexec. Remove profile optimization
+# flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))
+
LDFLAGS_purgatory.ro := -e purgatory_start -r --no-undefined
$(obj)/purgatory.ro: $(obj)/trampoline_$(BITS).o FORCE
--
2.40.1.495.gc816e09b53d-goog
[PATCH v6 2/4] x86/purgatory: Remove profile optimization flags
If PGO is enabled, the purgatory ends up with multiple .text sections.
This is not supported by kexec and crashes the system.
Cc: sta...@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda
---
arch/x86/purgatory/Makefile | 5 +
1 file changed, 5 insertions(+)
diff --git a/arch/x86/purgatory/Makefile b/arch/x86/purgatory/Makefile
index 82fec66d46d2..7a7a4901ed41 100644
--- a/arch/x86/purgatory/Makefile
+++ b/arch/x86/purgatory/Makefile
@@ -14,6 +14,11 @@ $(obj)/sha256.o: $(srctree)/lib/crypto/sha256.c FORCE
CFLAGS_sha256.o := -D__DISABLE_EXPORTS
+# When profile optimization is enabled, llvm emits two different overlapping
+# text sections, which is not supported by kexec. Remove profile optimization
+# flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))
+
# When linking purgatory.ro with -r unresolved symbols are not checked,
# also link a purgatory.chk binary without -r to check for unresolved symbols.
PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
--
2.40.1.495.gc816e09b53d-goog
[PATCH v6 0/4] kexec: Fix kexec_file_load for llvm16 with PGO
When upreving llvm I realised that kexec stopped working on my test platform. The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that. Signed-off-by: Ricardo Ribalda --- Changes in v6: - Replace linker script with Makefile rule. Thanks Nick - Link to v5: https://lore.kernel.org/r/20230321-kexec_clang16-v5-0-5563bf7c4...@chromium.org Changes in v5: - Add warning when multiple text sections are found. Thanks Simon! - Add Fixes tag. - Link to v4: https://lore.kernel.org/r/20230321-kexec_clang16-v4-0-1340518f9...@chromium.org Changes in v4: - Add Cc: stable - Add linker script for x86 - Add a warning when the kernel image has overlapping sections. - Link to v3: https://lore.kernel.org/r/20230321-kexec_clang16-v3-0-5f016c8d0...@chromium.org Changes in v3: - Fix initial value. Thanks Ross! - Link to v2: https://lore.kernel.org/r/20230321-kexec_clang16-v2-0-d10e5d517...@chromium.org Changes in v2: - Fix if condition. Thanks Steven!. - Update Philipp email. Thanks Baoquan. - Link to v1: https://lore.kernel.org/r/20230321-kexec_clang16-v1-0-a768fc2c7...@chromium.org --- Ricardo Ribalda (4): kexec: Support purgatories with .text.hot sections x86/purgatory: Remove profile optimization flags powerpc/purgatory: Remove profile optimization flags risc/purgatory: Add linker script arch/powerpc/purgatory/Makefile | 5 + arch/riscv/purgatory/Makefile | 5 + arch/x86/purgatory/Makefile | 5 + kernel/kexec_file.c | 14 +- 4 files changed, 28 insertions(+), 1 deletion(-) --- base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f change-id: 20230321-kexec_clang16-4510c23d129c Best regards, -- Ricardo Ribalda
[PATCH v6 1/4] kexec: Support purgatories with .text.hot sections
Clang16 links the purgatory text in two sections when PGO is in use:
[ 1] .text PROGBITS 0040
11a1 AX 0 0 16
[ 2] .rela.textRELA 3498
0648 0018 I 24 1 8
...
[17] .text.hot.PROGBITS 3220
020b AX 0 0 1
[18] .rela.text.hot. RELA 4428
0078 0018 I 2417 8
And both of them have their range [sh_addr ... sh_addr+sh_size] on the
area pointed by `e_entry`.
This causes that image->start is calculated twice, once for .text and
another time for .text.hot. The second calculation leaves image->start
in a random location.
Because of this, the system crashes immediately after:
kexec_core: Starting new kernel
Cc: sta...@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Reviewed-by: Ross Zwisler
Reviewed-by: Steven Rostedt (Google)
Reviewed-by: Philipp Rudo
Signed-off-by: Ricardo Ribalda
---
kernel/kexec_file.c | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index f989f5f1933b..69ee4a29136f 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -901,10 +901,22 @@ static int kexec_purgatory_setup_sechdrs(struct
purgatory_info *pi,
}
offset = ALIGN(offset, align);
+
+ /*
+* Check if the segment contains the entry point, if so,
+* calculate the value of image->start based on it.
+* If the compiler has produced more than one .text section
+* (Eg: .text.hot), they are generally after the main .text
+* section, and they shall not be used to calculate
+* image->start. So do not re-calculate image->start if it
+* is not set to the initial value, and warn the user so they
+* have a chance to fix their purgatory's linker script.
+*/
if (sechdrs[i].sh_flags & SHF_EXECINSTR &&
pi->ehdr->e_entry >= sechdrs[i].sh_addr &&
pi->ehdr->e_entry < (sechdrs[i].sh_addr
-+ sechdrs[i].sh_size)) {
++ sechdrs[i].sh_size) &&
+ !WARN_ON(kbuf->image->start != pi->ehdr->e_entry)) {
kbuf->image->start -= sechdrs[i].sh_addr;
kbuf->image->start += kbuf->mem + offset;
}
--
2.40.1.495.gc816e09b53d-goog
