Add a late PGD init callback to places that allocate a new MM
with a new PGD: copy_process() and exec().
The purpose of this callback is to allow architectures to implement
lockless initialization of task PGDs, to remove the scalability
limit of pgd_list/pgd_lock.
Architectures can opt in to this callback via the ARCH_HAS_PGD_INIT_LATE
Kconfig flag. There's zero overhead on architectures that are not using it.
Cc: Andrew Morton <a...@linux-foundation.org>
Cc: Andy Lutomirski <l...@amacapital.net>
Cc: Borislav Petkov <b...@alien8.de>
Cc: Brian Gerst <brge...@gmail.com>
Cc: Denys Vlasenko <dvlas...@redhat.com>
Cc: H. Peter Anvin <h...@zytor.com>
Cc: Linus Torvalds <torva...@linux-foundation.org>
Cc: Peter Zijlstra <pet...@infradead.org>
Cc: Rik van Riel <r...@redhat.com>
Cc: Thomas Gleixner <t...@linutronix.de>
Cc: Waiman Long <waiman.l...@hp.com>
Cc: linux-a...@vger.kernel.org
Cc: linux...@vger.kernel.org
Signed-off-by: Ingo Molnar <mi...@kernel.org>
---
arch/Kconfig | 9 +++++++++
fs/exec.c | 3 +++
include/linux/mm.h | 6 ++++++
kernel/fork.c | 16 ++++++++++++++++
4 files changed, 34 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
index a65eafb24997..a8e866cd4247 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -491,6 +491,15 @@ config PGTABLE_LEVELS
int
default 2
+config ARCH_HAS_PGD_INIT_LATE
+ bool
+ help
+ Architectures that want a late PGD initialization can define
+ the arch_pgd_init_late() callback and it will be called
+ by the generic new task (fork()) code after a new task has
+ been made visible on the task list, but before it has been
+ first scheduled.
+
config ARCH_HAS_ELF_RANDOMIZE
bool
help
diff --git a/fs/exec.c b/fs/exec.c
index 1977c2a553ac..c1d213c64fda 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -860,7 +860,10 @@ static int exec_mmap(struct mm_struct *mm)
}
task_lock(tsk);
active_mm = tsk->active_mm;
+
tsk->mm = mm;
+ arch_pgd_init_late(mm, mm->pgd);
+
tsk->active_mm = mm;
activate_mm(active_mm, mm);
tsk->mm->vmacache_seqnum = 0;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0755b9fd03a7..35d887d2b038 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1134,6 +1134,12 @@ int follow_phys(struct vm_area_struct *vma, unsigned
long address,
int generic_access_phys(struct vm_area_struct *vma, unsigned long addr,
void *buf, int len, int write);
+#ifdef CONFIG_ARCH_HAS_PGD_INIT_LATE
+void arch_pgd_init_late(struct mm_struct *mm, pgd_t *pgd);
+#else
+static inline void arch_pgd_init_late(struct mm_struct *mm, pgd_t *pgd) { }
+#endif
+
static inline void unmap_shared_mapping_range(struct address_space *mapping,
loff_t const holebegin, loff_t const holelen)
{
diff --git a/kernel/fork.c b/kernel/fork.c
index 03c1eaaa6ef5..1f83ceca6c6c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1592,6 +1592,22 @@ static struct task_struct *copy_process(unsigned long
clone_flags,
syscall_tracepoint_update(p);
write_unlock_irq(&tasklist_lock);
+ /*
+ * If we have a new PGD then initialize it:
+ *
+ * This method is called after a task has been made visible
+ * on the task list already.
+ *
+ * Architectures that manage per task kernel pagetables
+ * might use this callback to initialize them after they
+ * are already visible to new updates.
+ *
+ * NOTE: any user-space parts of the PGD are already initialized
+ * and must not be clobbered.
+ */
+ if (p->mm != current->mm)
+ arch_pgd_init_late(p->mm, p->mm->pgd);
+
proc_fork_connector(p);
cgroup_post_fork(p);
if (clone_flags & CLONE_THREAD)
--
2.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
