Commit:     fa28ba21cec24d3fa1279bcae7e5d5ff6224635a
Parent:     f212ec4b7b4d84290f12c9c0416cdea283bf5f40
Author:     Jeremy Fitzhardinge <[EMAIL PROTECTED]>
AuthorDate: Wed Jan 30 13:34:11 2008 +0100
Committer:  Ingo Molnar <[EMAIL PROTECTED]>
CommitDate: Wed Jan 30 13:34:11 2008 +0100

    x86: defer cr3 reload when doing pud_clear()
    PAE mode requires that we reload cr3 in order to guarantee that
    changes to the pgd will be noticed by the processor.  This means that
    in principle pud_clear needs to reload cr3 every time.  However,
    because reloading cr3 implies a tlb flush, we want to avoid it where
    pud_clear() is only used in a couple of places:
     - in free_pmd_range(), when pulling down a range of process address space, 
     - huge_pmd_unshare()
    In both cases, the calling code will do a a tlb flush anyway, so
    there's no need to do it within pud_clear().
    In free_pmd_range(), the pud_clear is immediately followed by
    pmd_free_tlb(); we can hook that to make the mmu_gather do an
    unconditional full flush to make sure cr3 gets reloaded.
    In huge_pmd_unshare, it is followed by flush_tlb_range, which always
    results in a full cr3-reload tlb flush.
    Signed-off-by: Jeremy Fitzhardinge <[EMAIL PROTECTED]>
    Cc: Andi Kleen <[EMAIL PROTECTED]>
    Cc: Linus Torvalds <[EMAIL PROTECTED]>
    Cc: H. Peter Anvin <[EMAIL PROTECTED]>
    Cc: William Irwin <[EMAIL PROTECTED]>
    Signed-off-by: Ingo Molnar <[EMAIL PROTECTED]>
    Signed-off-by: Thomas Gleixner <[EMAIL PROTECTED]>
 include/asm-x86/pgalloc_32.h     |    7 +++++++
 include/asm-x86/pgtable-3level.h |   21 +++++++++++++++------
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/include/asm-x86/pgalloc_32.h b/include/asm-x86/pgalloc_32.h
index 0caa37a..10c2b45 100644
--- a/include/asm-x86/pgalloc_32.h
+++ b/include/asm-x86/pgalloc_32.h
@@ -74,6 +74,13 @@ static inline void pmd_free(pmd_t *pmd)
 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
+       /* This is called just after the pmd has been detached from
+          the pgd, which requires a full tlb flush to be recognized
+          by the CPU.  Rather than incurring multiple tlb flushes
+          while the address space is being pulled down, make the tlb
+          gathering machinery do a full flush when we're done. */
+       tlb->fullmm = 1;
        paravirt_release_pd(__pa(pmd) >> PAGE_SHIFT);
        tlb_remove_page(tlb, virt_to_page(pmd));
diff --git a/include/asm-x86/pgtable-3level.h b/include/asm-x86/pgtable-3level.h
index ed4c6f0..a195c3e 100644
--- a/include/asm-x86/pgtable-3level.h
+++ b/include/asm-x86/pgtable-3level.h
@@ -96,14 +96,23 @@ static inline void pud_clear(pud_t *pudp)
        set_pud(pudp, __pud(0));
-        * Pentium-II erratum A13: in PAE mode we explicitly have to flush
-        * the TLB via cr3 if the top-level pgd is changed...
+        * In principle we need to do a cr3 reload here to make sure
+        * the processor recognizes the changed pgd.  In practice, all
+        * the places where pud_clear() gets called are followed by
+        * full tlb flushes anyway, so we can defer the cost here.
-        * XXX I don't think we need to worry about this here, since
-        * when clearing the pud, the calling code needs to flush the
-        * tlb anyway.  But do it now for safety's sake. - jsgf
+        * Specifically:
+        *
+        * mm/memory.c:free_pmd_range() - immediately after the
+        * pud_clear() it does a pmd_free_tlb().  We change the
+        * mmu_gather structure to do a full tlb flush (which has the
+        * effect of reloading cr3) when the pagetable free is
+        * complete.
+        *
+        * arch/x86/mm/hugetlbpage.c:huge_pmd_unshare() - the call to
+        * this is followed by a flush_tlb_range, which on x86 does a
+        * full tlb flush.
-       write_cr3(read_cr3());
 #define pud_page(pud) \
To unsubscribe from this list: send the line "unsubscribe git-commits-head" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at

Reply via email to