[RFC 0/2] Potential race condition with page lock

[Chintan Pandya]

In 4.14 kernel, observed following 2 BUG_ON(!PageLocked(page)) scenarios.
Both looks to be having similar cause.

Case: 1
[127823.176076] try_to_free_buffers+0xfc/0x108 (BUG_ON(), page lock was freed
   somehow)
[127823.176079] jbd2_journal_try_to_free_buffers+0x15c/0x194 (page lock was
  available till this function)
[127823.176083] ext4_releasepage+0xe0/0x110 
[127823.176087] try_to_release_page+0x68/0x90 (page lock was available till
  this function)
[127823.176090] invalidate_inode_page+0x94/0xa8
[127823.176093] invalidate_mapping_pages_without_uidlru+0xec/0x1a4 (page lock
  taken here)
...
...

Case: 2
[] el1_dbg+0x18
[] __remove_mapping+0x160  (BUG_ON(), page lock is not
 available. Some one might have
 free'd that.)
[] remove_mapping+0x28
[] invalidate_inode_page+0xa4
[] invalidate_mapping_pages+0xd4  (acquired the page lock)
[] inode_lru_isolate+0x128
[] __list_lru_walk_one+0x10c
[] list_lru_walk_one+0x58
[] prune_icache_sb+0x50
[] super_cache_scan+0xfc
[] shrink_slab+0x304
[] shrink_node+0x254
[] do_try_to_free_pages+0x144
[] try_to_free_pages+0x390
[] __alloc_pages_nodemask+0x940
[] __get_free_pages+0x28
[] proc_pid_readlink+0x6c
[] vfs_readlink+0x124
[] SyS_readlinkat+0xc8
[] __sys_trace_return+0x0

Both the scenarios say that current stack tried taking page lock but got
released in meantime by someone else. There could be 2 possiblities here.

1) Someone trying to update page flags and due to race condition, PG_locked
   bit got cleared, unwantedly.
2) Someone else took the lock without checking if it is really locked or not
   as there are explicit APIs to set PG_locked.

I didn't get traces of history for having PG_locked being set non-atomically.
I believe it could be because of performance reasons. Not sure though.

Chintan Pandya (2):
  page-flags: Make page lock operation atomic
  page-flags: Catch the double setter of page flags

 fs/cifs/file.c | 8 
 fs/pipe.c  | 2 +-
 include/linux/page-flags.h | 4 ++--
 include/linux/pagemap.h| 6 +++---
 mm/filemap.c   | 4 ++--
 mm/khugepaged.c| 2 +-
 mm/ksm.c   | 2 +-
 mm/memory-failure.c| 2 +-
 mm/memory.c| 2 +-
 mm/migrate.c   | 2 +-
 mm/shmem.c | 6 +++---
 mm/swap_state.c| 4 ++--
 mm/vmscan.c| 2 +-
 13 files changed, 23 insertions(+), 23 deletions(-)

-- 
2.17.1

[RFC 1/2] page-flags: Make page lock operation atomic

[Chintan Pandya]

Currently, page lock operation is non-atomic. This is opening
some scope for race condition. For ex, if 2 threads are accessing
same page flags, it may happen that our desired thread's page
lock bit (PG_locked) might get overwritten by other thread
leaving page unlocked. This can cause issues later when some
code expects page to be locked but it is not.

Make page lock/unlock operation use the atomic version of
set_bit API. There are other flag set operations which still
uses non-atomic version of set_bit API. Bit, that might be
the change for the future.

Change-Id: I13bdbedc2b198af014d885e1925c93b83ed6660e
Signed-off-by: Chintan Pandya 
---
 fs/cifs/file.c | 8 
 fs/pipe.c  | 2 +-
 include/linux/page-flags.h | 2 +-
 include/linux/pagemap.h| 6 +++---
 mm/filemap.c   | 4 ++--
 mm/khugepaged.c| 2 +-
 mm/ksm.c   | 2 +-
 mm/memory-failure.c| 2 +-
 mm/memory.c| 2 +-
 mm/migrate.c   | 2 +-
 mm/shmem.c | 6 +++---
 mm/swap_state.c| 4 ++--
 mm/vmscan.c| 2 +-
 13 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 7d6539a04fac..23bcdee37239 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -3661,13 +3661,13 @@ readpages_get_pages(struct address_space *mapping, 
struct list_head *page_list,
 * should have access to this page, we're safe to simply set
 * PG_locked without checking it first.
 */
-   __SetPageLocked(page);
+   SetPageLocked(page);
rc = add_to_page_cache_locked(page, mapping,
  page->index, gfp);
 
/* give up if we can't stick it in the cache */
if (rc) {
-   __ClearPageLocked(page);
+   ClearPageLocked(page);
return rc;
}
 
@@ -3688,9 +3688,9 @@ readpages_get_pages(struct address_space *mapping, struct 
list_head *page_list,
if (*bytes + PAGE_SIZE > rsize)
break;
 
-   __SetPageLocked(page);
+   SetPageLocked(page);
if (add_to_page_cache_locked(page, mapping, page->index, gfp)) {
-   __ClearPageLocked(page);
+   ClearPageLocked(page);
break;
}
list_move_tail(>lru, tmplist);
diff --git a/fs/pipe.c b/fs/pipe.c
index 8ef7d7bef775..1bab40a2ca44 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -147,7 +147,7 @@ static int anon_pipe_buf_steal(struct pipe_inode_info *pipe,
if (page_count(page) == 1) {
if (memcg_kmem_enabled())
memcg_kmem_uncharge(page, 0);
-   __SetPageLocked(page);
+   SetPageLocked(page);
return 0;
}
return 1;
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 5af67406b9c9..a56a9bd4bc6b 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -268,7 +268,7 @@ static inline int TestClearPage##uname(struct page *page) { 
return 0; }
 #define TESTSCFLAG_FALSE(uname)
\
TESTSETFLAG_FALSE(uname) TESTCLEARFLAG_FALSE(uname)
 
-__PAGEFLAG(Locked, locked, PF_NO_TAIL)
+PAGEFLAG(Locked, locked, PF_NO_TAIL)
 PAGEFLAG(Waiters, waiters, PF_ONLY_HEAD) __CLEARPAGEFLAG(Waiters, waiters, 
PF_ONLY_HEAD)
 PAGEFLAG(Error, error, PF_NO_COMPOUND) TESTCLEARFLAG(Error, error, 
PF_NO_COMPOUND)
 PAGEFLAG(Referenced, referenced, PF_HEAD)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 51a9a0af3281..87a0447cfbe0 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -619,17 +619,17 @@ int replace_page_cache_page(struct page *old, struct page 
*new, gfp_t gfp_mask);
 
 /*
  * Like add_to_page_cache_locked, but used to add newly allocated pages:
- * the page is new, so we can just run __SetPageLocked() against it.
+ * the page is new, so we can just run SetPageLocked() against it.
  */
 static inline int add_to_page_cache(struct page *page,
struct address_space *mapping, pgoff_t offset, gfp_t gfp_mask)
 {
int error;
 
-   __SetPageLocked(page);
+   SetPageLocked(page);
error = add_to_page_cache_locked(page, mapping, offset, gfp_mask);
if (unlikely(error))
-   __ClearPageLocked(page);
+   ClearPageLocked(page);
return error;
 }
 
diff --git a/mm/filemap.c b/mm/filemap.c
index 8e09304af1ec..14284726cf3a 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -807,11 +807,11 @@ int add_to_page_cache_lru(struct page *page, struct 
address_space *mapping,
void *shadow = NULL;
int ret;
 
-   __SetPageLocked(page);
+   SetPageLocked(page);
ret = __add_to_page_cache_locked(page, mapping, offset,
 gfp_mask, );
if (

[RFC 2/2] page-flags: Catch the double setter of page flags

[Chintan Pandya]

Some of the page flags, like PG_locked is not supposed to
be set twice. Currently, there is no protection around this
and many callers directly tries to set this bit. Others
follow trylock_page() which is much safer version of the
same. But, for performance issues, we may not want to
implement wait-until-set. So, at least, find out who is
doing double setting and fix them.

Change-Id: I1295fcb8527ce4b54d5d11c11287fc7516006cf0
Signed-off-by: Chintan Pandya 
---
 include/linux/page-flags.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index a56a9bd4bc6b..e307775c2b4a 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -208,7 +208,7 @@ static __always_inline int Page##uname(struct page *page)   
\
 
 #define SETPAGEFLAG(uname, lname, policy)  \
 static __always_inline void SetPage##uname(struct page *page)  \
-   { set_bit(PG_##lname, (page, 1)->flags); }
+   { WARN_ON(test_and_set_bit(PG_##lname, (page, 1)->flags)); }
 
 #define CLEARPAGEFLAG(uname, lname, policy)\
 static __always_inline void ClearPage##uname(struct page *page)
\
-- 
2.17.1

[tip:x86/mm] ioremap: Update pgtable free interfaces with addr

[tip-bot for Chintan Pandya]

Commit-ID:  785a19f9d1dd8a4ab2d0633be4656653bd3de1fc
Gitweb: https://git.kernel.org/tip/785a19f9d1dd8a4ab2d0633be4656653bd3de1fc
Author: Chintan Pandya 
AuthorDate: Wed, 27 Jun 2018 08:13:47 -0600
Committer:  Thomas Gleixner 
CommitDate: Wed, 4 Jul 2018 21:37:08 +0200

ioremap: Update pgtable free interfaces with addr

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[toshi.k...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Signed-off-by: Thomas Gleixner 
Cc: mho...@suse.com
Cc: a...@linux-foundation.org
Cc: h...@zytor.com
Cc: linux...@kvack.org
Cc: linux-arm-ker...@lists.infradead.org
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: sta...@vger.kernel.org
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: "H. Peter Anvin" 
Cc: 
Link: https://lkml.kernel.org/r/20180627141348.21777-3-toshi.k...@hpe.com

---
 arch/arm64/mm/mmu.c   |  4 ++--
 arch/x86/mm/pgtable.c | 12 +++-
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 4 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75670ff..8ae5d7ae4af3 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 1aeb7a5dbce5..fbd14e506758 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -723,11 +723,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -738,7 +739,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -750,11 +751,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
@@ -770,7 +772,7 @@ int pmd_free_pte_page(pmd_t *pmd)
 
 #else /* !CONFIG_X86_64 */
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
@@ -779,7 +781,7 @@ int pud_free_pmd_page(pud_t *pud)
  * Disable free page handling on x86-PAE. This assures that ioremap()
  * does not update sync'd pmd entries. See vmalloc_sync_one().
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639afaa39..b081794ba135 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t 
prot);
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)

[tip:x86/mm] ioremap: Update pgtable free interfaces with addr

[tip-bot for Chintan Pandya]

Commit-ID:  785a19f9d1dd8a4ab2d0633be4656653bd3de1fc
Gitweb: https://git.kernel.org/tip/785a19f9d1dd8a4ab2d0633be4656653bd3de1fc
Author: Chintan Pandya 
AuthorDate: Wed, 27 Jun 2018 08:13:47 -0600
Committer:  Thomas Gleixner 
CommitDate: Wed, 4 Jul 2018 21:37:08 +0200

ioremap: Update pgtable free interfaces with addr

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[toshi.k...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Signed-off-by: Thomas Gleixner 
Cc: mho...@suse.com
Cc: a...@linux-foundation.org
Cc: h...@zytor.com
Cc: linux...@kvack.org
Cc: linux-arm-ker...@lists.infradead.org
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: sta...@vger.kernel.org
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: "H. Peter Anvin" 
Cc: 
Link: https://lkml.kernel.org/r/20180627141348.21777-3-toshi.k...@hpe.com

---
 arch/arm64/mm/mmu.c   |  4 ++--
 arch/x86/mm/pgtable.c | 12 +++-
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 4 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75670ff..8ae5d7ae4af3 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 1aeb7a5dbce5..fbd14e506758 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -723,11 +723,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -738,7 +739,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -750,11 +751,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
@@ -770,7 +772,7 @@ int pmd_free_pte_page(pmd_t *pmd)
 
 #else /* !CONFIG_X86_64 */
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
@@ -779,7 +781,7 @@ int pud_free_pmd_page(pud_t *pud)
  * Disable free page handling on x86-PAE. This assures that ioremap()
  * does not update sync'd pmd entries. See vmalloc_sync_one().
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639afaa39..b081794ba135 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t 
prot);
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)

Re: [PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Hi Andrew,

On 6/6/2018 9:15 PM, Will Deacon wrote:

[...]


On Wed, Jun 06, 2018 at 12:31:18PM +0530, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.

Patch 1/3 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.



[...]




  arch/arm64/include/asm/tlbflush.h |  7 ++
  arch/arm64/mm/mmu.c   | 48 +++
  arch/x86/mm/pgtable.c |  8 ---
  include/asm-generic/pgtable.h |  8 +++
  lib/ioremap.c |  4 ++--


If you get an ack from the x86 folks, then I could take all of this via
arm64. Alternatively, now that I've reviewed the series this could happily
go via another tree (e.g. akpm).



Would you be able to take a look at this series ?
- 1/3 has been reviewed by Will and Toshi (as it touches arm64 and x86).
- 2/3 & 3/3 are arm64 specific changes.

If you think to take these patches, great ! Otherwise, I will try to
reach-out to x86 folks for their ack.


Thanks,

Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Hi Andrew,

On 6/6/2018 9:15 PM, Will Deacon wrote:

[...]


On Wed, Jun 06, 2018 at 12:31:18PM +0530, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.

Patch 1/3 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.



[...]




  arch/arm64/include/asm/tlbflush.h |  7 ++
  arch/arm64/mm/mmu.c   | 48 +++
  arch/x86/mm/pgtable.c |  8 ---
  include/asm-generic/pgtable.h |  8 +++
  lib/ioremap.c |  4 ++--


If you get an ack from the x86 folks, then I could take all of this via
arm64. Alternatively, now that I've reviewed the series this could happily
go via another tree (e.g. akpm).



Would you be able to take a look at this series ?
- 1/3 has been reviewed by Will and Toshi (as it touches arm64 and x86).
- 2/3 & 3/3 are arm64 specific changes.

If you think to take these patches, great ! Otherwise, I will try to
reach-out to x86 folks for their ack.


Thanks,

Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

On 6/6/2018 9:15 PM, Will Deacon wrote:

Hi Chintan,

Hi Will,



Thanks for sticking with this. I've reviewed the series now and I'm keen
for it to land in mainline. Just a couple of things below.



Thanks for all the reviews so far.


On Wed, Jun 06, 2018 at 12:31:18PM +0530, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.


...


These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.


Not sure we need to target -stable, since we solved the crash by disabling
the use of huge io mappings.


If disabling of huge io mappings have gone into stable trees, then I
won't push for these changes there.




  arch/arm64/include/asm/tlbflush.h |  7 ++
  arch/arm64/mm/mmu.c   | 48 +++
  arch/x86/mm/pgtable.c |  8 ---
  include/asm-generic/pgtable.h |  8 +++
  lib/ioremap.c |  4 ++--


If you get an ack from the x86 folks, then I could take all of this via
arm64. Alternatively, now that I've reviewed the series this could happily
go via another tree (e.g. akpm).


Sure. I would wait for few days before either of them take notice of
this. If required, I will communicated with them.



Thanks,

Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

On 6/6/2018 9:15 PM, Will Deacon wrote:

Hi Chintan,

Hi Will,



Thanks for sticking with this. I've reviewed the series now and I'm keen
for it to land in mainline. Just a couple of things below.



Thanks for all the reviews so far.


On Wed, Jun 06, 2018 at 12:31:18PM +0530, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.


...


These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.


Not sure we need to target -stable, since we solved the crash by disabling
the use of huge io mappings.


If disabling of huge io mappings have gone into stable trees, then I
won't push for these changes there.




  arch/arm64/include/asm/tlbflush.h |  7 ++
  arch/arm64/mm/mmu.c   | 48 +++
  arch/x86/mm/pgtable.c |  8 ---
  include/asm-generic/pgtable.h |  8 +++
  lib/ioremap.c |  4 ++--


If you get an ack from the x86 folks, then I could take all of this via
arm64. Alternatively, now that I've reviewed the series this could happily
go via another tree (e.g. akpm).


Sure. I would wait for few days before either of them take notice of
this. If required, I will communicated with them.



Thanks,

Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 2/3] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 1/3] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75..8ae5d7a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v13 3/3] arm64: Implement page table free interfaces

[Chintan Pandya]

arm64 requires break-before-make. Originally, before
setting up new pmd/pud entry for huge mapping, in few
cases, the modifying pmd/pud entry was still valid
and pointing to next level page table as we only
clear off leaf PTE in unmap leg.

 a) This was resulting into stale entry in TLBs (as few
TLBs also cache intermediate mapping for performance
reasons)
 b) Also, modifying pmd/pud was the only reference to
next level page table and it was getting lost without
freeing it. So, page leaks were happening.

Implement pud_free_pmd_page() and pmd_free_pte_page() to
enforce BBM and also free the leaking page tables.

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidation of TLB
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 48 
 1 file changed, 44 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..65f8627 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -977,12 +978,51 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+
+   /* No-op for empty entry and WARN_ON for valid entry */
+   if (!pmd_present(pmd) || !pmd_table(pmd)) {
+   VM_WARN_ON(!pmd_table(pmd));
+   return 1;
+   }
+
+   table = pte_offset_kernel(pmdp, addr);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *pmdp;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+
+   /* No-op for empty entry and WARN_ON for valid entry */
+   if (!pud_present(pud) || !pud_table(pud)) {
+   VM_WARN_ON(!pud_table(pud));
+   return 1;
+   }
+
+   table = pmd_offset(pudp, addr);
+   pmdp = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(pmdp, next);
+   } while (pmdp++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 2/3] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 1/3] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75..8ae5d7a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v13 3/3] arm64: Implement page table free interfaces

[Chintan Pandya]

arm64 requires break-before-make. Originally, before
setting up new pmd/pud entry for huge mapping, in few
cases, the modifying pmd/pud entry was still valid
and pointing to next level page table as we only
clear off leaf PTE in unmap leg.

 a) This was resulting into stale entry in TLBs (as few
TLBs also cache intermediate mapping for performance
reasons)
 b) Also, modifying pmd/pud was the only reference to
next level page table and it was getting lost without
freeing it. So, page leaks were happening.

Implement pud_free_pmd_page() and pmd_free_pte_page() to
enforce BBM and also free the leaking page tables.

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidation of TLB
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 48 
 1 file changed, 44 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..65f8627 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -977,12 +978,51 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+
+   /* No-op for empty entry and WARN_ON for valid entry */
+   if (!pmd_present(pmd) || !pmd_table(pmd)) {
+   VM_WARN_ON(!pmd_table(pmd));
+   return 1;
+   }
+
+   table = pte_offset_kernel(pmdp, addr);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *pmdp;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+
+   /* No-op for empty entry and WARN_ON for valid entry */
+   if (!pud_present(pud) || !pud_table(pud)) {
+   VM_WARN_ON(!pud_table(pud));
+   return 1;
+   }
+
+   table = pmd_offset(pudp, addr);
+   pmdp = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(pmdp, next);
+   } while (pmdp++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/3 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V12->V13:
 - Dropped v12 3/5 and v12 5/5
 - Using existing APIs like pte_offset_kernel and pmd_offset
   instead of pmd_page_vaddr & pud_page_vaddr
 - Updated commit log message for 3/3
 - Some other cosmetic corrections in 3/3

>From V11->V12:
 - Introduced p*d_page_vaddr helper macros and using them
 - Rebased over current tip

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64


Chintan Pandya (3):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces

 arch/arm64/include/asm/tlbflush.h |  7 ++
 arch/arm64/mm/mmu.c   | 48 +++
 arch/x86/mm/pgtable.c |  8 ---
 include/asm-generic/pgtable.h |  8 +++
 lib/ioremap.c |  4 ++--
 5 files changed, 62 insertions(+), 13 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v13 0/3] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/3 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V12->V13:
 - Dropped v12 3/5 and v12 5/5
 - Using existing APIs like pte_offset_kernel and pmd_offset
   instead of pmd_page_vaddr & pud_page_vaddr
 - Updated commit log message for 3/3
 - Some other cosmetic corrections in 3/3

>From V11->V12:
 - Introduced p*d_page_vaddr helper macros and using them
 - Rebased over current tip

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64


Chintan Pandya (3):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces

 arch/arm64/include/asm/tlbflush.h |  7 ++
 arch/arm64/mm/mmu.c   | 48 +++
 arch/x86/mm/pgtable.c |  8 ---
 include/asm-generic/pgtable.h |  8 +++
 lib/ioremap.c |  4 ++--
 5 files changed, 62 insertions(+), 13 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 3/5] arm64: pgtable: Add p*d_page_vaddr helper macros

[Chintan Pandya]

On 6/4/2018 5:43 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:16PM +0530, Chintan Pandya wrote:

Add helper macros to give virtual references to page
tables. These will be used while freeing dangling
page tables.

Signed-off-by: Chintan Pandya 
---
  arch/arm64/include/asm/pgtable.h | 3 +++
  1 file changed, 3 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7c4c8f3..ef4047f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -580,6 +580,9 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
  
  #endif  /* CONFIG_PGTABLE_LEVELS > 3 */
  
+#define pmd_page_vaddr(pmd) __va(pmd_page_paddr(pmd))

+#define pud_page_vaddr(pud) __va(pud_page_paddr(pud))


Are these actually needed, or do pte_offset_kernel and pmd_offset do the
job already?



I introduced these macros for consistency across different arch.

Looking at pte_offset_kernel, it seems to use READ_ONCE() which looks
little costly for its intended use (in next patch) where we already have
dereferenced value. Do you still suggest to remove this ?


Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 3/5] arm64: pgtable: Add p*d_page_vaddr helper macros

[Chintan Pandya]

On 6/4/2018 5:43 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:16PM +0530, Chintan Pandya wrote:

Add helper macros to give virtual references to page
tables. These will be used while freeing dangling
page tables.

Signed-off-by: Chintan Pandya 
---
  arch/arm64/include/asm/pgtable.h | 3 +++
  1 file changed, 3 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7c4c8f3..ef4047f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -580,6 +580,9 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
  
  #endif  /* CONFIG_PGTABLE_LEVELS > 3 */
  
+#define pmd_page_vaddr(pmd) __va(pmd_page_paddr(pmd))

+#define pud_page_vaddr(pud) __va(pud_page_paddr(pud))


Are these actually needed, or do pte_offset_kernel and pmd_offset do the
job already?



I introduced these macros for consistency across different arch.

Looking at pte_offset_kernel, it seems to use READ_ONCE() which looks
little costly for its intended use (in next patch) where we already have
dereferenced value. Do you still suggest to remove this ?


Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 5/5] arm64: Allow huge io mappings again

[Chintan Pandya]

On 6/4/2018 5:44 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:18PM +0530, Chintan Pandya wrote:

Huge mappings have had stability issues due to stale
TLB entry and memory leak issues. Since, those are
addressed in this series of patches, it is now safe
to allow huge mappings.

Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 18 ++
  1 file changed, 2 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 6e7e16c..c65abc4 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -934,15 +934,8 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
  {
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pud_t new_pud = pfn_pud(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
-  pud_val(new_pud)))
-   return 0;


Do you actually need to remove these checks? If we're doing
break-before-make properly, then the check won't fire but it would be
good to keep it there so we can catch misuse of these in future.

In other words, can we drop this patch?


Yes, we don't need this patch as BBM is happening before this. I missed
that. I'll remove this patch in v13.



Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 5/5] arm64: Allow huge io mappings again

[Chintan Pandya]

On 6/4/2018 5:44 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:18PM +0530, Chintan Pandya wrote:

Huge mappings have had stability issues due to stale
TLB entry and memory leak issues. Since, those are
addressed in this series of patches, it is now safe
to allow huge mappings.

Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 18 ++
  1 file changed, 2 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 6e7e16c..c65abc4 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -934,15 +934,8 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
  {
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pud_t new_pud = pfn_pud(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
-  pud_val(new_pud)))
-   return 0;


Do you actually need to remove these checks? If we're doing
break-before-make properly, then the check won't fire but it would be
good to keep it there so we can catch misuse of these in future.

In other words, can we drop this patch?


Yes, we don't need this patch as BBM is happening before this. I missed
that. I'll remove this patch in v13.



Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 4/5] arm64: Implement page table free interfaces

[Chintan Pandya]

On 6/4/2018 5:43 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:17PM +0530, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables


Please can you rewrite this describing the problem that you're solving,
rather than a brief summary of some requirements?


Okay. I'll fix this in v13.




Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 38 ++
  1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..6e7e16c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -977,12 +978,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = pmd_page_vaddr(pmd);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {


Just some stylistic stuff, but please can you rewrite this as:

if (!pud_present(pud) || VM_WARN_ON(!pud_table(pud)))
return 1;

similarly for the pmd/pte code above.


Okay. v13 will have this.




+   table = pud_page_vaddr(pud);
+   entry = table;


Could you rename entry -> pmdp, please?


Sure.




+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;


So with these patches, we only ever return 1 from these helpers. It looks
like the same is true for x86, so how about we make them void and move the
calls inside the conditionals in lib/ioremap.c? Obviously, this would be a
separate patch on the end.


That sounds valid code churn to me. But since x86 discussion is not
concluded yet, I would wait to share until that gets resolved. May be
not in v13 but separate effort. Would that be okay to you ?



Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 4/5] arm64: Implement page table free interfaces

[Chintan Pandya]

On 6/4/2018 5:43 PM, Will Deacon wrote:

On Fri, Jun 01, 2018 at 06:09:17PM +0530, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables


Please can you rewrite this describing the problem that you're solving,
rather than a brief summary of some requirements?


Okay. I'll fix this in v13.




Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 38 ++
  1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..6e7e16c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -977,12 +978,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = pmd_page_vaddr(pmd);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {


Just some stylistic stuff, but please can you rewrite this as:

if (!pud_present(pud) || VM_WARN_ON(!pud_table(pud)))
return 1;

similarly for the pmd/pte code above.


Okay. v13 will have this.




+   table = pud_page_vaddr(pud);
+   entry = table;


Could you rename entry -> pmdp, please?


Sure.




+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;


So with these patches, we only ever return 1 from these helpers. It looks
like the same is true for x86, so how about we make them void and move the
calls inside the conditionals in lib/ioremap.c? Obviously, this would be a
separate patch on the end.


That sounds valid code churn to me. But since x86 discussion is not
concluded yet, I would wait to share until that gets resolved. May be
not in v13 but separate effort. Would that be okay to you ?



Will



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 0/5] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Hi Will,

Just curious to know, is there anything that I should be addressing
in these patches ? For now, I don't see anything from my side that
requires modification, unless one has some more review comments on
this.

Status so far on and around this:
 - Status of Toshi's series of patches is still not clear to me.
   However, if this series can get through first, there won't
   be conflicting scenarios as far as arm64 is concerned.
 - I've rebased these patches on tip
 - Also re-tested these patches for long duration tests with
   1 GB mapping case also exercised enough. Test ended positively.

Thanks,

On 6/1/2018 6:09 PM, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

 From V11->V12:
  - Introduced p*d_page_vaddr helper macros and using them
  - Rebased over current tip

 From V10->V11:
  - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
conding style
  - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

 From V9->V10:
  - Updated commit log for patch 1/4 by Toshi
  - Addressed review comments by Will on patch 3/4

 From V8->V9:
  - Used __TLBI_VADDR macros in new TLB flush API

 From V7->V8:
  - Properly fixed compilation issue in x86 file

 From V6->V7:
  - Fixed compilation issue in x86 case
  - V6 patches were not properly enumarated

 From V5->V6:
  - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
"bool tlb_inv" based variance as it is not need now
  - Re-naming for consistency

 From V4->V5:
  - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
for kernel addresses

 From V3->V4:
  - Add header for 'addr' in x86 implementation
  - Re-order pmd/pud clear and table free
  - Avoid redundant TLB invalidatation in one perticular case

 From V2->V3:
  - Use the exisiting page table free interface to do arm64
specific things

 From V1->V2:
  - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
  - Honored BBM for ARM64

Chintan Pandya (5):
   ioremap: Update pgtable free interfaces with addr
   arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
   arm64: pgtable: Add p*d_page_vaddr helper macros
   arm64: Implement page table free interfaces
   arm64: Re-enable huge io mappings

  arch/arm64/include/asm/pgtable.h  |  3 +++
  arch/arm64/include/asm/tlbflush.h |  7 +
  arch/arm64/mm/mmu.c   | 56 +--
  arch/x86/mm/pgtable.c |  8 +++---
  include/asm-generic/pgtable.h |  8 +++---
  lib/ioremap.c |  4 +--
  6 files changed, 57 insertions(+), 29 deletions(-)



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v12 0/5] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Hi Will,

Just curious to know, is there anything that I should be addressing
in these patches ? For now, I don't see anything from my side that
requires modification, unless one has some more review comments on
this.

Status so far on and around this:
 - Status of Toshi's series of patches is still not clear to me.
   However, if this series can get through first, there won't
   be conflicting scenarios as far as arm64 is concerned.
 - I've rebased these patches on tip
 - Also re-tested these patches for long duration tests with
   1 GB mapping case also exercised enough. Test ended positively.

Thanks,

On 6/1/2018 6:09 PM, Chintan Pandya wrote:

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

 From V11->V12:
  - Introduced p*d_page_vaddr helper macros and using them
  - Rebased over current tip

 From V10->V11:
  - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
conding style
  - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

 From V9->V10:
  - Updated commit log for patch 1/4 by Toshi
  - Addressed review comments by Will on patch 3/4

 From V8->V9:
  - Used __TLBI_VADDR macros in new TLB flush API

 From V7->V8:
  - Properly fixed compilation issue in x86 file

 From V6->V7:
  - Fixed compilation issue in x86 case
  - V6 patches were not properly enumarated

 From V5->V6:
  - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
"bool tlb_inv" based variance as it is not need now
  - Re-naming for consistency

 From V4->V5:
  - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
for kernel addresses

 From V3->V4:
  - Add header for 'addr' in x86 implementation
  - Re-order pmd/pud clear and table free
  - Avoid redundant TLB invalidatation in one perticular case

 From V2->V3:
  - Use the exisiting page table free interface to do arm64
specific things

 From V1->V2:
  - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
  - Honored BBM for ARM64

Chintan Pandya (5):
   ioremap: Update pgtable free interfaces with addr
   arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
   arm64: pgtable: Add p*d_page_vaddr helper macros
   arm64: Implement page table free interfaces
   arm64: Re-enable huge io mappings

  arch/arm64/include/asm/pgtable.h  |  3 +++
  arch/arm64/include/asm/tlbflush.h |  7 +
  arch/arm64/mm/mmu.c   | 56 +--
  arch/x86/mm/pgtable.c |  8 +++---
  include/asm-generic/pgtable.h |  8 +++---
  lib/ioremap.c |  4 +--
  6 files changed, 57 insertions(+), 29 deletions(-)



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 3/5] arm64: pgtable: Add p*d_page_vaddr helper macros

[Chintan Pandya]

Add helper macros to give virtual references to page
tables. These will be used while freeing dangling
page tables.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/pgtable.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7c4c8f3..ef4047f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -580,6 +580,9 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 
 #endif  /* CONFIG_PGTABLE_LEVELS > 3 */
 
+#define pmd_page_vaddr(pmd) __va(pmd_page_paddr(pmd))
+#define pud_page_vaddr(pud) __va(pud_page_paddr(pud))
+
 #define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
 
 /* to find an entry in a page-table-directory */
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 3/5] arm64: pgtable: Add p*d_page_vaddr helper macros

[Chintan Pandya]

Add helper macros to give virtual references to page
tables. These will be used while freeing dangling
page tables.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/pgtable.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7c4c8f3..ef4047f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -580,6 +580,9 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 
 #endif  /* CONFIG_PGTABLE_LEVELS > 3 */
 
+#define pmd_page_vaddr(pmd) __va(pmd_page_paddr(pmd))
+#define pud_page_vaddr(pud) __va(pud_page_paddr(pud))
+
 #define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
 
 /* to find an entry in a page-table-directory */
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 1/5] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75..8ae5d7a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v12 2/5] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 1/5] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 493ff75..8ae5d7a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -977,12 +977,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v12 2/5] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 4/5] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 38 ++
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..6e7e16c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -977,12 +978,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = pmd_page_vaddr(pmd);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = pud_page_vaddr(pud);
+   entry = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 0/5] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V11->V12:
 - Introduced p*d_page_vaddr helper macros and using them
 - Rebased over current tip

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (5):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: pgtable: Add p*d_page_vaddr helper macros
  arm64: Implement page table free interfaces
  arm64: Re-enable huge io mappings

 arch/arm64/include/asm/pgtable.h  |  3 +++
 arch/arm64/include/asm/tlbflush.h |  7 +
 arch/arm64/mm/mmu.c   | 56 +--
 arch/x86/mm/pgtable.c |  8 +++---
 include/asm-generic/pgtable.h |  8 +++---
 lib/ioremap.c |  4 +--
 6 files changed, 57 insertions(+), 29 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 5/5] arm64: Allow huge io mappings again

[Chintan Pandya]

Huge mappings have had stability issues due to stale
TLB entry and memory leak issues. Since, those are
addressed in this series of patches, it is now safe
to allow huge mappings.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 18 ++
 1 file changed, 2 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 6e7e16c..c65abc4 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -934,15 +934,8 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
 {
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pud_t new_pud = pfn_pud(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
-  pud_val(new_pud)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
-   set_pud(pudp, new_pud);
+   set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
 }
 
@@ -950,15 +943,8 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
 {
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
-  pmd_val(new_pmd)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
-   set_pmd(pmdp, new_pmd);
+   set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
 }
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 4/5] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 38 ++
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8ae5d7a..6e7e16c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -977,12 +978,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = pmd_page_vaddr(pmd);
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = pud_page_vaddr(pud);
+   entry = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 0/5] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 1G etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V11->V12:
 - Introduced p*d_page_vaddr helper macros and using them
 - Rebased over current tip

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (5):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: pgtable: Add p*d_page_vaddr helper macros
  arm64: Implement page table free interfaces
  arm64: Re-enable huge io mappings

 arch/arm64/include/asm/pgtable.h  |  3 +++
 arch/arm64/include/asm/tlbflush.h |  7 +
 arch/arm64/mm/mmu.c   | 56 +--
 arch/x86/mm/pgtable.c |  8 +++---
 include/asm-generic/pgtable.h |  8 +++---
 lib/ioremap.c |  4 +--
 6 files changed, 57 insertions(+), 29 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v12 5/5] arm64: Allow huge io mappings again

[Chintan Pandya]

Huge mappings have had stability issues due to stale
TLB entry and memory leak issues. Since, those are
addressed in this series of patches, it is now safe
to allow huge mappings.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 18 ++
 1 file changed, 2 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 6e7e16c..c65abc4 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -934,15 +934,8 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
 {
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pud_t new_pud = pfn_pud(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
-  pud_val(new_pud)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
-   set_pud(pudp, new_pud);
+   set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
 }
 
@@ -950,15 +943,8 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
 {
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
-   pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), sect_prot);
-
-   /* Only allow permission changes for now */
-   if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
-  pmd_val(new_pmd)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
-   set_pmd(pmdp, new_pmd);
+   set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
 }
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 38 ++
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..5f2da5c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_page_paddr(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_page_paddr(pud));
+   entry = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 38 ++
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..5f2da5c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,41 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pte_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_page_paddr(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pmd_t *entry;
+   pud_t pud;
+   unsigned long next, end;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_page_paddr(pud));
+   entry = table;
+   next = addr;
+   end = addr + PUD_SIZE;
+   do {
+   pmd_free_pte_page(entry, next);
+   } while (entry++, next += PMD_SIZE, next != end);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya <cpan...@codeaurora.org>

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
Signed-off-by: Toshi Kani <toshi.k...@hpe.com>
Cc: Andrew Morton <a...@linux-foundation.org>
Cc: Michal Hocko <mho...@suse.com>
Cc: Thomas Gleixner <t...@linutronix.de>
Cc: Ingo Molnar <mi...@redhat.com>
Cc: "H. Peter Anvin" <h...@zytor.com>
Cc: Will Deacon <will.dea...@arm.com>
Cc: Joerg Roedel <j...@8bytes.org>
Cc: <sta...@vger.kernel.org>

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd))

[PATCH v11 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 

---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v11 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 5f2da5c..c71e9df 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 5f2da5c..c71e9df 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

Patch 4/4 may conflict with Laura's patch "[PATCHv2] arm64: Make
sure permission updates happen for pmd/pud". It looks to me that,
problem addressed by this patch will also get resolved by below
series of patches. There could be multiple ways to resolve this.
I would let maintainer suggest the best approach, and I will do
the needful.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  7 ++
 arch/arm64/mm/mmu.c   | 46 +--
 arch/x86/mm/pgtable.c |  8 ---
 include/asm-generic/pgtable.h |  8 +++
 lib/ioremap.c |  4 ++--
 5 files changed, 52 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v11 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

Patch 4/4 may conflict with Laura's patch "[PATCHv2] arm64: Make
sure permission updates happen for pmd/pud". It looks to me that,
problem addressed by this patch will also get resolved by below
series of patches. There could be multiple ways to resolve this.
I would let maintainer suggest the best approach, and I will do
the needful.

These patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V10->V11:
 - Updated pud_free_pmd_page & pmd_free_pte_page to use consistent
   conding style
 - Fixed few bugs by using pmd_page_paddr & pud_page_paddr

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  7 ++
 arch/arm64/mm/mmu.c   | 46 +--
 arch/x86/mm/pgtable.c |  8 ---
 include/asm-generic/pgtable.h |  8 +++
 lib/ioremap.c |  4 ++--
 5 files changed, 52 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v10 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/24/2018 7:27 PM, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
  arch/arm64/mm/mmu.c | 34 ++
  1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..17d9282 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -973,12 +974,37 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pmd_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_val(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, (pte_t *) table);
+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pud_t pud;
+   int i = 0;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_val(pud));
+   do {
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+   } while (i++, i < PTRS_PER_PMD);

Hi Will,

Right after sending these patches, I realized that do-while condition
can be made to look like what we see in ioremap/vmalloc code. I guess,
that's what you suggested. So, I'll raise v11 fixing that. Any more
concerns ?


+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
  }



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v10 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/24/2018 7:27 PM, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 34 ++
  1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..17d9282 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -973,12 +974,37 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pmd_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_val(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, (pte_t *) table);
+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pud_t pud;
+   int i = 0;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_val(pud));
+   do {
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+   } while (i++, i < PTRS_PER_PMD);

Hi Will,

Right after sending these patches, I realized that do-while condition
can be made to look like what we see in ioremap/vmalloc code. I guess,
that's what you suggested. So, I'll raise v11 fixing that. Any more
concerns ?


+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
  }



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 17d9282..defb3bb 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 17d9282..defb3bb 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya <cpan...@codeaurora.org>

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
Signed-off-by: Toshi Kani <toshi.k...@hpe.com>
Cc: Andrew Morton <a...@linux-foundation.org>
Cc: Michal Hocko <mho...@suse.com>
Cc: Thomas Gleixner <t...@linutronix.de>
Cc: Ingo Molnar <mi...@redhat.com>
Cc: "H. Peter Anvin" <h...@zytor.com>
Cc: Will Deacon <will.dea...@arm.com>
Cc: Joerg Roedel <j...@8bytes.org>
Cc: <sta...@vger.kernel.org>
---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd))

[PATCH v10 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

From: Chintan Pandya 

The following kernel panic was observed on ARM64 platform due to a stale
TLB entry.

 1. ioremap with 4K size, a valid pte page table is set.
 2. iounmap it, its pte entry is set to 0.
 3. ioremap the same address with 2M size, update its pmd entry with
a new value.
 4. CPU may hit an exception because the old pmd entry is still in TLB,
which leads to a kernel panic.

Commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page
table") has addressed this panic by falling to pte mappings in the above
case on ARM64.

To support pmd mappings in all cases, TLB purge needs to be performed
in this case on ARM64.

Add a new arg, 'addr', to pud_free_pmd_page() and pmd_free_pte_page()
so that TLB purge can be added later in seprate patches.

[to...@hpe.com: merge changes, rewrite patch description]
Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces")
Signed-off-by: Chintan Pandya 
Signed-off-by: Toshi Kani 
Cc: Andrew Morton 
Cc: Michal Hocko 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: Will Deacon 
Cc: Joerg Roedel 
Cc: 
---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_en

[PATCH v10 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 34 ++
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..17d9282 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,37 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pmd_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_val(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, (pte_t *) table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pud_t pud;
+   int i = 0;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_val(pud));
+   do {
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+   } while (i++, i < PTRS_PER_PMD);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 34 ++
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..17d9282 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,37 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pmd_t *table;
+   pmd_t pmd;
+
+   pmd = READ_ONCE(*pmdp);
+   if (pmd_present(pmd)) {
+   table = __va(pmd_val(pmd));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   pte_free_kernel(NULL, (pte_t *) table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   pud_t pud;
+   int i = 0;
+
+   pud = READ_ONCE(*pudp);
+   if (pud_present(pud)) {
+   table = __va(pud_val(pud));
+   do {
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+   } while (i++, i < PTRS_PER_PMD);
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   pmd_free(NULL, table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

Patch 4/4 may conflict with Laura's patch "[PATCHv2] arm64: Make
sure permission updates happen for pmd/pud". It looks to me that,
problem addressed by this patch will also get resolved by below
series of patches. There could be multiple ways to resolve this.
I would let maintainer suggest the best approach, and I will do
the needful.

This patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64



Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  7 +++
 arch/arm64/mm/mmu.c   | 42 ---
 arch/x86/mm/pgtable.c |  8 +---
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 5 files changed, 48 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v10 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches re-bring huge vmap back for arm64.

Patch 1/4 has been taken by Toshi in his series of patches
by name "[PATCH v3 0/3] fix free pmd/pte page handlings on x86"
to avoid merge conflict with this series.

Patch 4/4 may conflict with Laura's patch "[PATCHv2] arm64: Make
sure permission updates happen for pmd/pud". It looks to me that,
problem addressed by this patch will also get resolved by below
series of patches. There could be multiple ways to resolve this.
I would let maintainer suggest the best approach, and I will do
the needful.

This patches are tested on 4.16 kernel with Cortex-A75 based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V9->V10:
 - Updated commit log for patch 1/4 by Toshi
 - Addressed review comments by Will on patch 3/4

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64



Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  7 +++
 arch/arm64/mm/mmu.c   | 42 ---
 arch/x86/mm/pgtable.c |  8 +---
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 5 files changed, 48 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/23/2018 7:31 PM, Will Deacon wrote:

Hi Chintan,


Hi Will,



[as a side note: I'm confused on the status of this patch series, as part
  of it was reposted separately by Toshi. Please can you work together?]


I will share all 4 patches once again as v10 and take latest version of
1/4 as updated by Toshi.



On Mon, Apr 30, 2018 at 01:11:33PM +0530, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
  arch/arm64/mm/mmu.c | 29 +
  1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..0f651db 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -973,12 +974,32 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pmd_t *table;
+
+   if (pmd_present(READ_ONCE(*pmdp))) {


Might also be worth checking pmd_table here, just in case. (same for pud)


I had that check in v2 as below.

if (pud_val(*pud) && !pud_huge(*pud))

But removed that in v3 as unmap should change this to NONE if it is
not table. I still don't see the need of it.




+   table = __va(pmd_val(*pmdp));


Can you avoid dereferencing *pmdp twice, and instead READ_ONCE into a local
variable, please? (same for pud)


Okay.




+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);


Shouldn't this be pte_free_kernel, to pair with pte_alloc_kernel which
was used to allocate the page in the first place? (similarly for pud)


Okay.




+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   int i;
+
+   if (pud_present(READ_ONCE(*pudp))) {
+   table = __va(pud_val(*pudp));
+   for (i = 0; i < PTRS_PER_PMD; i++)
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));


I think it would be cleaner to write this as a do { ... } while, for
consistency with the ioremap and vmalloc code.


Okay.

I'll raise v10 fixing above things. Thanks for the review.



Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/23/2018 7:31 PM, Will Deacon wrote:

Hi Chintan,


Hi Will,



[as a side note: I'm confused on the status of this patch series, as part
  of it was reposted separately by Toshi. Please can you work together?]


I will share all 4 patches once again as v10 and take latest version of
1/4 as updated by Toshi.



On Mon, Apr 30, 2018 at 01:11:33PM +0530, Chintan Pandya wrote:

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
  1) Clearing off the current pud/pmd entry
  2) Invalidate TLB which could have previously
 valid but not stale entry
  3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
  arch/arm64/mm/mmu.c | 29 +
  1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..0f651db 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #define NO_BLOCK_MAPPINGS	BIT(0)

  #define NO_CONT_MAPPINGS  BIT(1)
@@ -973,12 +974,32 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
  }
  
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)

+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
  {
-   return pud_none(*pud);
+   pmd_t *table;
+
+   if (pmd_present(READ_ONCE(*pmdp))) {


Might also be worth checking pmd_table here, just in case. (same for pud)


I had that check in v2 as below.

if (pud_val(*pud) && !pud_huge(*pud))

But removed that in v3 as unmap should change this to NONE if it is
not table. I still don't see the need of it.




+   table = __va(pmd_val(*pmdp));


Can you avoid dereferencing *pmdp twice, and instead READ_ONCE into a local
variable, please? (same for pud)


Okay.




+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);


Shouldn't this be pte_free_kernel, to pair with pte_alloc_kernel which
was used to allocate the page in the first place? (similarly for pud)


Okay.




+   }
+   return 1;
  }
  
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)

+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
  {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   int i;
+
+   if (pud_present(READ_ONCE(*pudp))) {
+   table = __va(pud_val(*pudp));
+   for (i = 0; i < PTRS_PER_PMD; i++)
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));


I think it would be cleaner to write this as a do { ... } while, for
consistency with the ioremap and vmalloc code.


Okay.

I'll raise v10 fixing above things. Thanks for the review.



Will

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/23/2018 8:04 PM, Kani, Toshi wrote:

On Wed, 2018-05-23 at 15:01 +0100, Will Deacon wrote:

Hi Chintan,

[as a side note: I'm confused on the status of this patch series, as part
  of it was reposted separately by Toshi. Please can you work together?]


I do not know the status of my patch series, either... That being said,
I made my x86 patches based off from Chintan's 1/4 patch (which changes
both x86 and arm) so that my series won't conflict with his.

Chintan,

Hi Toshi,


If you need to update your series before mine's accepted, please make
sure to use the updated 1/4 below. I've updated the descriptions per
review comment.
https://patchwork.kernel.org/patch/10407065/

For the sake of completeness, I will re-push my previous 1/4 but will
take your version of change log. I've seen this and your change log
describes the change better.



Thanks,
-Toshi



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

On 5/23/2018 8:04 PM, Kani, Toshi wrote:

On Wed, 2018-05-23 at 15:01 +0100, Will Deacon wrote:

Hi Chintan,

[as a side note: I'm confused on the status of this patch series, as part
  of it was reposted separately by Toshi. Please can you work together?]


I do not know the status of my patch series, either... That being said,
I made my x86 patches based off from Chintan's 1/4 patch (which changes
both x86 and arm) so that my series won't conflict with his.

Chintan,

Hi Toshi,


If you need to update your series before mine's accepted, please make
sure to use the updated 1/4 below. I've updated the descriptions per
review comment.
https://patchwork.kernel.org/patch/10407065/

For the sake of completeness, I will re-push my previous 1/4 but will
take your version of change log. I've seen this and your change log
describes the change better.



Thanks,
-Toshi



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

On 5/4/2018 3:12 AM, Andrew Morton wrote:

On Tue, 17 Apr 2018 16:13:48 +0530 Chintan Pandya <cpan...@codeaurora.org> 
wrote:


Client can call vunmap with some intermediate 'addr'
which may not be the start of the VM area. Entire
unmap code works with vm->vm_start which is proper
but debug object API is called with 'addr'. This
could be a problem within debug objects.

Pass proper start address into debug object API.

--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1124,15 +1124,15 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(addr > VMALLOC_END);
BUG_ON(!PAGE_ALIGNED(addr));
  
-	debug_check_no_locks_freed(mem, size);

-
if (likely(count <= VMAP_MAX_ALLOC)) {
+   debug_check_no_locks_freed(mem, size);
vb_free(mem, size);
return;
}
  
  	va = find_vmap_area(addr);

BUG_ON(!va);
+   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
free_unmap_vmap_area(va);
  }
  EXPORT_SYMBOL(vm_unmap_ram);


hm, how did this sneak through?

My bad. I had tested them but missed bringing these compile fixes to the
patch file. Will be careful next time.



mm/vmalloc.c:1139:29: warning: passing argument 1 of debug_check_no_locks_freed 
makes pointer from integer without a cast [-Wint-conversion]
   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));

--- a/mm/vmalloc.c~mm-vmalloc-pass-proper-vm_start-into-debugobjects-fix
+++ a/mm/vmalloc.c
@@ -1136,7 +1136,8 @@ void vm_unmap_ram(const void *mem, unsig
  
  	va = find_vmap_area(addr);

BUG_ON(!va);
-   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
+   debug_check_no_locks_freed((void *)va->va_start,
+   (va->va_end - va->va_start));
free_unmap_vmap_area(va);
  }
  EXPORT_SYMBOL(vm_unmap_ram);



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

On 5/4/2018 3:12 AM, Andrew Morton wrote:

On Tue, 17 Apr 2018 16:13:48 +0530 Chintan Pandya  
wrote:


Client can call vunmap with some intermediate 'addr'
which may not be the start of the VM area. Entire
unmap code works with vm->vm_start which is proper
but debug object API is called with 'addr'. This
could be a problem within debug objects.

Pass proper start address into debug object API.

--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1124,15 +1124,15 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(addr > VMALLOC_END);
BUG_ON(!PAGE_ALIGNED(addr));
  
-	debug_check_no_locks_freed(mem, size);

-
if (likely(count <= VMAP_MAX_ALLOC)) {
+   debug_check_no_locks_freed(mem, size);
vb_free(mem, size);
return;
}
  
  	va = find_vmap_area(addr);

BUG_ON(!va);
+   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
free_unmap_vmap_area(va);
  }
  EXPORT_SYMBOL(vm_unmap_ram);


hm, how did this sneak through?

My bad. I had tested them but missed bringing these compile fixes to the
patch file. Will be careful next time.



mm/vmalloc.c:1139:29: warning: passing argument 1 of debug_check_no_locks_freed 
makes pointer from integer without a cast [-Wint-conversion]
   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));

--- a/mm/vmalloc.c~mm-vmalloc-pass-proper-vm_start-into-debugobjects-fix
+++ a/mm/vmalloc.c
@@ -1136,7 +1136,8 @@ void vm_unmap_ram(const void *mem, unsig
  
  	va = find_vmap_area(addr);

BUG_ON(!va);
-   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
+   debug_check_no_locks_freed((void *)va->va_start,
+   (va->va_end - va->va_start));
free_unmap_vmap_area(va);
  }
  EXPORT_SYMBOL(vm_unmap_ram);



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH 1/2] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

[Chintan Pandya]

On 5/2/2018 1:24 PM, Ganesh Mahendran wrote:

Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This
enables Speculative Page Fault handler.

Signed-off-by: Ganesh Mahendran 
---
This patch is on top of Laurent's v10 spf
---
  arch/arm64/Kconfig | 1 +
  1 file changed, 1 insertion(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index eb2cf49..cd583a9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -144,6 +144,7 @@ config ARM64
select SPARSE_IRQ
select SYSCTL_EXCEPTION_TRACE
select THREAD_INFO_IN_TASK
+   select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT if SMP
help
  ARM 64-bit (AArch64) Linux support.
  



You may also consider re-ordering of patches in next version. Generally,
config enablement assumes effectiveness of that config.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH 1/2] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

[Chintan Pandya]

On 5/2/2018 1:24 PM, Ganesh Mahendran wrote:

Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This
enables Speculative Page Fault handler.

Signed-off-by: Ganesh Mahendran 
---
This patch is on top of Laurent's v10 spf
---
  arch/arm64/Kconfig | 1 +
  1 file changed, 1 insertion(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index eb2cf49..cd583a9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -144,6 +144,7 @@ config ARM64
select SPARSE_IRQ
select SYSCTL_EXCEPTION_TRACE
select THREAD_INFO_IN_TASK
+   select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT if SMP
help
  ARM 64-bit (AArch64) Linux support.
  



You may also consider re-ordering of patches in next version. Generally,
config enablement assumes effectiveness of that config.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v3] mm: vmalloc: Clean up vunmap to avoid pgtable ops twice

[Chintan Pandya]

vunmap does page table clear operations twice in the
case when DEBUG_PAGEALLOC_ENABLE_DEFAULT is enabled.

So, clean up the code as that is unintended.

As a perf gain, we save few us. Below ftrace data was
obtained while doing 1 MB of vmalloc/vfree on ARM64
based SoC *without* this patch applied. After this
patch, we can save ~3 us (on 1 extra vunmap_page_range).

  CPU  DURATION  FUNCTION CALLS
  | |   | |   |   |   |
 6)   |  __vunmap() {
 6)   |vmap_debug_free_range() {
 6)   3.281 us|  vunmap_page_range();
 6) + 45.468 us   |}
 6)   2.760 us|vunmap_page_range();
 6) ! 505.105 us  |  }

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 mm/vmalloc.c | 29 +++--
 1 file changed, 7 insertions(+), 22 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ebff729..781ce02 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -603,26 +603,6 @@ static void unmap_vmap_area(struct vmap_area *va)
vunmap_page_range(va->va_start, va->va_end);
 }
 
-static void vmap_debug_free_range(unsigned long start, unsigned long end)
-{
-   /*
-* Unmap page tables and force a TLB flush immediately if pagealloc
-* debugging is enabled.  This catches use after free bugs similarly to
-* those in linear kernel virtual address space after a page has been
-* freed.
-*
-* All the lazy freeing logic is still retained, in order to minimise
-* intrusiveness of this debugging feature.
-*
-* This is going to be *slow* (linear kernel virtual address debugging
-* doesn't do a broadcast TLB flush so it is a lot faster).
-*/
-   if (debug_pagealloc_enabled()) {
-   vunmap_page_range(start, end);
-   flush_tlb_kernel_range(start, end);
-   }
-}
-
 /*
  * lazy_max_pages is the maximum amount of virtual address space we gather up
  * before attempting to purge with a TLB flush.
@@ -756,6 +736,9 @@ static void free_unmap_vmap_area(struct vmap_area *va)
 {
flush_cache_vunmap(va->va_start, va->va_end);
unmap_vmap_area(va);
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range(va->va_start, va->va_end);
+
free_vmap_area_noflush(va);
 }
 
@@ -1053,6 +1036,10 @@ static void vb_free(const void *addr, unsigned long size)
 
vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);
 
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range((unsigned long)addr,
+   (unsigned long)addr + size);
+
spin_lock(>lock);
 
/* Expand dirty range */
@@ -1142,7 +1129,6 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(!PAGE_ALIGNED(addr));
 
debug_check_no_locks_freed(mem, size);
-   vmap_debug_free_range(addr, addr+size);
 
if (likely(count <= VMAP_MAX_ALLOC)) {
vb_free(mem, size);
@@ -1499,7 +1485,6 @@ struct vm_struct *remove_vm_area(const void *addr)
va->flags |= VM_LAZY_FREE;
spin_unlock(_area_lock);
 
-   vmap_debug_free_range(va->va_start, va->va_end);
kasan_free_shadow(vm);
free_unmap_vmap_area(va);
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v3] mm: vmalloc: Clean up vunmap to avoid pgtable ops twice

[Chintan Pandya]

vunmap does page table clear operations twice in the
case when DEBUG_PAGEALLOC_ENABLE_DEFAULT is enabled.

So, clean up the code as that is unintended.

As a perf gain, we save few us. Below ftrace data was
obtained while doing 1 MB of vmalloc/vfree on ARM64
based SoC *without* this patch applied. After this
patch, we can save ~3 us (on 1 extra vunmap_page_range).

  CPU  DURATION  FUNCTION CALLS
  | |   | |   |   |   |
 6)   |  __vunmap() {
 6)   |vmap_debug_free_range() {
 6)   3.281 us|  vunmap_page_range();
 6) + 45.468 us   |}
 6)   2.760 us|vunmap_page_range();
 6) ! 505.105 us  |  }

Signed-off-by: Chintan Pandya 
---
 mm/vmalloc.c | 29 +++--
 1 file changed, 7 insertions(+), 22 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ebff729..781ce02 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -603,26 +603,6 @@ static void unmap_vmap_area(struct vmap_area *va)
vunmap_page_range(va->va_start, va->va_end);
 }
 
-static void vmap_debug_free_range(unsigned long start, unsigned long end)
-{
-   /*
-* Unmap page tables and force a TLB flush immediately if pagealloc
-* debugging is enabled.  This catches use after free bugs similarly to
-* those in linear kernel virtual address space after a page has been
-* freed.
-*
-* All the lazy freeing logic is still retained, in order to minimise
-* intrusiveness of this debugging feature.
-*
-* This is going to be *slow* (linear kernel virtual address debugging
-* doesn't do a broadcast TLB flush so it is a lot faster).
-*/
-   if (debug_pagealloc_enabled()) {
-   vunmap_page_range(start, end);
-   flush_tlb_kernel_range(start, end);
-   }
-}
-
 /*
  * lazy_max_pages is the maximum amount of virtual address space we gather up
  * before attempting to purge with a TLB flush.
@@ -756,6 +736,9 @@ static void free_unmap_vmap_area(struct vmap_area *va)
 {
flush_cache_vunmap(va->va_start, va->va_end);
unmap_vmap_area(va);
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range(va->va_start, va->va_end);
+
free_vmap_area_noflush(va);
 }
 
@@ -1053,6 +1036,10 @@ static void vb_free(const void *addr, unsigned long size)
 
vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);
 
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range((unsigned long)addr,
+   (unsigned long)addr + size);
+
spin_lock(>lock);
 
/* Expand dirty range */
@@ -1142,7 +1129,6 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(!PAGE_ALIGNED(addr));
 
debug_check_no_locks_freed(mem, size);
-   vmap_debug_free_range(addr, addr+size);
 
if (likely(count <= VMAP_MAX_ALLOC)) {
vb_free(mem, size);
@@ -1499,7 +1485,6 @@ struct vm_struct *remove_vm_area(const void *addr)
va->flags |= VM_LAZY_FREE;
spin_unlock(_area_lock);
 
-   vmap_debug_free_range(va->va_start, va->va_end);
kasan_free_shadow(vm);
free_unmap_vmap_area(va);
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2] mm: vmalloc: Clean up vunmap to avoid pgtable ops twice

[Chintan Pandya]

On 5/1/2018 4:22 AM, Andrew Morton wrote:

On Mon, 16 Apr 2018 16:29:02 +0530 Chintan Pandya <cpan...@codeaurora.org> 
wrote:


vunmap does page table clear operations twice in the
case when DEBUG_PAGEALLOC_ENABLE_DEFAULT is enabled.

So, clean up the code as that is unintended.

As a perf gain, we save few us. Below ftrace data was
obtained while doing 1 MB of vmalloc/vfree on ARM64
based SoC *without* this patch applied. After this
patch, we can save ~3 us (on 1 extra vunmap_page_range).

   CPU  DURATION  FUNCTION CALLS
   | |   | |   |   |   |
  6)   |  __vunmap() {
  6)   |vmap_debug_free_range() {
  6)   3.281 us|  vunmap_page_range();
  6) + 45.468 us   |}
  6)   2.760 us|vunmap_page_range();
  6) ! 505.105 us  |  }


It's been a long time since I looked at the vmap code :(


--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -603,26 +603,6 @@ static void unmap_vmap_area(struct vmap_area *va)
vunmap_page_range(va->va_start, va->va_end);
  }
  
-static void vmap_debug_free_range(unsigned long start, unsigned long end)

-{
-   /*
-* Unmap page tables and force a TLB flush immediately if pagealloc
-* debugging is enabled.  This catches use after free bugs similarly to
-* those in linear kernel virtual address space after a page has been
-* freed.
-*
-* All the lazy freeing logic is still retained, in order to minimise
-* intrusiveness of this debugging feature.
-*
-* This is going to be *slow* (linear kernel virtual address debugging
-* doesn't do a broadcast TLB flush so it is a lot faster).
-*/
-   if (debug_pagealloc_enabled()) {
-   vunmap_page_range(start, end);
-   flush_tlb_kernel_range(start, end);
-   }
-}
-
  /*
   * lazy_max_pages is the maximum amount of virtual address space we gather up
   * before attempting to purge with a TLB flush.
@@ -756,6 +736,9 @@ static void free_unmap_vmap_area(struct vmap_area *va)
  {
flush_cache_vunmap(va->va_start, va->va_end);
unmap_vmap_area(va);
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range(va->va_start, va->va_end);
+
free_vmap_area_noflush(va);
  }
  
@@ -1142,7 +1125,6 @@ void vm_unmap_ram(const void *mem, unsigned int count)

BUG_ON(!PAGE_ALIGNED(addr));
  
  	debug_check_no_locks_freed(mem, size);

-   vmap_debug_free_range(addr, addr+size);


This appears to be a functional change: if (count <= VMAP_MAX_ALLOC)
and we're in debug mode then the
vunmap_page_range/flush_tlb_kernel_range will no longer be performed.
Why is this ok?



Yes, you are right. In vb_free(), we do vunmap_page_range() but not
flush_tlb_kernel_range(). I will add this stub for debug benefits and
share v3.

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6729400..781ce02 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1036,6 +1036,10 @@ static void vb_free(const void *addr, unsigned 
long size)


vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);

+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range((unsigned long)addr,
+   (unsigned long)addr + size);
+
spin_lock(>lock);

/* Expand dirty range */




if (likely(count <= VMAP_MAX_ALLOC)) {
vb_free(mem, size);
@@ -1499,7 +1481,6 @@ struct vm_struct *remove_vm_area(const void *addr)
va->flags |= VM_LAZY_FREE;
spin_unlock(_area_lock);
  
-		vmap_debug_free_range(va->va_start, va->va_end);

kasan_free_shadow(vm);
free_unmap_vmap_area(va);
  




Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2] mm: vmalloc: Clean up vunmap to avoid pgtable ops twice

[Chintan Pandya]

On 5/1/2018 4:22 AM, Andrew Morton wrote:

On Mon, 16 Apr 2018 16:29:02 +0530 Chintan Pandya  
wrote:


vunmap does page table clear operations twice in the
case when DEBUG_PAGEALLOC_ENABLE_DEFAULT is enabled.

So, clean up the code as that is unintended.

As a perf gain, we save few us. Below ftrace data was
obtained while doing 1 MB of vmalloc/vfree on ARM64
based SoC *without* this patch applied. After this
patch, we can save ~3 us (on 1 extra vunmap_page_range).

   CPU  DURATION  FUNCTION CALLS
   | |   | |   |   |   |
  6)   |  __vunmap() {
  6)   |vmap_debug_free_range() {
  6)   3.281 us|  vunmap_page_range();
  6) + 45.468 us   |}
  6)   2.760 us|vunmap_page_range();
  6) ! 505.105 us  |  }


It's been a long time since I looked at the vmap code :(


--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -603,26 +603,6 @@ static void unmap_vmap_area(struct vmap_area *va)
vunmap_page_range(va->va_start, va->va_end);
  }
  
-static void vmap_debug_free_range(unsigned long start, unsigned long end)

-{
-   /*
-* Unmap page tables and force a TLB flush immediately if pagealloc
-* debugging is enabled.  This catches use after free bugs similarly to
-* those in linear kernel virtual address space after a page has been
-* freed.
-*
-* All the lazy freeing logic is still retained, in order to minimise
-* intrusiveness of this debugging feature.
-*
-* This is going to be *slow* (linear kernel virtual address debugging
-* doesn't do a broadcast TLB flush so it is a lot faster).
-*/
-   if (debug_pagealloc_enabled()) {
-   vunmap_page_range(start, end);
-   flush_tlb_kernel_range(start, end);
-   }
-}
-
  /*
   * lazy_max_pages is the maximum amount of virtual address space we gather up
   * before attempting to purge with a TLB flush.
@@ -756,6 +736,9 @@ static void free_unmap_vmap_area(struct vmap_area *va)
  {
flush_cache_vunmap(va->va_start, va->va_end);
unmap_vmap_area(va);
+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range(va->va_start, va->va_end);
+
free_vmap_area_noflush(va);
  }
  
@@ -1142,7 +1125,6 @@ void vm_unmap_ram(const void *mem, unsigned int count)

BUG_ON(!PAGE_ALIGNED(addr));
  
  	debug_check_no_locks_freed(mem, size);

-   vmap_debug_free_range(addr, addr+size);


This appears to be a functional change: if (count <= VMAP_MAX_ALLOC)
and we're in debug mode then the
vunmap_page_range/flush_tlb_kernel_range will no longer be performed.
Why is this ok?



Yes, you are right. In vb_free(), we do vunmap_page_range() but not
flush_tlb_kernel_range(). I will add this stub for debug benefits and
share v3.

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6729400..781ce02 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1036,6 +1036,10 @@ static void vb_free(const void *addr, unsigned 
long size)


vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);

+   if (debug_pagealloc_enabled())
+   flush_tlb_kernel_range((unsigned long)addr,
+   (unsigned long)addr + size);
+
spin_lock(>lock);

/* Expand dirty range */




if (likely(count <= VMAP_MAX_ALLOC)) {
vb_free(mem, size);
@@ -1499,7 +1481,6 @@ struct vm_struct *remove_vm_area(const void *addr)
va->flags |= VM_LAZY_FREE;
spin_unlock(_area_lock);
  
-		vmap_debug_free_range(va->va_start, va->va_end);

kasan_free_shadow(vm);
free_unmap_vmap_area(va);
  




Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

On 5/1/2018 4:34 AM, Andrew Morton wrote:

should check for it and do a WARN_ONCE so it gets fixed.


Yes, that was an idea in discussion but I've been suggested that it
could be intentional. But since you are raising this, I will try to dig
once again and share a patch with WARN_ONCE if passing intermediate
'addr' is absolutely not right thing to do.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

On 5/1/2018 4:34 AM, Andrew Morton wrote:

should check for it and do a WARN_ONCE so it gets fixed.


Yes, that was an idea in discussion but I've been suggested that it
could be intentional. But since you are raising this, I will try to dig
once again and share a patch with WARN_ONCE if passing intermediate
'addr' is absolutely not right thing to do.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 29 +
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..0f651db 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,32 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pmd_t *table;
+
+   if (pmd_present(READ_ONCE(*pmdp))) {
+   table = __va(pmd_val(*pmdp));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   int i;
+
+   if (pud_present(READ_ONCE(*pudp))) {
+   table = __va(pud_val(*pudp));
+   for (i = 0; i < PTRS_PER_PMD; i++)
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 3/4] arm64: Implement page table free interfaces

[Chintan Pandya]

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
 1) Clearing off the current pud/pmd entry
 2) Invalidate TLB which could have previously
valid but not stale entry
 3) Freeing of the un-used next level page tables

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 29 +
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index da98828..0f651db 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -45,6 +45,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define NO_BLOCK_MAPPINGS  BIT(0)
 #define NO_CONT_MAPPINGS   BIT(1)
@@ -973,12 +974,32 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-   return pud_none(*pud);
+   pmd_t *table;
+
+   if (pmd_present(READ_ONCE(*pmdp))) {
+   table = __va(pmd_val(*pmdp));
+   pmd_clear(pmdp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);
+   }
+   return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-   return pmd_none(*pmd);
+   pmd_t *table;
+   int i;
+
+   if (pud_present(READ_ONCE(*pudp))) {
+   table = __va(pud_val(*pudp));
+   for (i = 0; i < PTRS_PER_PMD; i++)
+   pmd_free_pte_page([i], addr + (i * PMD_SIZE));
+
+   pud_clear(pudp);
+   __flush_tlb_kernel_pgtable(addr);
+   free_page((unsigned long) table);
+   }
+   return 1;
 }
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

This patch ("mm/vmalloc: Add interfaces to free unmapped
page table") adds following 2 interfaces to free the page
table in case we implement huge mapping.

pud_free_pmd_page() and pmd_free_pte_page()

Some architectures (like arm64) needs to do proper TLB
maintanance after updating pagetable entry even in map.
Why ? Read this,
https://patchwork.kernel.org/patch/10134581/

Pass 'addr' in these interfaces so that proper TLB ops
can be performed.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_enabled() &&
((next - addr) == PUD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PUD_SIZE) &&
-   pud_free_pmd_page(pud)) {
+   pud_free_pmd_page(pud, addr)) {
if (pud_set_huge(pud, phys_addr + addr, prot))
continue;
}
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 0f651db..170009b 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 1/4] ioremap: Update pgtable free interfaces with addr

[Chintan Pandya]

This patch ("mm/vmalloc: Add interfaces to free unmapped
page table") adds following 2 interfaces to free the page
table in case we implement huge mapping.

pud_free_pmd_page() and pmd_free_pte_page()

Some architectures (like arm64) needs to do proper TLB
maintanance after updating pagetable entry even in map.
Why ? Read this,
https://patchwork.kernel.org/patch/10134581/

Pass 'addr' in these interfaces so that proper TLB ops
can be performed.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c   | 4 ++--
 arch/x86/mm/pgtable.c | 8 +---
 include/asm-generic/pgtable.h | 8 
 lib/ioremap.c | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 2dbb2c9..da98828 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return pmd_none(*pmd);
 }
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index ffc8c13..37e3cba 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -718,11 +718,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
pmd_t *pmd;
int i;
@@ -733,7 +734,7 @@ int pud_free_pmd_page(pud_t *pud)
pmd = (pmd_t *)pud_page_vaddr(*pud);
 
for (i = 0; i < PTRS_PER_PMD; i++)
-   if (!pmd_free_pte_page([i]))
+   if (!pmd_free_pte_page([i], addr + (i * PMD_SIZE)))
return 0;
 
pud_clear(pud);
@@ -745,11 +746,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
pte_t *pte;
 
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f59639a..b081794 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -1019,8 +1019,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else  /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {
@@ -1046,11 +1046,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
return 0;
 }
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 54e5bba..517f585 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long 
addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-   pmd_free_pte_page(pmd)) {
+   pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned 
long addr,
if (ioremap_pud_enabled() &&
((next - addr) == PUD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PUD_SIZE) &&
-   pud_free_pmd_page(pud)) {
+   pud_free_pmd_page(pud, addr)) {
if (pud_set_huge(pud, phys_addr + addr, prot))
continue;
}
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 4/4] Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

[Chintan Pandya]

This commit 15122ee2c515a ("arm64: Enforce BBM for huge
IO/VMAP mappings") is a temporary work-around until the
issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/mm/mmu.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 0f651db..170009b 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pud_present(READ_ONCE(*pudp)))
-   return 0;
-
BUG_ON(phys & ~PUD_MASK);
set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
return 1;
@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t 
prot)
pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
pgprot_val(mk_sect_prot(prot)));
 
-   /* ioremap_page_range doesn't honour BBM */
-   if (pmd_present(READ_ONCE(*pmdp)))
-   return 0;
-
BUG_ON(phys & ~PMD_MASK);
set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
return 1;
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
 arch/arm64/include/asm/tlbflush.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index dfc61d7..a4a1901 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -218,6 +218,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
+{
+   unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
 #endif
 
 #endif
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches takes Toshi Kani <toshi.k...@hpe.com>'s
patches ("fix memory leak/panic in ioremap huge pages") as base
and re-bring huge_vmap back for arm64.

This series of patches are tested on 4.16 kernel with Cortex-A75
based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  6 ++
 arch/arm64/mm/mmu.c   | 37 +
 arch/x86/mm/pgtable.c |  8 +---
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 5 files changed, 42 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v9 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

This series of patches takes Toshi Kani 's
patches ("fix memory leak/panic in ioremap huge pages") as base
and re-bring huge_vmap back for arm64.

This series of patches are tested on 4.16 kernel with Cortex-A75
based SoC.

The test used for verifying these patches is a stress test on
ioremap/unmap which tries to re-use same io-address but changes
size of mapping randomly i.e. 4K to 2M to 16K etc. The same test
used to reproduce 3rd level translation fault without these fixes
(and also of course with Revert "arm64: Enforce BBM for huge IO/VMAP
mappings" being part of the tree).

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

>From V8->V9:
 - Used __TLBI_VADDR macros in new TLB flush API

>From V7->V8:
 - Properly fixed compilation issue in x86 file

>From V6->V7:
 - Fixed compilation issue in x86 case
 - V6 patches were not properly enumarated

>From V5->V6:
 - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
   "bool tlb_inv" based variance as it is not need now
 - Re-naming for consistency

>From V4->V5:
 - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
   for kernel addresses

>From V3->V4:
 - Add header for 'addr' in x86 implementation
 - Re-order pmd/pud clear and table free
 - Avoid redundant TLB invalidatation in one perticular case

>From V2->V3:
 - Use the exisiting page table free interface to do arm64
   specific things

>From V1->V2:
 - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
   Add interfaces to free unmapped page table"
 - Honored BBM for ARM64

Chintan Pandya (4):
  ioremap: Update pgtable free interfaces with addr
  arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
  arm64: Implement page table free interfaces
  Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  6 ++
 arch/arm64/mm/mmu.c   | 37 +
 arch/x86/mm/pgtable.c |  8 +---
 include/asm-generic/pgtable.h |  8 
 lib/ioremap.c |  4 ++--
 5 files changed, 42 insertions(+), 21 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/29/2018 2:24 AM, Kani, Toshi wrote:

On Sat, 2018-04-28 at 11:02 +0200, j...@8bytes.org wrote:

On Fri, Apr 27, 2018 at 02:31:51PM +, Kani, Toshi wrote:

So, we can add the step 2 on top of this patch.
  1. Clear pud/pmd entry.
  2. System wide TLB flush <-- TO BE ADDED BY NEW PATCH
  3. Free its underlining pmd/pte page.


This still lacks the page-table synchronization and will thus not fix
the BUG_ON being triggered.


The BUG_ON issue is specific to PAE that it syncs at the pmd level.
x86/64 does not have this issue since it syncs at the pgd or p4d level.


We do not need to revert this patch.  We can make the above change I
mentioned.


Please note that we are not in the merge window anymore and that any fix
needs to be simple and obviously correct.


Understood.  Changing the x86/32 sync point is risky.  So, I am going to
revert the free page handling for PAE.


Will this affect pmd_free_pte_page() & pud_free_pmd_page() 's existence
or its parameters ? I'm asking because, I've similar change for arm64
and ready to send v9 patches.

I'm thinking to share my v9 patches in any case. If you are going to do
TLB invalidation within these APIs, my first patch will help.



Thanks,
-Toshi

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/29/2018 2:24 AM, Kani, Toshi wrote:

On Sat, 2018-04-28 at 11:02 +0200, j...@8bytes.org wrote:

On Fri, Apr 27, 2018 at 02:31:51PM +, Kani, Toshi wrote:

So, we can add the step 2 on top of this patch.
  1. Clear pud/pmd entry.
  2. System wide TLB flush <-- TO BE ADDED BY NEW PATCH
  3. Free its underlining pmd/pte page.


This still lacks the page-table synchronization and will thus not fix
the BUG_ON being triggered.


The BUG_ON issue is specific to PAE that it syncs at the pmd level.
x86/64 does not have this issue since it syncs at the pgd or p4d level.


We do not need to revert this patch.  We can make the above change I
mentioned.


Please note that we are not in the merge window anymore and that any fix
needs to be simple and obviously correct.


Understood.  Changing the x86/32 sync point is risky.  So, I am going to
revert the free page handling for PAE.


Will this affect pmd_free_pte_page() & pud_free_pmd_page() 's existence
or its parameters ? I'm asking because, I've similar change for arm64
and ready to send v9 patches.

I'm thinking to share my v9 patches in any case. If you are going to do
TLB invalidation within these APIs, my first patch will help.



Thanks,
-Toshi

___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/27/2018 6:18 PM, j...@8bytes.org wrote:

On Fri, Apr 27, 2018 at 05:22:28PM +0530, Chintan Pandya wrote:

I'm bit confused here. Are you pointing to race within ioremap/vmalloc
framework while updating the page table or race during tlb ops. Since
later is arch dependent, I would not comment. But if the race being
discussed here while altering page tables, I'm not on the same page.


The race condition is between hardware and software. It is not
sufficient to just remove the software references to the page that is
about to be freed (by clearing the PMD/PUD), also the hardware
references in the page-walk cache need to be removed with a TLB flush.
Otherwise the hardware can use the freed (and possibly reused) page to
establish new TLB entries.


Sure ! This is my understanding too (from arm64 context).





Joerg


___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/27/2018 6:18 PM, j...@8bytes.org wrote:

On Fri, Apr 27, 2018 at 05:22:28PM +0530, Chintan Pandya wrote:

I'm bit confused here. Are you pointing to race within ioremap/vmalloc
framework while updating the page table or race during tlb ops. Since
later is arch dependent, I would not comment. But if the race being
discussed here while altering page tables, I'm not on the same page.


The race condition is between hardware and software. It is not
sufficient to just remove the software references to the page that is
about to be freed (by clearing the PMD/PUD), also the hardware
references in the page-walk cache need to be removed with a TLB flush.
Otherwise the hardware can use the freed (and possibly reused) page to
establish new TLB entries.


Sure ! This is my understanding too (from arm64 context).





Joerg


___
linux-arm-kernel mailing list
linux-arm-ker...@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v8 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

On 4/27/2018 3:59 PM, Catalin Marinas wrote:

On Tue, Apr 03, 2018 at 01:30:44PM +0530, Chintan Pandya wrote:

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
  arch/arm64/include/asm/tlbflush.h | 6 ++
  1 file changed, 6 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index 9e82dd7..6a4816d 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -209,6 +209,12 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
  }
  
+static inline void __flush_tlb_kernel_pgtable(unsigned long addr)

+{
+   addr >>= 12;
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
  #endif


Please use __TLBI_VADDR here as it does some additional masking.


Sure.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v8 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

[Chintan Pandya]

On 4/27/2018 3:59 PM, Catalin Marinas wrote:

On Tue, Apr 03, 2018 at 01:30:44PM +0530, Chintan Pandya wrote:

Add an interface to invalidate intermediate page tables
from TLB for kernel.

Signed-off-by: Chintan Pandya 
---
  arch/arm64/include/asm/tlbflush.h | 6 ++
  1 file changed, 6 insertions(+)

diff --git a/arch/arm64/include/asm/tlbflush.h 
b/arch/arm64/include/asm/tlbflush.h
index 9e82dd7..6a4816d 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -209,6 +209,12 @@ static inline void __flush_tlb_pgtable(struct mm_struct 
*mm,
dsb(ish);
  }
  
+static inline void __flush_tlb_kernel_pgtable(unsigned long addr)

+{
+   addr >>= 12;
+   __tlbi(vaae1is, addr);
+   dsb(ish);
+}
  #endif


Please use __TLBI_VADDR here as it does some additional masking.


Sure.

Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/27/2018 1:07 PM, j...@8bytes.org wrote:

On Thu, Apr 26, 2018 at 10:30:14PM +, Kani, Toshi wrote:

Thanks for the clarification. After reading through SDM one more time, I
agree that we need a TLB purge here. Here is my current understanding.

  - INVLPG purges both TLB and paging-structure caches. So, PMD cache was
purged once.
  - However, processor may cache this PMD entry later in speculation
since it has p-bit set. (This is where my misunderstanding was.
Speculation is not allowed to access a target address, but it may still
cache this PMD entry.)
  - A single INVLPG on each processor purges this PMD cache. It does not
need a range purge (which was already done).

Does it sound right to you?


The right fix is to first synchronize the changes when the PMD/PUD is
cleared and then flush the TLB system-wide. After that is done you can
free the page.



I'm bit confused here. Are you pointing to race within ioremap/vmalloc
framework while updating the page table or race during tlb ops. Since
later is arch dependent, I would not comment. But if the race being 
discussed here while altering page tables, I'm not on the same page.


Current ioremap/vmalloc framework works with reserved virtual area for 
its own purpose. Within this virtual area, we maintain mutual 
exclusiveness by maintaining separate rbtree which is of course 
synchronized. In the __vunmap leg, we perform page table ops first and
then release the virtual area for someone else to re-use. This way, 
without taking any additional locks for page table modifications, we are

good.

If that's not the case and I'm missing something here.

Also, I'm curious to know what race you are observing at your end.


Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v2 2/2] x86/mm: implement free pmd/pte page interfaces

[Chintan Pandya]

On 4/27/2018 1:07 PM, j...@8bytes.org wrote:

On Thu, Apr 26, 2018 at 10:30:14PM +, Kani, Toshi wrote:

Thanks for the clarification. After reading through SDM one more time, I
agree that we need a TLB purge here. Here is my current understanding.

  - INVLPG purges both TLB and paging-structure caches. So, PMD cache was
purged once.
  - However, processor may cache this PMD entry later in speculation
since it has p-bit set. (This is where my misunderstanding was.
Speculation is not allowed to access a target address, but it may still
cache this PMD entry.)
  - A single INVLPG on each processor purges this PMD cache. It does not
need a range purge (which was already done).

Does it sound right to you?


The right fix is to first synchronize the changes when the PMD/PUD is
cleared and then flush the TLB system-wide. After that is done you can
free the page.



I'm bit confused here. Are you pointing to race within ioremap/vmalloc
framework while updating the page table or race during tlb ops. Since
later is arch dependent, I would not comment. But if the race being 
discussed here while altering page tables, I'm not on the same page.


Current ioremap/vmalloc framework works with reserved virtual area for 
its own purpose. Within this virtual area, we maintain mutual 
exclusiveness by maintaining separate rbtree which is of course 
synchronized. In the __vunmap leg, we perform page table ops first and
then release the virtual area for someone else to re-use. This way, 
without taking any additional locks for page table modifications, we are

good.

If that's not the case and I'm missing something here.

Also, I'm curious to know what race you are observing at your end.


Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 1/2] mm: vmalloc: Avoid racy handling of debugobjects in vunmap

[Chintan Pandya]

Currently, __vunmap flow is,
 1) Release the VM area
 2) Free the debug objects corresponding to that vm area.

This leave some race window open.
 1) Release the VM area
 1.5) Some other client gets the same vm area
 1.6) This client allocates new debug objects on the same
  vm area
 2) Free the debug objects corresponding to this vm area.

Here, we actually free 'other' client's debug objects.

Fix this by freeing the debug objects first and then
releasing the VM area.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 mm/vmalloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6729400..12d675c 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1500,7 +1500,7 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
addr))
return;
 
-   area = remove_vm_area(addr);
+   area = find_vmap_area((unsigned long)addr)->vm;
if (unlikely(!area)) {
WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n",
addr);
@@ -1510,6 +1510,7 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
debug_check_no_locks_freed(addr, get_vm_area_size(area));
debug_check_no_obj_freed(addr, get_vm_area_size(area));
 
+   remove_vm_area(addr);
if (deallocate_pages) {
int i;
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 1/2] mm: vmalloc: Avoid racy handling of debugobjects in vunmap

[Chintan Pandya]

Currently, __vunmap flow is,
 1) Release the VM area
 2) Free the debug objects corresponding to that vm area.

This leave some race window open.
 1) Release the VM area
 1.5) Some other client gets the same vm area
 1.6) This client allocates new debug objects on the same
  vm area
 2) Free the debug objects corresponding to this vm area.

Here, we actually free 'other' client's debug objects.

Fix this by freeing the debug objects first and then
releasing the VM area.

Signed-off-by: Chintan Pandya 
---
 mm/vmalloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6729400..12d675c 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1500,7 +1500,7 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
addr))
return;
 
-   area = remove_vm_area(addr);
+   area = find_vmap_area((unsigned long)addr)->vm;
if (unlikely(!area)) {
WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n",
addr);
@@ -1510,6 +1510,7 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
debug_check_no_locks_freed(addr, get_vm_area_size(area));
debug_check_no_obj_freed(addr, get_vm_area_size(area));
 
+   remove_vm_area(addr);
if (deallocate_pages) {
int i;
 
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

Client can call vunmap with some intermediate 'addr'
which may not be the start of the VM area. Entire
unmap code works with vm->vm_start which is proper
but debug object API is called with 'addr'. This
could be a problem within debug objects.

Pass proper start address into debug object API.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
 mm/vmalloc.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 12d675c..033c918 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1124,15 +1124,15 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(addr > VMALLOC_END);
BUG_ON(!PAGE_ALIGNED(addr));
 
-   debug_check_no_locks_freed(mem, size);
-
if (likely(count <= VMAP_MAX_ALLOC)) {
+   debug_check_no_locks_freed(mem, size);
vb_free(mem, size);
return;
}
 
va = find_vmap_area(addr);
BUG_ON(!va);
+   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
free_unmap_vmap_area(va);
 }
 EXPORT_SYMBOL(vm_unmap_ram);
@@ -1507,8 +1507,8 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
return;
}
 
-   debug_check_no_locks_freed(addr, get_vm_area_size(area));
-   debug_check_no_obj_freed(addr, get_vm_area_size(area));
+   debug_check_no_locks_freed(area->addr, get_vm_area_size(area));
+   debug_check_no_obj_freed(area->addr, get_vm_area_size(area));
 
remove_vm_area(addr);
if (deallocate_pages) {
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

Client can call vunmap with some intermediate 'addr'
which may not be the start of the VM area. Entire
unmap code works with vm->vm_start which is proper
but debug object API is called with 'addr'. This
could be a problem within debug objects.

Pass proper start address into debug object API.

Signed-off-by: Chintan Pandya 
---
 mm/vmalloc.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 12d675c..033c918 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1124,15 +1124,15 @@ void vm_unmap_ram(const void *mem, unsigned int count)
BUG_ON(addr > VMALLOC_END);
BUG_ON(!PAGE_ALIGNED(addr));
 
-   debug_check_no_locks_freed(mem, size);
-
if (likely(count <= VMAP_MAX_ALLOC)) {
+   debug_check_no_locks_freed(mem, size);
vb_free(mem, size);
return;
}
 
va = find_vmap_area(addr);
BUG_ON(!va);
+   debug_check_no_locks_freed(va->va_start, (va->va_end - va->va_start));
free_unmap_vmap_area(va);
 }
 EXPORT_SYMBOL(vm_unmap_ram);
@@ -1507,8 +1507,8 @@ static void __vunmap(const void *addr, int 
deallocate_pages)
return;
}
 
-   debug_check_no_locks_freed(addr, get_vm_area_size(area));
-   debug_check_no_obj_freed(addr, get_vm_area_size(area));
+   debug_check_no_locks_freed(area->addr, get_vm_area_size(area));
+   debug_check_no_obj_freed(area->addr, get_vm_area_size(area));
 
remove_vm_area(addr);
if (deallocate_pages) {
-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 0/2] vunmap and debug objects

[Chintan Pandya]

I'm not entirely sure, how debug objects are really
useful in vmalloc framework.

I'm assuming they are useful in some ways. So, there
are 2 issues in that. First patch is avoiding possible
race scenario and second patch passes _proper_ args
in debug object APIs. Both these patches can help
debug objects to be in consistent state.

We've observed some list corruptions in debug objects.
However, no claims that these patches will be fixing
them.

If one has an opinion that debug object has no use in
vmalloc framework, I would raise a patch to remove
them from the vunmap leg.

Below 2 patches are rebased over tip + my other patch in
review "[PATCH v2] mm: vmalloc: Clean up vunmap to avoid
pgtable ops twice"

Chintan Pandya (2):
  mm: vmalloc: Avoid racy handling of debugobjects in vunmap
  mm: vmalloc: Pass proper vm_start into debugobjects

>From V1->V2:
 - Incorporated Anshuman's comment about missing corrections
   in vm_unmap_ram()

 mm/vmalloc.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

[PATCH v2 0/2] vunmap and debug objects

[Chintan Pandya]

I'm not entirely sure, how debug objects are really
useful in vmalloc framework.

I'm assuming they are useful in some ways. So, there
are 2 issues in that. First patch is avoiding possible
race scenario and second patch passes _proper_ args
in debug object APIs. Both these patches can help
debug objects to be in consistent state.

We've observed some list corruptions in debug objects.
However, no claims that these patches will be fixing
them.

If one has an opinion that debug object has no use in
vmalloc framework, I would raise a patch to remove
them from the vunmap leg.

Below 2 patches are rebased over tip + my other patch in
review "[PATCH v2] mm: vmalloc: Clean up vunmap to avoid
pgtable ops twice"

Chintan Pandya (2):
  mm: vmalloc: Avoid racy handling of debugobjects in vunmap
  mm: vmalloc: Pass proper vm_start into debugobjects

>From V1->V2:
 - Incorporated Anshuman's comment about missing corrections
   in vm_unmap_ram()

 mm/vmalloc.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

-- 
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v8 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Ping...

On 4/3/2018 1:30 PM, Chintan Pandya wrote:

This series of patches are follow up work (and depends on)
Toshi Kani <toshi.k...@hpe.com>'s patches "fix memory leak/
panic in ioremap huge pages".

This series of patches are tested on 4.9 kernel with Cortex-A75
based SoC.

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

 From V7->V8:
  - Properly fixed compilation issue in x86 file

 From V6->V7:
  - Fixed compilation issue in x86 case
  - V6 patches were not properly enumarated

 From V5->V6:
  - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
"bool tlb_inv" based variance as it is not need now
  - Re-naming for consistency

 From V4->V5:
  - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
for kernel addresses

 From V3->V4:
  - Add header for 'addr' in x86 implementation
  - Re-order pmd/pud clear and table free
  - Avoid redundant TLB invalidatation in one perticular case

 From V2->V3:
  - Use the exisiting page table free interface to do arm64
specific things

 From V1->V2:
  - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
  - Honored BBM for ARM64

Chintan Pandya (4):
   ioremap: Update pgtable free interfaces with addr
   arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
   arm64: Implement page table free interfaces
   Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

  arch/arm64/include/asm/tlbflush.h |  6 ++
  arch/arm64/mm/mmu.c   | 37 +
  arch/x86/mm/pgtable.c |  8 +---
  include/asm-generic/pgtable.h |  8 
  lib/ioremap.c |  4 ++--
  5 files changed, 42 insertions(+), 21 deletions(-)



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH v8 0/4] Fix issues with huge mapping in ioremap for ARM64

[Chintan Pandya]

Ping...

On 4/3/2018 1:30 PM, Chintan Pandya wrote:

This series of patches are follow up work (and depends on)
Toshi Kani 's patches "fix memory leak/
panic in ioremap huge pages".

This series of patches are tested on 4.9 kernel with Cortex-A75
based SoC.

These patches can also go into '-stable' branch (if accepted)
for 4.6 onwards.

 From V7->V8:
  - Properly fixed compilation issue in x86 file

 From V6->V7:
  - Fixed compilation issue in x86 case
  - V6 patches were not properly enumarated

 From V5->V6:
  - Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove
"bool tlb_inv" based variance as it is not need now
  - Re-naming for consistency

 From V4->V5:
  - Add new API __flush_tlb_kernel_pgtable(unsigned long addr)
for kernel addresses

 From V3->V4:
  - Add header for 'addr' in x86 implementation
  - Re-order pmd/pud clear and table free
  - Avoid redundant TLB invalidatation in one perticular case

 From V2->V3:
  - Use the exisiting page table free interface to do arm64
specific things

 From V1->V2:
  - Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
  - Honored BBM for ARM64

Chintan Pandya (4):
   ioremap: Update pgtable free interfaces with addr
   arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
   arm64: Implement page table free interfaces
   Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

  arch/arm64/include/asm/tlbflush.h |  6 ++
  arch/arm64/mm/mmu.c   | 37 +
  arch/x86/mm/pgtable.c |  8 +---
  include/asm-generic/pgtable.h |  8 
  lib/ioremap.c |  4 ++--
  5 files changed, 42 insertions(+), 21 deletions(-)



Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project

Re: [PATCH 2/2] mm: vmalloc: Pass proper vm_start into debugobjects

[Chintan Pandya]

On 4/17/2018 8:39 AM, Anshuman Khandual wrote:

On 04/16/2018 05:39 PM, Chintan Pandya wrote:



On 4/13/2018 5:31 PM, Anshuman Khandual wrote:

On 04/13/2018 05:03 PM, Chintan Pandya wrote:

Client can call vunmap with some intermediate 'addr'
which may not be the start of the VM area. Entire
unmap code works with vm->vm_start which is proper
but debug object API is called with 'addr'. This
could be a problem within debug objects.

Pass proper start address into debug object API.

Signed-off-by: Chintan Pandya <cpan...@codeaurora.org>
---
   mm/vmalloc.c | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 9ff21a1..28034c55 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1526,8 +1526,8 @@ static void __vunmap(const void *addr, int
deallocate_pages)
   return;
   }
   -debug_check_no_locks_freed(addr, get_vm_area_size(area));
-debug_check_no_obj_freed(addr, get_vm_area_size(area));
+debug_check_no_locks_freed(area->addr, get_vm_area_size(area));
+debug_check_no_obj_freed(area->addr, get_vm_area_size(area));


This kind of makes sense to me but I am not sure. We also have another
instance of this inside the function vm_unmap_ram() where we call for

Right, I missed it. I plan to add below stub in v2.

--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1124,15 +1124,15 @@ void vm_unmap_ram(const void *mem, unsigned int
count)
 BUG_ON(addr > VMALLOC_END);
 BUG_ON(!PAGE_ALIGNED(addr));

-   debug_check_no_locks_freed(mem, size);
-
 if (likely(count <= VMAP_MAX_ALLOC)) {
+   debug_check_no_locks_freed(mem, size);


It should have been 'va->va_start' instead of 'mem' in here but as
said before it looks correct to me but I am not really sure.


vb_free() doesn't honor va->va_start. If mem is not va_start and
deliberate, one will provide proper size. And that should be okay
to do as per the code. So, I don't think this particular debug_check
should have passed va_start in args.





Chintan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project