From: "Barry Song (Xiaomi)" <[email protected]> Try to align the vmap virtual address to PMD_SHIFT or a larger PTE mapping size hinted by the architecture, so contiguous pages can be batch-mapped when setting PMD or PTE entries. We do not expect any significant overhead for this.
Add __get_vm_area_node_aligned_caller() as a wrapper over __get_vm_area_node() to simplify repeated calls with fixed arguments. Signed-off-by: Barry Song (Xiaomi) <[email protected]> Signed-off-by: Wen Jiang <[email protected]> Tested-by: Xueyuan Chen <[email protected]> Tested-by: Leo Yan <[email protected]> Reviewed-by: Uladzislau Rezki (Sony) <[email protected]> Reviewed-by: Dev Jain <[email protected]> --- Sashiko review: raised two concerns: - Hardcoding GFP_KERNEL in __get_vm_area_node_aligned_caller() strips the caller's ability to prevent blocking, reclaim, or warnings. - The speculative PMD-aligned allocation, on failure, could trigger spurious vmalloc purges (reclaim_and_purge_vmap_areas(), synchronous TLB flushes, shrinker storms), a possible DoS for vmap() >= PMD_SIZE. Both follow existing behavior: GFP_KERNEL matches the adjacent get_vm_area_caller(), which hardcodes it the same way. The purge concern is not new -- __vmalloc_node_range() already attempts PMD-aligned allocation in __get_vm_area_node() with the same fallback through alloc_vmap_area() overflow/purge, so vmap_get_aligned_vm_area() adds no new purge behavior. mm/vmalloc.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 42b3addd20f9..e15bcef804fc 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3361,6 +3361,14 @@ struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } +static struct vm_struct *__get_vm_area_node_aligned_caller(unsigned long size, + unsigned long align, unsigned long flags, const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + NUMA_NO_NODE, GFP_KERNEL, caller); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address @@ -3692,6 +3700,30 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end, return err; } +static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size, + unsigned long flags, pgprot_t prot, const void *caller) +{ + struct vm_struct *vm_area; + unsigned int shift; + + if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE) { + vm_area = __get_vm_area_node_aligned_caller(size, PMD_SIZE, + flags, caller); + if (vm_area) + return vm_area; + } + + shift = arch_vmap_pte_supported_shift(size); + if (shift > PAGE_SHIFT) { + vm_area = __get_vm_area_node_aligned_caller(size, 1UL << shift, + flags, caller); + if (vm_area) + return vm_area; + } + + return __get_vm_area_node_aligned_caller(size, PAGE_SIZE, flags, caller); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3730,7 +3762,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; size = (unsigned long)count << PAGE_SHIFT; - area = get_vm_area_caller(size, flags, __builtin_return_address(0)); + area = vmap_get_aligned_vm_area(size, flags, prot, + __builtin_return_address(0)); if (!area) return NULL; -- 2.34.1
