bpf_map_alloc_pages() can use a non-blocking allocator that never reclaims. Add bpf_map_alloc_page_sleepable() which always uses the blocking allocator, so the allocation can reclaim. It uses __GFP_RETRY_MAYFAIL so it never invokes the OOM killer: the page is charged to the map's memcg, which need not be the caller's, so an OOM there could kill unrelated tasks in the map's cgroup while a foreign caller could never be its victim. The caller is responsible for a context where blocking is safe; see the function comment.
Like the other bpf map allocators it places the page on the map's numa_node and does not follow the faulting task's NUMA mempolicy; arena memory is shared, so the map's node is the right placement policy. The next patch uses it from the arena page fault handler. Signed-off-by: Jiayuan Chen <[email protected]> Reviewed-by: Emil Tsalapatis <[email protected]> --- To sashiko: - The alloc_pages_nolock() fallback still zeroes: it forces __GFP_ZERO internally and only accepts __GFP_ACCOUNT. - __GFP_ZERO is unchanged from the existing __bpf_alloc_page(), and no arena-capable arch has D-cache aliasing, so there is no dcache concern. --- include/linux/bpf.h | 1 + kernel/bpf/syscall.c | 28 ++++++++++++++++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e80963971f680..345fe4c2e6422 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2786,6 +2786,7 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **page_array); +struct page *bpf_map_alloc_page_sleepable(const struct bpf_map *map); #ifdef CONFIG_MEMCG void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg, struct mem_cgroup **new_memcg); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index c7bc9ba9b331f..abd0f67c68d97 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -602,15 +602,14 @@ static bool can_alloc_pages(void) !IS_ENABLED(CONFIG_PREEMPT_RT); } +#define BPF_PAGE_GFP (GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT | __GFP_NOWARN) + static struct page *__bpf_alloc_page(int nid) { if (!can_alloc_pages()) return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0); - return alloc_pages_node(nid, - GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT - | __GFP_NOWARN, - 0); + return alloc_pages_node(nid, BPF_PAGE_GFP, 0); } int bpf_map_alloc_pages(const struct bpf_map *map, int nid, @@ -636,6 +635,27 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid, return ret; } +/* + * Allocate a page for map memory with the blocking allocator so it can + * reclaim. __GFP_RETRY_MAYFAIL keeps it from invoking the OOM killer: the + * page is charged to the map's memcg, which need not be the caller's, so + * an OOM there could kill unrelated tasks in the map's cgroup while a + * foreign caller could never be its victim. + * + * bpf_map_alloc_pages() serves arbitrary BPF program context and stays + * reentrancy-safe by falling back to the non-blocking allocator. This + * helper always blocks, so a sleepable context alone is not enough: the + * caller must guarantee it is not already inside the page allocator or + * reclaim, where blocking here would reenter mm and deadlock. The only + * user is arena_vm_fault(), a userspace page fault in task context. + */ +struct page *bpf_map_alloc_page_sleepable(const struct bpf_map *map) +{ + might_sleep(); + return alloc_pages_node(map->numa_node, + BPF_PAGE_GFP | __GFP_RETRY_MAYFAIL, 0); +} + static int btf_field_cmp(const void *a, const void *b) { const struct btf_field *f1 = a, *f2 = b; -- 2.43.0
