Re: WARNING in kvmalloc_node
On Wed, Feb 14, 2018 at 3:55 AM, Matthew Wilcoxwrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: >> Hello, >> >> syzbot hit the following crash on bpf-next commit >> 7928b2cbe55b2a410a0f5c1f154610059c57b1b2 (Sun Feb 11 23:04:29 2018 +) >> Linux 4.16-rc1 >> >> So far this crash happened 236 times on bpf-next. >> C reproducer is attached. >> syzkaller reproducer is attached. >> Raw console output is attached. >> compiler: gcc (GCC) 7.1.1 20170620 >> .config is attached. >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit: >> Reported-by: syzbot+1a240cdb1f4cc8881...@syzkaller.appspotmail.com >> It will help syzbot understand when the bug is fixed. See footer for >> details. >> If you forward the report, please keep this part and the footer. >> >> audit: type=1400 audit(1518457683.474:7): avc: denied { map } for >> pid=4183 comm="syzkaller238030" path="/root/syzkaller238030826" dev="sda1" >> ino=16481 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1 >> WARNING: CPU: 1 PID: 4183 at mm/util.c:403 kvmalloc_node+0xc3/0xd0 >> mm/util.c:403 > > This WARN_ON is telling us that we were called with the wrong GFP flags. > >> audit: type=1400 audit(1518457683.474:8): avc: denied { map_create } for >> pid=4183 comm="syzkaller238030" >> scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=bpf >> permissive=1 >> Kernel panic - not syncing: panic_on_warn set ... >> >> CPU: 1 PID: 4183 Comm: syzkaller238030 Not tainted 4.16.0-rc1+ #12 >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS >> Google 01/01/2011 >> Call Trace: >> __dump_stack lib/dump_stack.c:17 [inline] >> dump_stack+0x194/0x257 lib/dump_stack.c:53 >> audit: type=1400 audit(1518457683.474:9): avc: denied { map_read map_write >> } for pid=4183 comm="syzkaller238030" >> scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=bpf >> permissive=1 >> panic+0x1e4/0x41c kernel/panic.c:183 >> __warn+0x1dc/0x200 kernel/panic.c:547 >> report_bug+0x211/0x2d0 lib/bug.c:184 >> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178 >> fixup_bug arch/x86/kernel/traps.c:247 [inline] >> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296 >> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315 >> invalid_op+0x22/0x40 arch/x86/entry/entry_64.S:988 >> RIP: 0010:kvmalloc_node+0xc3/0xd0 mm/util.c:403 >> RSP: 0018:8801b436f6e8 EFLAGS: 00010293 >> RAX: 8801b1dd25c0 RBX: 01088220 RCX: 81970ca3 >> RDX: RSI: 01088220 RDI: 0070 >> RBP: 8801b436f708 R08: R09: >> R10: R11: R12: 0070 >> R13: R14: R15: 8801d304cd00 >> kvmalloc include/linux/mm.h:541 [inline] >> kvmalloc_array include/linux/mm.h:557 [inline] >> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >> map_update_elem kernel/bpf/syscall.c:698 [inline] > > Blame the BPF people, not the MM people ;-) Done: https://github.com/google/syzkaller/commit/77ed06bf1628ff4554aa800240fbc22bb2a133b7 Now will be attributed to kernel/bpf/cpumap.c and titled "WARNING: kmalloc bug in cpu_map_update_elem". Thanks! >> SYSC_bpf kernel/bpf/syscall.c:1872 [inline] >> SyS_bpf+0x215f/0x4860 kernel/bpf/syscall.c:1843 >> do_syscall_64+0x282/0x940 arch/x86/entry/common.c:287 >> entry_SYSCALL_64_after_hwframe+0x26/0x9b >> RIP: 0033:0x43fda9 >> RSP: 002b:7ffe6b075798 EFLAGS: 0203 ORIG_RAX: 0141 >> RAX: ffda RBX: RCX: 0043fda9 >> RDX: 0020 RSI: 20ef4fe0 RDI: 0002 >> RBP: 006ca018 R08: R09: >> R10: R11: 0203 R12: 004016d0 >> R13: 00401760 R14: R15: >> Dumping ftrace buffer: >>(ftrace buffer empty) >> Kernel Offset: disabled >> Rebooting in 86400 seconds.. >> >> >> --- >> This bug is generated by a dumb bot. It may contain errors. >> See https://goo.gl/tpsmEJ for details. >> Direct all questions to syzkal...@googlegroups.com. >> >> syzbot will keep track of this bug report. >> If you forgot to add the Reported-by tag, once the fix for this bug is >> merged >> into any tree, please reply to this email with: >> #syz fix: exact-commit-title >> If you want to test a patch for this bug, please reply with: >> #syz test: git://repo/address.git branch >> and provide the patch inline or as an attachment. >> To mark this as a duplicate of another syzbot report, please
Re: WARNING in kvmalloc_node
On Wed, Feb 14, 2018 at 3:55 AM, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: >> Hello, >> >> syzbot hit the following crash on bpf-next commit >> 7928b2cbe55b2a410a0f5c1f154610059c57b1b2 (Sun Feb 11 23:04:29 2018 +) >> Linux 4.16-rc1 >> >> So far this crash happened 236 times on bpf-next. >> C reproducer is attached. >> syzkaller reproducer is attached. >> Raw console output is attached. >> compiler: gcc (GCC) 7.1.1 20170620 >> .config is attached. >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit: >> Reported-by: syzbot+1a240cdb1f4cc8881...@syzkaller.appspotmail.com >> It will help syzbot understand when the bug is fixed. See footer for >> details. >> If you forward the report, please keep this part and the footer. >> >> audit: type=1400 audit(1518457683.474:7): avc: denied { map } for >> pid=4183 comm="syzkaller238030" path="/root/syzkaller238030826" dev="sda1" >> ino=16481 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1 >> WARNING: CPU: 1 PID: 4183 at mm/util.c:403 kvmalloc_node+0xc3/0xd0 >> mm/util.c:403 > > This WARN_ON is telling us that we were called with the wrong GFP flags. > >> audit: type=1400 audit(1518457683.474:8): avc: denied { map_create } for >> pid=4183 comm="syzkaller238030" >> scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=bpf >> permissive=1 >> Kernel panic - not syncing: panic_on_warn set ... >> >> CPU: 1 PID: 4183 Comm: syzkaller238030 Not tainted 4.16.0-rc1+ #12 >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS >> Google 01/01/2011 >> Call Trace: >> __dump_stack lib/dump_stack.c:17 [inline] >> dump_stack+0x194/0x257 lib/dump_stack.c:53 >> audit: type=1400 audit(1518457683.474:9): avc: denied { map_read map_write >> } for pid=4183 comm="syzkaller238030" >> scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 >> tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=bpf >> permissive=1 >> panic+0x1e4/0x41c kernel/panic.c:183 >> __warn+0x1dc/0x200 kernel/panic.c:547 >> report_bug+0x211/0x2d0 lib/bug.c:184 >> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178 >> fixup_bug arch/x86/kernel/traps.c:247 [inline] >> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296 >> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315 >> invalid_op+0x22/0x40 arch/x86/entry/entry_64.S:988 >> RIP: 0010:kvmalloc_node+0xc3/0xd0 mm/util.c:403 >> RSP: 0018:8801b436f6e8 EFLAGS: 00010293 >> RAX: 8801b1dd25c0 RBX: 01088220 RCX: 81970ca3 >> RDX: RSI: 01088220 RDI: 0070 >> RBP: 8801b436f708 R08: R09: >> R10: R11: R12: 0070 >> R13: R14: R15: 8801d304cd00 >> kvmalloc include/linux/mm.h:541 [inline] >> kvmalloc_array include/linux/mm.h:557 [inline] >> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >> map_update_elem kernel/bpf/syscall.c:698 [inline] > > Blame the BPF people, not the MM people ;-) Done: https://github.com/google/syzkaller/commit/77ed06bf1628ff4554aa800240fbc22bb2a133b7 Now will be attributed to kernel/bpf/cpumap.c and titled "WARNING: kmalloc bug in cpu_map_update_elem". Thanks! >> SYSC_bpf kernel/bpf/syscall.c:1872 [inline] >> SyS_bpf+0x215f/0x4860 kernel/bpf/syscall.c:1843 >> do_syscall_64+0x282/0x940 arch/x86/entry/common.c:287 >> entry_SYSCALL_64_after_hwframe+0x26/0x9b >> RIP: 0033:0x43fda9 >> RSP: 002b:7ffe6b075798 EFLAGS: 0203 ORIG_RAX: 0141 >> RAX: ffda RBX: RCX: 0043fda9 >> RDX: 0020 RSI: 20ef4fe0 RDI: 0002 >> RBP: 006ca018 R08: R09: >> R10: R11: 0203 R12: 004016d0 >> R13: 00401760 R14: R15: >> Dumping ftrace buffer: >>(ftrace buffer empty) >> Kernel Offset: disabled >> Rebooting in 86400 seconds.. >> >> >> --- >> This bug is generated by a dumb bot. It may contain errors. >> See https://goo.gl/tpsmEJ for details. >> Direct all questions to syzkal...@googlegroups.com. >> >> syzbot will keep track of this bug report. >> If you forgot to add the Reported-by tag, once the fix for this bug is >> merged >> into any tree, please reply to this email with: >> #syz fix: exact-commit-title >> If you want to test a patch for this bug, please reply with: >> #syz test: git://repo/address.git branch >> and provide the patch inline or as an attachment. >> To mark this as a duplicate of another syzbot report, please reply with: >> #syz
Re: WARNING in kvmalloc_node
On 02/14/2018 01:47 PM, Jason Wang wrote: > On 2018年02月14日 20:29, Jesper Dangaard Brouer wrote: >> On Wed, 14 Feb 2018 13:17:18 +0100 >> Daniel Borkmannwrote: >>> On 02/14/2018 01:02 PM, Jason Wang wrote: On 2018年02月14日 19:51, Michal Hocko wrote: > On Wed 14-02-18 19:47:30, Jason Wang wrote: >> On 2018年02月14日 17:28, Daniel Borkmann wrote: >>> [ +Jason, +Jesper ] >>> >>> On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] >> kvmalloc include/linux/mm.h:541 [inline] >> kvmalloc_array include/linux/mm.h:557 [inline] >> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >> map_update_elem kernel/bpf/syscall.c:698 [inline] > Blame the BPF people, not the MM people ;-) >>> Heh, not really. ;-) >>> Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. >>> Agree, that doesn't work. >>> >>> Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when >>> kmalloc() fails"). >>> >>> Jason, please take a look at fixing this, thanks! >> It looks to me the only solution is to revert that commit. > Do you really need this to be GFP_ATOMIC? I can see some callers are > under RCU read lock but can we perhaps do the allocation outside of this > section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). >>> That would be one option indeed (probably useful in any case to make the API >>> more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking >>> at >>> it, update can neither be called out of a BPF prog since prevented by >>> verifier >>> nor under RCU reader side when updating this type of map from syscall path. >>> Jesper, any concrete reason we still need GFP_ATOMIC here? >> Allocations in cpumap (related to ptr_ring) should only be possible to >> be initiated through userspace via bpf-syscall. > > I see verifier guarantees this. > >> Thus, there isn't any >> reason for GFP_ATOMIC here. > > Want me to send a patch to remove GFP_ATOMIC here? Sounds good, thanks Jason!
Re: WARNING in kvmalloc_node
On 02/14/2018 01:47 PM, Jason Wang wrote: > On 2018年02月14日 20:29, Jesper Dangaard Brouer wrote: >> On Wed, 14 Feb 2018 13:17:18 +0100 >> Daniel Borkmann wrote: >>> On 02/14/2018 01:02 PM, Jason Wang wrote: On 2018年02月14日 19:51, Michal Hocko wrote: > On Wed 14-02-18 19:47:30, Jason Wang wrote: >> On 2018年02月14日 17:28, Daniel Borkmann wrote: >>> [ +Jason, +Jesper ] >>> >>> On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] >> kvmalloc include/linux/mm.h:541 [inline] >> kvmalloc_array include/linux/mm.h:557 [inline] >> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >> map_update_elem kernel/bpf/syscall.c:698 [inline] > Blame the BPF people, not the MM people ;-) >>> Heh, not really. ;-) >>> Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. >>> Agree, that doesn't work. >>> >>> Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when >>> kmalloc() fails"). >>> >>> Jason, please take a look at fixing this, thanks! >> It looks to me the only solution is to revert that commit. > Do you really need this to be GFP_ATOMIC? I can see some callers are > under RCU read lock but can we perhaps do the allocation outside of this > section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). >>> That would be one option indeed (probably useful in any case to make the API >>> more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking >>> at >>> it, update can neither be called out of a BPF prog since prevented by >>> verifier >>> nor under RCU reader side when updating this type of map from syscall path. >>> Jesper, any concrete reason we still need GFP_ATOMIC here? >> Allocations in cpumap (related to ptr_ring) should only be possible to >> be initiated through userspace via bpf-syscall. > > I see verifier guarantees this. > >> Thus, there isn't any >> reason for GFP_ATOMIC here. > > Want me to send a patch to remove GFP_ATOMIC here? Sounds good, thanks Jason!
Re: WARNING in kvmalloc_node
On 2018年02月14日 20:29, Jesper Dangaard Brouer wrote: On Wed, 14 Feb 2018 13:17:18 +0100 Daniel Borkmannwrote: On 02/14/2018 01:02 PM, Jason Wang wrote: On 2018年02月14日 19:51, Michal Hocko wrote: On Wed 14-02-18 19:47:30, Jason Wang wrote: On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). That would be one option indeed (probably useful in any case to make the API more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at it, update can neither be called out of a BPF prog since prevented by verifier nor under RCU reader side when updating this type of map from syscall path. Jesper, any concrete reason we still need GFP_ATOMIC here? Allocations in cpumap (related to ptr_ring) should only be possible to be initiated through userspace via bpf-syscall. I see verifier guarantees this. Thus, there isn't any reason for GFP_ATOMIC here. Want me to send a patch to remove GFP_ATOMIC here? Thanks
Re: WARNING in kvmalloc_node
On 2018年02月14日 20:29, Jesper Dangaard Brouer wrote: On Wed, 14 Feb 2018 13:17:18 +0100 Daniel Borkmann wrote: On 02/14/2018 01:02 PM, Jason Wang wrote: On 2018年02月14日 19:51, Michal Hocko wrote: On Wed 14-02-18 19:47:30, Jason Wang wrote: On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). That would be one option indeed (probably useful in any case to make the API more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at it, update can neither be called out of a BPF prog since prevented by verifier nor under RCU reader side when updating this type of map from syscall path. Jesper, any concrete reason we still need GFP_ATOMIC here? Allocations in cpumap (related to ptr_ring) should only be possible to be initiated through userspace via bpf-syscall. I see verifier guarantees this. Thus, there isn't any reason for GFP_ATOMIC here. Want me to send a patch to remove GFP_ATOMIC here? Thanks
Re: WARNING in kvmalloc_node
On Wed, 14 Feb 2018 13:17:18 +0100 Daniel Borkmannwrote: > On 02/14/2018 01:02 PM, Jason Wang wrote: > > On 2018年02月14日 19:51, Michal Hocko wrote: > >> On Wed 14-02-18 19:47:30, Jason Wang wrote: > >>> On 2018年02月14日 17:28, Daniel Borkmann wrote: > [ +Jason, +Jesper ] > > On 02/14/2018 09:43 AM, Michal Hocko wrote: > > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > > [...] > >>> kvmalloc include/linux/mm.h:541 [inline] > >>> kvmalloc_array include/linux/mm.h:557 [inline] > >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] > >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > >>> map_update_elem kernel/bpf/syscall.c:698 [inline] > >> Blame the BPF people, not the MM people ;-) > Heh, not really. ;-) > > > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. > Agree, that doesn't work. > > Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when > kmalloc() fails"). > > Jason, please take a look at fixing this, thanks! > >>> It looks to me the only solution is to revert that commit. > >> Do you really need this to be GFP_ATOMIC? I can see some callers are > >> under RCU read lock but can we perhaps do the allocation outside of this > >> section? > > > > If I understand the code correctly, the code would be called by XDP program > > (usually run inside a bh) which makes it hard to do this. > > > > Rethink of this, we can probably test gfp and not call kvmalloc if > > GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). > > That would be one option indeed (probably useful in any case to make the API > more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at > it, update can neither be called out of a BPF prog since prevented by verifier > nor under RCU reader side when updating this type of map from syscall path. > Jesper, any concrete reason we still need GFP_ATOMIC here? Allocations in cpumap (related to ptr_ring) should only be possible to be initiated through userspace via bpf-syscall. Thus, there isn't any reason for GFP_ATOMIC here. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: WARNING in kvmalloc_node
On Wed, 14 Feb 2018 13:17:18 +0100 Daniel Borkmann wrote: > On 02/14/2018 01:02 PM, Jason Wang wrote: > > On 2018年02月14日 19:51, Michal Hocko wrote: > >> On Wed 14-02-18 19:47:30, Jason Wang wrote: > >>> On 2018年02月14日 17:28, Daniel Borkmann wrote: > [ +Jason, +Jesper ] > > On 02/14/2018 09:43 AM, Michal Hocko wrote: > > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > > [...] > >>> kvmalloc include/linux/mm.h:541 [inline] > >>> kvmalloc_array include/linux/mm.h:557 [inline] > >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] > >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > >>> map_update_elem kernel/bpf/syscall.c:698 [inline] > >> Blame the BPF people, not the MM people ;-) > Heh, not really. ;-) > > > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. > Agree, that doesn't work. > > Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when > kmalloc() fails"). > > Jason, please take a look at fixing this, thanks! > >>> It looks to me the only solution is to revert that commit. > >> Do you really need this to be GFP_ATOMIC? I can see some callers are > >> under RCU read lock but can we perhaps do the allocation outside of this > >> section? > > > > If I understand the code correctly, the code would be called by XDP program > > (usually run inside a bh) which makes it hard to do this. > > > > Rethink of this, we can probably test gfp and not call kvmalloc if > > GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). > > That would be one option indeed (probably useful in any case to make the API > more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at > it, update can neither be called out of a BPF prog since prevented by verifier > nor under RCU reader side when updating this type of map from syscall path. > Jesper, any concrete reason we still need GFP_ATOMIC here? Allocations in cpumap (related to ptr_ring) should only be possible to be initiated through userspace via bpf-syscall. Thus, there isn't any reason for GFP_ATOMIC here. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: WARNING in kvmalloc_node
On 02/14/2018 01:02 PM, Jason Wang wrote: > On 2018年02月14日 19:51, Michal Hocko wrote: >> On Wed 14-02-18 19:47:30, Jason Wang wrote: >>> On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > [...] >>> kvmalloc include/linux/mm.h:541 [inline] >>> kvmalloc_array include/linux/mm.h:557 [inline] >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >>> map_update_elem kernel/bpf/syscall.c:698 [inline] >> Blame the BPF people, not the MM people ;-) Heh, not really. ;-) > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! >>> It looks to me the only solution is to revert that commit. >> Do you really need this to be GFP_ATOMIC? I can see some callers are >> under RCU read lock but can we perhaps do the allocation outside of this >> section? > > If I understand the code correctly, the code would be called by XDP program > (usually run inside a bh) which makes it hard to do this. > > Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC > is set in __ptr_ring_init_queue_alloc(). That would be one option indeed (probably useful in any case to make the API more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at it, update can neither be called out of a BPF prog since prevented by verifier nor under RCU reader side when updating this type of map from syscall path. Jesper, any concrete reason we still need GFP_ATOMIC here?
Re: WARNING in kvmalloc_node
On 02/14/2018 01:02 PM, Jason Wang wrote: > On 2018年02月14日 19:51, Michal Hocko wrote: >> On Wed 14-02-18 19:47:30, Jason Wang wrote: >>> On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > [...] >>> kvmalloc include/linux/mm.h:541 [inline] >>> kvmalloc_array include/linux/mm.h:557 [inline] >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >>> map_update_elem kernel/bpf/syscall.c:698 [inline] >> Blame the BPF people, not the MM people ;-) Heh, not really. ;-) > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! >>> It looks to me the only solution is to revert that commit. >> Do you really need this to be GFP_ATOMIC? I can see some callers are >> under RCU read lock but can we perhaps do the allocation outside of this >> section? > > If I understand the code correctly, the code would be called by XDP program > (usually run inside a bh) which makes it hard to do this. > > Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC > is set in __ptr_ring_init_queue_alloc(). That would be one option indeed (probably useful in any case to make the API more robust). Another one is to just not use GFP_ATOMIC in cpumap. Looking at it, update can neither be called out of a BPF prog since prevented by verifier nor under RCU reader side when updating this type of map from syscall path. Jesper, any concrete reason we still need GFP_ATOMIC here?
Re: WARNING in kvmalloc_node
On 2018年02月14日 19:51, Michal Hocko wrote: On Wed 14-02-18 19:47:30, Jason Wang wrote: On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). Thanks
Re: WARNING in kvmalloc_node
On 2018年02月14日 19:51, Michal Hocko wrote: On Wed 14-02-18 19:47:30, Jason Wang wrote: On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? If I understand the code correctly, the code would be called by XDP program (usually run inside a bh) which makes it hard to do this. Rethink of this, we can probably test gfp and not call kvmalloc if GFP_ATOMIC is set in __ptr_ring_init_queue_alloc(). Thanks
Re: WARNING in kvmalloc_node
On Wed 14-02-18 19:47:30, Jason Wang wrote: > > > On 2018年02月14日 17:28, Daniel Borkmann wrote: > > [ +Jason, +Jesper ] > > > > On 02/14/2018 09:43 AM, Michal Hocko wrote: > > > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > > > > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > > > [...] > > > > > kvmalloc include/linux/mm.h:541 [inline] > > > > > kvmalloc_array include/linux/mm.h:557 [inline] > > > > > __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > > > > > ptr_ring_init include/linux/ptr_ring.h:492 [inline] > > > > > __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > > > > > cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > > > > > map_update_elem kernel/bpf/syscall.c:698 [inline] > > > > Blame the BPF people, not the MM people ;-) > > Heh, not really. ;-) > > > > > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. > > Agree, that doesn't work. > > > > Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when > > kmalloc() fails"). > > > > Jason, please take a look at fixing this, thanks! > > It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? -- Michal Hocko SUSE Labs
Re: WARNING in kvmalloc_node
On Wed 14-02-18 19:47:30, Jason Wang wrote: > > > On 2018年02月14日 17:28, Daniel Borkmann wrote: > > [ +Jason, +Jesper ] > > > > On 02/14/2018 09:43 AM, Michal Hocko wrote: > > > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > > > > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > > > [...] > > > > > kvmalloc include/linux/mm.h:541 [inline] > > > > > kvmalloc_array include/linux/mm.h:557 [inline] > > > > > __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > > > > > ptr_ring_init include/linux/ptr_ring.h:492 [inline] > > > > > __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > > > > > cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > > > > > map_update_elem kernel/bpf/syscall.c:698 [inline] > > > > Blame the BPF people, not the MM people ;-) > > Heh, not really. ;-) > > > > > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. > > Agree, that doesn't work. > > > > Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when > > kmalloc() fails"). > > > > Jason, please take a look at fixing this, thanks! > > It looks to me the only solution is to revert that commit. Do you really need this to be GFP_ATOMIC? I can see some callers are under RCU read lock but can we perhaps do the allocation outside of this section? -- Michal Hocko SUSE Labs
Re: WARNING in kvmalloc_node
On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Will post a patch. Thanks
Re: WARNING in kvmalloc_node
On 2018年02月14日 17:28, Daniel Borkmann wrote: [ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] kvmalloc include/linux/mm.h:541 [inline] kvmalloc_array include/linux/mm.h:557 [inline] __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] ptr_ring_init include/linux/ptr_ring.h:492 [inline] __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 map_update_elem kernel/bpf/syscall.c:698 [inline] Blame the BPF people, not the MM people ;-) Heh, not really. ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks! It looks to me the only solution is to revert that commit. Will post a patch. Thanks
Re: WARNING in kvmalloc_node
[ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > [...] >>> kvmalloc include/linux/mm.h:541 [inline] >>> kvmalloc_array include/linux/mm.h:557 [inline] >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >>> map_update_elem kernel/bpf/syscall.c:698 [inline] >> >> Blame the BPF people, not the MM people ;-) Heh, not really. ;-) > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks!
Re: WARNING in kvmalloc_node
[ +Jason, +Jesper ] On 02/14/2018 09:43 AM, Michal Hocko wrote: > On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: >> On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: > [...] >>> kvmalloc include/linux/mm.h:541 [inline] >>> kvmalloc_array include/linux/mm.h:557 [inline] >>> __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] >>> ptr_ring_init include/linux/ptr_ring.h:492 [inline] >>> __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] >>> cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 >>> map_update_elem kernel/bpf/syscall.c:698 [inline] >> >> Blame the BPF people, not the MM people ;-) Heh, not really. ;-) > Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. Agree, that doesn't work. Bug was added in commit 0bf7800f1799 ("ptr_ring: try vmalloc() when kmalloc() fails"). Jason, please take a look at fixing this, thanks!
Re: WARNING in kvmalloc_node
On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] > > kvmalloc include/linux/mm.h:541 [inline] > > kvmalloc_array include/linux/mm.h:557 [inline] > > __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > > ptr_ring_init include/linux/ptr_ring.h:492 [inline] > > __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > > cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > > map_update_elem kernel/bpf/syscall.c:698 [inline] > > Blame the BPF people, not the MM people ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. -- Michal Hocko SUSE Labs
Re: WARNING in kvmalloc_node
On Tue 13-02-18 18:55:33, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 03:59:01PM -0800, syzbot wrote: [...] > > kvmalloc include/linux/mm.h:541 [inline] > > kvmalloc_array include/linux/mm.h:557 [inline] > > __ptr_ring_init_queue_alloc include/linux/ptr_ring.h:474 [inline] > > ptr_ring_init include/linux/ptr_ring.h:492 [inline] > > __cpu_map_entry_alloc kernel/bpf/cpumap.c:359 [inline] > > cpu_map_update_elem+0x3c3/0x8e0 kernel/bpf/cpumap.c:490 > > map_update_elem kernel/bpf/syscall.c:698 [inline] > > Blame the BPF people, not the MM people ;-) Yes. kvmalloc (the vmalloc part) doesn't support GFP_ATOMIC semantic. -- Michal Hocko SUSE Labs