Re: INFO: rcu detected stall in netlink_sendmsg (4)
syzbot has bisected this issue to:
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes
Date: Sat Sep 29 00:59:43 2018 +
tc: Add support for configuring the taprio scheduler
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=16d46e1b10
start commit: 7cc2a8ea Merge tag 'block-5.8-2020-07-01' of git://git.ker..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11d46e1b10
kernel config: https://syzkaller.appspot.com/x/.config?x=7be693511b29b338
dashboard link: https://syzkaller.appspot.com/bug?extid=0fb70e87d8e0ac278fe9
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1023588f10
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1647a88f10
Reported-by: syzbot+0fb70e87d8e0ac278...@syzkaller.appspotmail.com
Fixes: 5a781ccbd19e ("tc: Add support for configuring the taprio scheduler")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
Re: INFO: rcu detected stall in netlink_sendmsg (4)
syzbot has found a reproducer for the following crash on: HEAD commit:9e50b94b Add linux-next specific files for 20200703 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=13e6ec3310 kernel config: https://syzkaller.appspot.com/x/.config?x=f99cc0faa1476ed6 dashboard link: https://syzkaller.appspot.com/bug?extid=0fb70e87d8e0ac278fe9 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=168ab5d510 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1771c5d510 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+0fb70e87d8e0ac278...@syzkaller.appspotmail.com rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu:0-...0: (3 ticks this GP) idle=ff2/1/0x4000 softirq=8592/8593 fqs=5250 (detected by 1, t=10502 jiffies, g=8273, q=66) Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 6802 Comm: syz-executor688 Not tainted 5.8.0-rc3-next-20200703-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__preempt_count_dec_and_test arch/x86/include/asm/preempt.h:94 [inline] RIP: 0010:rcu_lockdep_current_cpu_online kernel/rcu/tree.c:1144 [inline] RIP: 0010:rcu_lockdep_current_cpu_online+0xc8/0x110 kernel/rcu/tree.c:1131 Code: 59 48 8d 7d 70 48 8b 5b 20 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 41 48 8b 45 70 48 85 c3 0f 95 c0 <65> ff 0d d1 18 a1 7e 74 07 48 83 c4 08 5b 5d c3 e8 52 93 9f ff eb RSP: 0018:c9007db8 EFLAGS: 0002 RAX: 0001 RBX: 0001 RCX: 11303b28 RDX: 11378c1e RSI: 00010204 RDI: 89bc60f0 RBP: 89bc6080 R08: R09: 8aaf028f R10: R11: R12: 0001 R13: 8880ae627840 R14: 888094512340 R15: dc00 FS: 017fe880() GS:8880ae60() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 2610 CR3: 9aba2000 CR4: 001506f0 DR0: DR1: DR2: DR3: DR6: fffe0ff0 DR7: 0400 Call Trace: rcu_read_lock_held_common kernel/rcu/update.c:110 [inline] rcu_read_lock_held_common kernel/rcu/update.c:100 [inline] rcu_read_lock_sched_held+0x25/0xb0 kernel/rcu/update.c:121 trace_hrtimer_expire_exit include/trace/events/timer.h:279 [inline] __run_hrtimer kernel/time/hrtimer.c:1523 [inline] __hrtimer_run_queues+0xd13/0xfc0 kernel/time/hrtimer.c:1584 hrtimer_interrupt+0x32a/0x930 kernel/time/hrtimer.c:1646 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1080 [inline] __sysvec_apic_timer_interrupt+0x142/0x5e0 arch/x86/kernel/apic/apic.c:1097 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline] run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline] sysvec_apic_timer_interrupt+0xe0/0x120 arch/x86/kernel/apic/apic.c:1091 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:596 RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:765 [inline] RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline] RIP: 0010:_raw_spin_unlock_irqrestore+0x8c/0xe0 kernel/locking/spinlock.c:191 Code: 48 c7 c0 00 ff b4 89 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 37 48 83 3d 9b 74 c8 01 00 74 22 48 89 df 57 9d <0f> 1f 44 00 00 bf 01 00 00 00 e8 95 fb 62 f9 65 8b 05 fe 73 15 78 RSP: 0018:c900010872c0 EFLAGS: 0282 RAX: 11369fe0 RBX: 0282 RCX: 0002 RDX: dc00 RSI: RDI: 0282 RBP: 8880945122e8 R08: R09: R10: 0001 R11: R12: 0282 R13: 161f14abb88be58f R14: 888094512000 R15: spin_unlock_irqrestore include/linux/spinlock.h:409 [inline] taprio_change+0x1fdc/0x2960 net/sched/sch_taprio.c:1556 taprio_init+0x52e/0x670 net/sched/sch_taprio.c:1669 qdisc_create+0x4b6/0x12e0 net/sched/sch_api.c:1245 tc_modify_qdisc+0x4c8/0x1990 net/sched/sch_api.c:1661 rtnetlink_rcv_msg+0x44e/0xad0 net/core/rtnetlink.c:5460 netlink_rcv_skb+0x15a/0x430 net/netlink/af_netlink.c:2469 netlink_unicast_kernel net/netlink/af_netlink.c:1303 [inline] netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1329 netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1918 sock_sendmsg_nosec net/socket.c:652 [inline] sock_sendmsg+0xcf/0x120 net/socket.c:672 sys_sendmsg+0x6e8/0x810 net/socket.c:2352 ___sys_sendmsg+0xf3/0x170 net/socket.c:2406 __sys_sendmsg+0xe5/0x1b0 net/socket.c:2439 do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:367 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x443799 Code: Bad RIP value. RSP: 002b:7ffceabd28c8 EFLAGS:
Re: INFO: rcu detected stall in netlink_sendmsg
syzbot has found a reproducer for the following crash on: HEAD commit:8d33316d5205 Merge branch 'x86-urgent-for-linus' of git://.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=14d5f3bcc0 kernel config: https://syzkaller.appspot.com/x/.config?x=ee434566c893c7b1 dashboard link: https://syzkaller.appspot.com/bug?extid=a910a514846e27f15348 compiler: gcc (GCC) 9.0.0 20181231 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13923b60c0 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+a910a514846e27f15...@syzkaller.appspotmail.com kernel msg: ebtables bug: please report to author: Entries_size never zero kernel msg: ebtables bug: please report to author: Entries_size never zero kernel msg: ebtables bug: please report to author: Entries_size never zero kernel msg: ebtables bug: please report to author: Entries_size never zero 32-bit node address hash set to aa1414ac rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 1-: (1 GPs behind) idle=5aa/1/0x4002 softirq=10469/10470 fqs=5225 rcu: (t=10500 jiffies g=6081 q=489) NMI backtrace for cpu 1 CPU: 1 PID: 7809 Comm: syz-executor.3 Not tainted 5.0.0-rc6+ #76 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 nmi_cpu_backtrace.cold+0x63/0xa4 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x1be/0x236 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree.c:1211 print_cpu_stall kernel/rcu/tree.c:1348 [inline] check_cpu_stall kernel/rcu/tree.c:1422 [inline] rcu_pending kernel/rcu/tree.c:3018 [inline] rcu_check_callbacks.cold+0x500/0xa4a kernel/rcu/tree.c:2521 update_process_times+0x32/0x80 kernel/time/timer.c:1635 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:161 tick_sched_timer+0x47/0x130 kernel/time/tick-sched.c:1271 __run_hrtimer kernel/time/hrtimer.c:1389 [inline] __hrtimer_run_queues+0x33e/0xde0 kernel/time/hrtimer.c:1451 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1509 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1035 [inline] smp_apic_timer_interrupt+0x120/0x570 arch/x86/kernel/apic/apic.c:1060 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 RIP: 0010:check_memory_region+0x21/0x190 mm/kasan/generic.c:190 Code: 2e 0f 1f 84 00 00 00 00 00 48 85 f6 0f 84 21 01 00 00 48 b8 ff ff ff ff ff 7f ff ff 55 0f b6 d2 48 39 c7 48 89 e5 41 55 41 54 <53> 0f 86 f6 00 00 00 4c 8d 5c 37 ff 49 89 f8 48 b8 00 00 00 00 00 RSP: 0018:88808b8fea60 EFLAGS: 0212 ORIG_RAX: ff13 RAX: 7fff RBX: e8d2f348 RCX: 8157be27 RDX: RSI: 0004 RDI: e8d2f348 RBP: 88808b8fea70 R08: 1d1a5e69 R09: f91a5e6a R10: f91a5e69 R11: e8d2f34b R12: 0001 R13: 0003 R14: f91a5e69 R15: 05e8 kasan_check_read+0x11/0x20 mm/kasan/common.c:100 atomic_read include/asm-generic/atomic-instrumented.h:21 [inline] virt_spin_lock arch/x86/include/asm/qspinlock.h:83 [inline] native_queued_spin_lock_slowpath+0xb7/0x970 kernel/locking/qspinlock.c:337 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:653 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline] queued_spin_lock include/asm-generic/qspinlock.h:90 [inline] do_raw_spin_lock+0x20e/0x2e0 kernel/locking/spinlock_debug.c:113 __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline] _raw_spin_lock+0x37/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] nf_ct_add_to_unconfirmed_list net/netfilter/nf_conntrack_core.c:462 [inline] init_conntrack.isra.0+0xa15/0x1180 net/netfilter/nf_conntrack_core.c:1437 resolve_normal_ct net/netfilter/nf_conntrack_core.c:1479 [inline] nf_conntrack_in+0xa68/0x1070 net/netfilter/nf_conntrack_core.c:1585 ipv4_conntrack_local+0x169/0x210 net/netfilter/nf_conntrack_proto.c:444 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline] nf_hook_slow+0xbf/0x1f0 net/netfilter/core.c:511 nf_hook include/linux/netfilter.h:244 [inline] __ip_local_out+0x403/0x880 net/ipv4/ip_output.c:113 ip_local_out+0x2d/0x1b0 net/ipv4/ip_output.c:122 iptunnel_xmit+0x58e/0x980 net/ipv4/ip_tunnel_core.c:91 udp_tunnel_xmit_skb+0x236/0x310 net/ipv4/udp_tunnel.c:200 tipc_udp_xmit.isra.0+0x7fd/0xcc0 net/tipc/udp_media.c:181 tipc_udp_send_msg+0x295/0x4a0 net/tipc/udp_media.c:247 tipc_bearer_xmit_skb+0x172/0x360 net/tipc/bearer.c:503 tipc_enable_bearer+0xac4/0xd20 net/tipc/bearer.c:328 __tipc_nl_bearer_enable+0x2d1/0x3b0 net/tipc/bearer.c:899 tipc_nl_bearer_enable+0x23/0x40 net/tipc/bearer.c:907
INFO: rcu detected stall in netlink_sendmsg
Hello, syzbot found the following crash on: HEAD commit:ce28bb445388 Merge git://git.kernel.org/pub/scm/linux/kern.. git tree: net-next console output: https://syzkaller.appspot.com/x/log.txt?x=12662e6740 kernel config: https://syzkaller.appspot.com/x/.config?x=67a2081147a23142 dashboard link: https://syzkaller.appspot.com/bug?extid=a910a514846e27f15348 compiler: gcc (GCC) 8.0.1 20180413 (experimental) Unfortunately, I don't have any reproducer for this crash yet. IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+a910a514846e27f15...@syzkaller.appspotmail.com netlink: 8 bytes leftover after parsing attributes in process `syz-executor5'. rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 1-...!: (10500 ticks this GP) idle=58a/1/0x4002 softirq=84145/84145 fqs=0 rcu: (t=10500 jiffies g=122913 q=1655) rcu: rcu_preempt kthread starved for 10500 jiffies! g122913 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 rcu: RCU grace-period kthread stack dump: rcu_preempt I2256810 2 0x8000 Call Trace: context_switch kernel/sched/core.c:2831 [inline] __schedule+0x86c/0x1ed0 kernel/sched/core.c:3472 schedule+0xfe/0x460 kernel/sched/core.c:3516 schedule_timeout+0x140/0x260 kernel/time/timer.c:1804 rcu_gp_fqs_loop+0x762/0xa80 kernel/rcu/tree.c:1934 rcu_gp_kthread+0x341/0xc70 kernel/rcu/tree.c:2090 kthread+0x35a/0x440 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 NMI backtrace for cpu 1 CPU: 1 PID: 10942 Comm: syz-executor1 Not tainted 4.20.0-rc7+ #358 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1d3/0x2c6 lib/dump_stack.c:113 nmi_cpu_backtrace.cold.4+0x63/0xa2 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x1c2/0x22c lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x16f/0x1bc kernel/rcu/tree.c:1195 print_cpu_stall.cold.65+0x1f3/0x3c6 kernel/rcu/tree.c:1334 check_cpu_stall kernel/rcu/tree.c:1408 [inline] rcu_pending kernel/rcu/tree.c:2961 [inline] rcu_check_callbacks+0xac1/0x1410 kernel/rcu/tree.c:2506 update_process_times+0x2d/0x70 kernel/time/timer.c:1636 tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164 tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274 __run_hrtimer kernel/time/hrtimer.c:1398 [inline] __hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460 hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1034 [inline] smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1059 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 RIP: 0010:__sanitizer_cov_trace_const_cmp4+0x1/0x20 kernel/kcov.c:188 Code: fe ff ff 5d c3 0f 1f 40 00 55 0f b7 d6 0f b7 f7 bf 03 00 00 00 48 89 e5 48 8b 4d 08 e8 88 fe ff ff 5d c3 66 0f 1f 44 00 00 55 <89> f2 89 fe bf 05 00 00 00 48 89 e5 48 8b 4d 08 e8 6a fe ff ff 5d RSP: 0018:8881c195ef50 EFLAGS: 0286 ORIG_RAX: ff13 RAX: 0004 RBX: 0004 RCX: c90005f36000 RDX: 0004 RSI: 0004 RDI: 000e RBP: 8881c195efc0 R08: 8881d7da2040 R09: R10: R11: 8881d7da2040 R12: dc00 R13: 8881c2e07c18 R14: 8881b8142d20 R15: 0100 xfrm_policy_bysel_ctx+0x883/0x1050 net/xfrm/xfrm_policy.c:1664 xfrm_get_policy+0x6a3/0x1140 net/xfrm/xfrm_user.c:1887 xfrm_user_rcv_msg+0x44c/0x8e0 net/xfrm/xfrm_user.c:2663 netlink_rcv_skb+0x16c/0x430 net/netlink/af_netlink.c:2477 xfrm_netlink_rcv+0x6f/0x90 net/xfrm/xfrm_user.c:2671 netlink_unicast_kernel net/netlink/af_netlink.c:1310 [inline] netlink_unicast+0x59f/0x750 net/netlink/af_netlink.c:1336 netlink_sendmsg+0xa18/0xfc0 net/netlink/af_netlink.c:1917 sock_sendmsg_nosec net/socket.c:621 [inline] sock_sendmsg+0xd5/0x120 net/socket.c:631 ___sys_sendmsg+0x7fd/0x930 net/socket.c:2116 __sys_sendmsg+0x11d/0x280 net/socket.c:2154 __do_sys_sendmsg net/socket.c:2163 [inline] __se_sys_sendmsg net/socket.c:2161 [inline] __x64_sys_sendmsg+0x78/0xb0 net/socket.c:2161 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457669 Code: fd b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:7fc47e41ac78 EFLAGS: 0246 ORIG_RAX: 002e RAX: ffda RBX: 0003 RCX: 00457669 RDX: RSI: 2014f000 RDI: 0003 RBP: 0072bf00 R08: R09: R10: R11: 0246 R12:
