Re: WARNING in hrtimer_forward
On Mon, Sep 28, 2020 at 8:36 PM Thomas Gleixner wrote: > > On Tue, Sep 29 2020 at 01:11, Hillf Danton wrote: > > On Mon, 28 Sep 2020 18:13:42 +0200 Thomas Gleixner wrote: > >> So the timer was armed at some point and then the expiry which does the > >> forward races with the ioctl which starts the timer. Lack of > >> serialization or such ... > > > > To make syzbot happy, s/hrtimer_is_queued/hrtimer_active/ can close > > that race but this warning looks benign. > > Why only make sysbot happy? It's clearly an issue and the warning is not > benign simply because forwarding a queued timer is an absolute NONO. > timers (both timer_list and hrtimer) need external synchronization. Oh, Thomas, it's so nice to hear this interpretation of things among all the cases where people only fixing tools and making them happy :) Don't make my tools happy! They don't need that! :)
Re: WARNING in hrtimer_forward
On Tue, Sep 29 2020 at 01:11, Hillf Danton wrote:
> On Mon, 28 Sep 2020 18:13:42 +0200 Thomas Gleixner wrote:
>> So the timer was armed at some point and then the expiry which does the
>> forward races with the ioctl which starts the timer. Lack of
>> serialization or such ...
>
> To make syzbot happy, s/hrtimer_is_queued/hrtimer_active/ can close
> that race but this warning looks benign.
Why only make sysbot happy? It's clearly an issue and the warning is not
benign simply because forwarding a queued timer is an absolute NONO.
timers (both timer_list and hrtimer) need external synchronization.
> --- a/drivers/net/wireless/mac80211_hwsim.c
> +++ b/drivers/net/wireless/mac80211_hwsim.c
> @@ -1698,7 +1698,7 @@ static int mac80211_hwsim_config(struct
>
> if (!data->started || !data->beacon_int)
> hrtimer_cancel(>beacon_timer);
> - else if (!hrtimer_is_queued(>beacon_timer)) {
> + else if (!hrtimer_active(>beacon_timer)) {
> u64 tsf = mac80211_hwsim_get_tsf(hw, NULL);
> u32 bcn_int = data->beacon_int;
> u64 until_tbtt = bcn_int - do_div(tsf, bcn_int);
> @@ -1768,7 +1768,7 @@ static void mac80211_hwsim_bss_info_chan
> info->enable_beacon, info->beacon_int);
> vp->bcn_en = info->enable_beacon;
> if (data->started &&
> - !hrtimer_is_queued(>beacon_timer) &&
> + !hrtimer_active(>beacon_timer) &&
> info->enable_beacon) {
> u64 tsf, until_tbtt;
> u32 bcn_int;
Looks about right.
Thanks,
tglx
Re: WARNING in hrtimer_forward
On Sun, Sep 27 2020 at 07:29, syzbot wrote:
> syzbot has bisected this issue to:
>
> commit 0e7bbcc104baaade4f64205e9706b7d43c46db7d
> Author: Julian Anastasov
> Date: Wed Jul 27 06:56:50 2016 +
>
> neigh: allow admin to set NUD_STALE
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1661d18790
> start commit: ba5f4cfe bpf: Add comment to document BTF type PTR_TO_BTF_..
> git tree: bpf-next
> final oops: https://syzkaller.appspot.com/x/report.txt?x=1561d18790
> console output: https://syzkaller.appspot.com/x/log.txt?x=1161d18790
> kernel config: https://syzkaller.appspot.com/x/.config?x=d44e1360b76d34dc
> dashboard link: https://syzkaller.appspot.com/bug?extid=ca740b95a16399ceb9a5
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1148fe4b90
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12f5218d90
>
> Reported-by: syzbot+ca740b95a16399ceb...@syzkaller.appspotmail.com
> Fixes: 0e7bbcc104ba ("neigh: allow admin to set NUD_STALE")
That bisect does not make any sense and reverting the commit on top of
next does not help either.
What happens is:
fail-16132 [029] 933.714866: sys_enter: NR 16 (3, 8b28,
2000, 0, 0, 0)
-0 [001] d.s2 933.715768: hrtimer_cancel:
hrtimer=fe9fe1b9
-0 [001] ..s1 933.715771: hrtimer_expire_entry:
hrtimer=fe9fe1b9 function=mac80211_hwsim_beacon now=933716506319
fail-16132 [029] d..1 933.715794: hrtimer_start:
hrtimer=fe9fe1b9 function=mac80211_hwsim_beacon expires=933818720770
softexpires=933818720770 mode=REL|SOFT
-0 [001] ..s1 933.715812: hrtimer_forward:
hrtimer=fe9fe1b9
So the timer was armed at some point and then the expiry which does the
forward races with the ioctl which starts the timer. Lack of
serialization or such ...
Thanks,
tglx
Re: WARNING in hrtimer_forward
On Sun, Sep 27 2020 at 16:04, Hillf Danton wrote: > Sat, 26 Sep 2020 17:38:16 -0700 > > Dunno if it's down to memory barrier. > > --- a/kernel/time/hrtimer.c > +++ b/kernel/time/hrtimer.c > @@ -929,7 +929,7 @@ u64 hrtimer_forward(struct hrtimer *time > if (delta < 0) > return 0; > > - if (WARN_ON(timer->state & HRTIMER_STATE_ENQUEUED)) > + if (WARN_ON(hrtimer_is_queued(timer))) > return 0; The point of that exercise is?
Re: WARNING in hrtimer_forward
syzbot has bisected this issue to:
commit 0e7bbcc104baaade4f64205e9706b7d43c46db7d
Author: Julian Anastasov
Date: Wed Jul 27 06:56:50 2016 +
neigh: allow admin to set NUD_STALE
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1661d18790
start commit: ba5f4cfe bpf: Add comment to document BTF type PTR_TO_BTF_..
git tree: bpf-next
final oops: https://syzkaller.appspot.com/x/report.txt?x=1561d18790
console output: https://syzkaller.appspot.com/x/log.txt?x=1161d18790
kernel config: https://syzkaller.appspot.com/x/.config?x=d44e1360b76d34dc
dashboard link: https://syzkaller.appspot.com/bug?extid=ca740b95a16399ceb9a5
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1148fe4b90
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12f5218d90
Reported-by: syzbot+ca740b95a16399ceb...@syzkaller.appspotmail.com
Fixes: 0e7bbcc104ba ("neigh: allow admin to set NUD_STALE")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
Re: WARNING in hrtimer_forward
syzbot has found a reproducer for the following issue on: HEAD commit:ba5f4cfe bpf: Add comment to document BTF type PTR_TO_BTF_.. git tree: bpf-next console output: https://syzkaller.appspot.com/x/log.txt?x=13f316e590 kernel config: https://syzkaller.appspot.com/x/.config?x=d44e1360b76d34dc dashboard link: https://syzkaller.appspot.com/bug?extid=ca740b95a16399ceb9a5 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1148fe4b90 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12f5218d90 IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+ca740b95a16399ceb...@syzkaller.appspotmail.com [ cut here ] WARNING: CPU: 0 PID: 6901 at kernel/time/hrtimer.c:932 hrtimer_forward+0x1e3/0x260 kernel/time/hrtimer.c:932 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 6901 Comm: kworker/u4:1 Not tainted 5.9.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: phy4 ieee80211_iface_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x198/0x1fd lib/dump_stack.c:118 panic+0x382/0x7fb kernel/panic.c:231 __warn.cold+0x20/0x4b kernel/panic.c:600 report_bug+0x1bd/0x210 lib/bug.c:198 handle_bug+0x38/0x90 arch/x86/kernel/traps.c:234 exc_invalid_op+0x14/0x40 arch/x86/kernel/traps.c:254 asm_exc_invalid_op+0x12/0x20 arch/x86/include/asm/idtentry.h:536 RIP: 0010:hrtimer_forward+0x1e3/0x260 kernel/time/hrtimer.c:932 Code: e5 4d 0f 4e ec e8 ad 24 10 00 4c 89 6b 20 e8 a4 24 10 00 4c 89 f0 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 8d 24 10 00 <0f> 0b 45 31 f6 eb dd e8 81 24 10 00 4c 89 e0 48 8b 3c 24 48 99 48 RSP: 0018:c9007d90 EFLAGS: 00010246 RAX: RBX: 88808ded4b78 RCX: 81666168 RDX: 8880942f0200 RSI: 816662b3 RDI: 0001 RBP: 061a8000 R08: 0001 R09: 8880942f0b00 R10: R11: R12: R13: 00a6d77ff62e R14: 0001 R15: dc00 mac80211_hwsim_beacon+0x159/0x1a0 drivers/net/wireless/mac80211_hwsim.c:1726 __run_hrtimer kernel/time/hrtimer.c:1524 [inline] __hrtimer_run_queues+0x6a9/0xfc0 kernel/time/hrtimer.c:1588 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1605 __do_softirq+0x1f8/0xb23 kernel/softirq.c:298 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline] run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline] do_softirq_own_stack+0x9d/0xd0 arch/x86/kernel/irq_64.c:77 invoke_softirq kernel/softirq.c:393 [inline] __irq_exit_rcu kernel/softirq.c:423 [inline] irq_exit_rcu+0x235/0x280 kernel/softirq.c:435 sysvec_apic_timer_interrupt+0x51/0xf0 arch/x86/kernel/apic/apic.c:1091 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581 RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:770 [inline] RIP: 0010:lock_acquire+0x27b/0xaf0 kernel/locking/lockdep.c:5032 Code: ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 0f 85 1d 07 00 00 48 83 3d d8 41 a0 08 00 0f 84 73 05 00 00 4c 89 ff 57 9d <0f> 1f 44 00 00 48 b8 00 00 00 00 00 fc ff df 48 03 44 24 08 48 c7 RSP: 0018:c9e37c18 EFLAGS: 0286 RAX: 113f8d7d RBX: 8880942f0200 RCX: 0001 RDX: dc00 RSI: 0008 RDI: 0286 RBP: c9e37da8 R08: R09: 8d108aa7 R10: fbfff1a21154 R11: R12: R13: R14: R15: 0286 process_one_work+0x8bb/0x1670 kernel/workqueue.c:2245 worker_thread+0x64c/0x1120 kernel/workqueue.c:2415 kthread+0x3b5/0x4a0 kernel/kthread.c:292 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294 Kernel Offset: disabled Rebooting in 86400 seconds..
WARNING in hrtimer_forward
Hello, syzbot found the following issue on: HEAD commit:12450081 libbpf: Fix native endian assumption when parsing.. git tree: bpf console output: https://syzkaller.appspot.com/x/log.txt?x=10bf85c590 kernel config: https://syzkaller.appspot.com/x/.config?x=5ac0d21536db480b dashboard link: https://syzkaller.appspot.com/bug?extid=ca740b95a16399ceb9a5 compiler: gcc (GCC) 10.1.0-syz 20200507 Unfortunately, I don't have any reproducer for this issue yet. IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+ca740b95a16399ceb...@syzkaller.appspotmail.com [ cut here ] WARNING: CPU: 1 PID: 9082 at kernel/time/hrtimer.c:932 hrtimer_forward+0x1e3/0x260 kernel/time/hrtimer.c:932 Kernel panic - not syncing: panic_on_warn set ... CPU: 1 PID: 9082 Comm: syz-executor.4 Not tainted 5.9.0-rc3-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x198/0x1fd lib/dump_stack.c:118 panic+0x382/0x7fb kernel/panic.c:231 __warn.cold+0x20/0x4b kernel/panic.c:600 report_bug+0x1bd/0x210 lib/bug.c:198 handle_bug+0x38/0x90 arch/x86/kernel/traps.c:234 exc_invalid_op+0x14/0x40 arch/x86/kernel/traps.c:254 asm_exc_invalid_op+0x12/0x20 arch/x86/include/asm/idtentry.h:536 RIP: 0010:hrtimer_forward+0x1e3/0x260 kernel/time/hrtimer.c:932 Code: e5 4d 0f 4e ec e8 1d 25 10 00 4c 89 6b 20 e8 14 25 10 00 4c 89 f0 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 fd 24 10 00 <0f> 0b 45 31 f6 eb dd e8 f1 24 10 00 4c 89 e0 48 8b 3c 24 48 99 48 RSP: 0018:c9da8d90 EFLAGS: 00010246 RAX: RBX: 88805cb74b78 RCX: 816606b8 RDX: 88805a4aa380 RSI: 81660803 RDI: 0001 RBP: 061a8000 R08: 0001 R09: 88805a4aac60 R10: R11: R12: R13: 002b545ee359 R14: 0001 R15: dc00 mac80211_hwsim_beacon+0x159/0x1a0 drivers/net/wireless/mac80211_hwsim.c:1726 __run_hrtimer kernel/time/hrtimer.c:1524 [inline] __hrtimer_run_queues+0x6a9/0xfc0 kernel/time/hrtimer.c:1588 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1605 __do_softirq+0x1f8/0xb23 kernel/softirq.c:298 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline] run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline] do_softirq_own_stack+0x9d/0xd0 arch/x86/kernel/irq_64.c:77 invoke_softirq kernel/softirq.c:393 [inline] __irq_exit_rcu kernel/softirq.c:423 [inline] irq_exit_rcu+0x235/0x280 kernel/softirq.c:435 sysvec_apic_timer_interrupt+0x51/0xf0 arch/x86/kernel/apic/apic.c:1091 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581 RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline] RIP: 0010:_raw_spin_unlock_irq+0x4b/0x80 kernel/locking/spinlock.c:199 Code: c0 b8 6b fc 89 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 31 48 83 3d c6 df d5 01 00 74 25 fb 66 0f 1f 44 00 00 01 00 00 00 e8 ab 59 2a f9 65 8b 05 74 72 db 77 85 c0 74 02 5d RSP: 0018:c9000767f550 EFLAGS: 0286 RAX: 113f8d77 RBX: 88805a4aa380 RCX: 0006 RDX: dc00 RSI: 0002 RDI: RBP: 8880ae535e00 R08: 0001 R09: 8d0b69ef R10: fbfff1a16d3d R11: R12: 8880ae535e00 R13: 888087046300 R14: R15: 0001 finish_lock_switch kernel/sched/core.c:3517 [inline] finish_task_switch+0x150/0x790 kernel/sched/core.c:3617 context_switch kernel/sched/core.c:3781 [inline] __schedule+0xed1/0x2280 kernel/sched/core.c:4527 preempt_schedule_irq+0xbf/0x1b0 kernel/sched/core.c:4785 irqentry_exit_cond_resched kernel/entry/common.c:333 [inline] irqentry_exit_cond_resched kernel/entry/common.c:325 [inline] irqentry_exit+0x65/0x90 kernel/entry/common.c:363 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581 RIP: 0010:__sanitizer_cov_trace_pc+0x42/0x60 kernel/kcov.c:202 Code: 24 74 0f f6 c4 01 74 35 8b 82 2c 14 00 00 85 c0 74 2b 8b 82 08 14 00 00 83 f8 02 75 20 48 8b 8a 10 14 00 00 8b 92 0c 14 00 00 <48> 8b 01 48 83 c0 01 48 39 c2 76 07 48 89 34 c1 48 89 01 c3 66 2e RSP: 0018:c9000767f780 EFLAGS: 0246 RAX: 0002 RBX: c9000767f918 RCX: c90011138000 RDX: 0004 RSI: 83889163 RDI: c9000767f938 RBP: 88808c3a5f00 R08: R09: 8a68da47 R10: 0002 R11: R12: R13: 88808ea8f818 R14: 88808c3a5f18 R15: 0002 tomoyo_same_path_number_acl+0x63/0x2c0 security/tomoyo/file.c:639 tomoyo_update_domain+0x34c/0x850 security/tomoyo/domain.c:128 tomoyo_update_path_number_acl security/tomoyo/file.c:691 [inline] tomoyo_write_file+0x68b/0x7f0 security/tomoyo/file.c:1034
