Re: possible deadlock in send_sigurg (2)
On Thu, 2020-11-05 at 14:23 +0800, Boqun Feng wrote:
> Hi,
>
> On Wed, Nov 04, 2020 at 04:18:08AM -0800, syzbot wrote:
> > syzbot has bisected this issue to:
> >
> > commit e918188611f073063415f40fae568fa4d86d9044
> > Author: Boqun Feng
> > Date: Fri Aug 7 07:42:20 2020 +
> >
> > locking: More accurate annotations for read_lock()
> >
> > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1414273250
> > start commit: 4ef8451b Merge tag 'perf-tools-for-v5.10-2020-11-03' of gi..
> > git tree: upstream
> > final oops: https://syzkaller.appspot.com/x/report.txt?x=1614273250
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1214273250
> > kernel config: https://syzkaller.appspot.com/x/.config?x=61033507391c77ff
> > dashboard link: https://syzkaller.appspot.com/bug?extid=c5e32344981ad9f33750
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1519786250
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13c59f6c50
> >
> > Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
> > Fixes: e918188611f0 ("locking: More accurate annotations for read_lock()")
> >
> > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
>
> Thanks for reporting this, and this is actually a deadlock potential
> detected by the newly added recursive read deadlock detection as my
> analysis:
>
>
> https://lore.kernel.org/lkml/20200910071523.gf7...@debian-boqun.qqnc3lrjykvubdpftowmye0fmh.lx.internal.cloudapp.net
>
> Besides, other reports[1][2] are caused by the same problem. I made a
> fix for this, please have a try and see if it's get fixed.
>
> Regards,
> Boqun
>
> [1]: https://lore.kernel.org/lkml/d7136005aee14...@google.com
> [2]: https://lore.kernel.org/lkml/6e29ed05b3009...@google.com
>
> ->8
> From 7fbe730fcff2d7909be034cf6dc8bf0604d0bf14 Mon Sep 17 00:00:00 2001
> From: Boqun Feng
> Date: Thu, 5 Nov 2020 14:02:57 +0800
> Subject: [PATCH] fs/fcntl: Fix potential deadlock in send_sig{io, urg}()
>
> Syzbot reports a potential deadlock found by the newly added recursive
> read deadlock detection in lockdep:
>
> [...]
> [...] WARNING: possible irq lock inversion dependency detected
> [...] 5.9.0-rc2-syzkaller #0 Not tainted
> [...]
> [...] syz-executor.1/10214 just changed the state of lock:
> [...] 88811f506338 (>f_owner.lock){.+..}-{2:2}, at:
> send_sigurg+0x1d/0x200
> [...] but this lock was taken by another, HARDIRQ-safe lock in the past:
> [...] (>event_lock){-...}-{2:2}
> [...]
> [...]
> [...] and interrupts could create inverse lock ordering between them.
> [...]
> [...]
> [...] other info that might help us debug this:
> [...] Chain exists of:
> [...] >event_lock --> >fa_lock --> >f_owner.lock
> [...]
> [...] Possible interrupt unsafe locking scenario:
> [...]
> [...]CPU0CPU1
> [...]
> [...] lock(>f_owner.lock);
> [...]local_irq_disable();
> [...]lock(>event_lock);
> [...]lock(>fa_lock);
> [...]
> [...] lock(>event_lock);
> [...]
> [...] *** DEADLOCK ***
>
> The corresponding deadlock case is as followed:
>
> CPU 0 CPU 1 CPU 2
> read_lock(>lock);
> spin_lock_irqsave(>event_lock, ...)
> write_lock_irq(>f_owner.lock); //
> wait for the lock
> read_lock(); // have to wait until the writer
> release
> // due to the fairness
>
> spin_lock_irqsave(>event_lock); // wait for the lock
>
> The lock dependency on CPU 1 happens if there exists a call sequence:
>
> input_inject_event():
> spin_lock_irqsave(>event_lock,...);
> input_handle_event():
> input_pass_values():
> input_to_handler():
> handler->event(): // evdev_event()
> evdev_pass_values():
> spin_lock(>buffer_lock);
> __pass_event():
> kill_fasync():
> kill_fasync_rcu():
> read_lock(>fa_lock);
> send_sigio():
> read_lock(>lock);
>
> To fix this, make the reader in send_sigurg() and send_sigio() use
> read_lock_irqsave() and read_lock_irqrestore().
>
> Reported-by: syzbot+22e87cdf94021b984...@syzkaller.appspotmail.com
> Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
> Signed-off-by: Boqun Feng
> ---
> fs/fcntl.c | 10 ++
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/fs/fcntl.c b/fs/fcntl.c
> index 19ac5baad50f..05b36b28f2e8
Re: possible deadlock in send_sigurg (2)
Hi,
On Wed, Nov 04, 2020 at 04:18:08AM -0800, syzbot wrote:
> syzbot has bisected this issue to:
>
> commit e918188611f073063415f40fae568fa4d86d9044
> Author: Boqun Feng
> Date: Fri Aug 7 07:42:20 2020 +
>
> locking: More accurate annotations for read_lock()
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1414273250
> start commit: 4ef8451b Merge tag 'perf-tools-for-v5.10-2020-11-03' of gi..
> git tree: upstream
> final oops: https://syzkaller.appspot.com/x/report.txt?x=1614273250
> console output: https://syzkaller.appspot.com/x/log.txt?x=1214273250
> kernel config: https://syzkaller.appspot.com/x/.config?x=61033507391c77ff
> dashboard link: https://syzkaller.appspot.com/bug?extid=c5e32344981ad9f33750
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1519786250
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13c59f6c50
>
> Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
> Fixes: e918188611f0 ("locking: More accurate annotations for read_lock()")
>
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection
Thanks for reporting this, and this is actually a deadlock potential
detected by the newly added recursive read deadlock detection as my
analysis:
https://lore.kernel.org/lkml/20200910071523.gf7...@debian-boqun.qqnc3lrjykvubdpftowmye0fmh.lx.internal.cloudapp.net
Besides, other reports[1][2] are caused by the same problem. I made a
fix for this, please have a try and see if it's get fixed.
Regards,
Boqun
[1]: https://lore.kernel.org/lkml/d7136005aee14...@google.com
[2]: https://lore.kernel.org/lkml/6e29ed05b3009...@google.com
->8
>From 7fbe730fcff2d7909be034cf6dc8bf0604d0bf14 Mon Sep 17 00:00:00 2001
From: Boqun Feng
Date: Thu, 5 Nov 2020 14:02:57 +0800
Subject: [PATCH] fs/fcntl: Fix potential deadlock in send_sig{io, urg}()
Syzbot reports a potential deadlock found by the newly added recursive
read deadlock detection in lockdep:
[...]
[...] WARNING: possible irq lock inversion dependency detected
[...] 5.9.0-rc2-syzkaller #0 Not tainted
[...]
[...] syz-executor.1/10214 just changed the state of lock:
[...] 88811f506338 (>f_owner.lock){.+..}-{2:2}, at:
send_sigurg+0x1d/0x200
[...] but this lock was taken by another, HARDIRQ-safe lock in the past:
[...] (>event_lock){-...}-{2:2}
[...]
[...]
[...] and interrupts could create inverse lock ordering between them.
[...]
[...]
[...] other info that might help us debug this:
[...] Chain exists of:
[...] >event_lock --> >fa_lock --> >f_owner.lock
[...]
[...] Possible interrupt unsafe locking scenario:
[...]
[...]CPU0CPU1
[...]
[...] lock(>f_owner.lock);
[...]local_irq_disable();
[...]lock(>event_lock);
[...]lock(>fa_lock);
[...]
[...] lock(>event_lock);
[...]
[...] *** DEADLOCK ***
The corresponding deadlock case is as followed:
CPU 0 CPU 1 CPU 2
read_lock(>lock);
spin_lock_irqsave(>event_lock, ...)
write_lock_irq(>f_owner.lock); //
wait for the lock
read_lock(); // have to wait until the writer
release
// due to the fairness
spin_lock_irqsave(>event_lock); // wait for the lock
The lock dependency on CPU 1 happens if there exists a call sequence:
input_inject_event():
spin_lock_irqsave(>event_lock,...);
input_handle_event():
input_pass_values():
input_to_handler():
handler->event(): // evdev_event()
evdev_pass_values():
spin_lock(>buffer_lock);
__pass_event():
kill_fasync():
kill_fasync_rcu():
read_lock(>fa_lock);
send_sigio():
read_lock(>lock);
To fix this, make the reader in send_sigurg() and send_sigio() use
read_lock_irqsave() and read_lock_irqrestore().
Reported-by: syzbot+22e87cdf94021b984...@syzkaller.appspotmail.com
Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
Signed-off-by: Boqun Feng
---
fs/fcntl.c | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/fs/fcntl.c b/fs/fcntl.c
index 19ac5baad50f..05b36b28f2e8 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -781,9 +781,10 @@ void send_sigio(struct fown_struct *fown, int fd, int band)
{
struct task_struct *p;
enum pid_type type;
+ unsigned long flags;
struct pid *pid;
-
Re: possible deadlock in send_sigurg (2)
syzbot has bisected this issue to:
commit e918188611f073063415f40fae568fa4d86d9044
Author: Boqun Feng
Date: Fri Aug 7 07:42:20 2020 +
locking: More accurate annotations for read_lock()
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1414273250
start commit: 4ef8451b Merge tag 'perf-tools-for-v5.10-2020-11-03' of gi..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=1614273250
console output: https://syzkaller.appspot.com/x/log.txt?x=1214273250
kernel config: https://syzkaller.appspot.com/x/.config?x=61033507391c77ff
dashboard link: https://syzkaller.appspot.com/bug?extid=c5e32344981ad9f33750
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1519786250
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13c59f6c50
Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
Fixes: e918188611f0 ("locking: More accurate annotations for read_lock()")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
Re: possible deadlock in send_sigurg (2)
syzbot has found a reproducer for the following issue on:
HEAD commit:4ef8451b Merge tag 'perf-tools-for-v5.10-2020-11-03' of gi..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=16ab063a50
kernel config: https://syzkaller.appspot.com/x/.config?x=61033507391c77ff
dashboard link: https://syzkaller.appspot.com/bug?extid=c5e32344981ad9f33750
compiler: gcc (GCC) 10.1.0-syz 20200507
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1519786250
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13c59f6c50
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
netdevsim netdevsim0 netdevsim1: set [1, 0] type 2 family 0 port 6081 - 0
netdevsim netdevsim0 netdevsim2: set [1, 0] type 2 family 0 port 6081 - 0
netdevsim netdevsim0 netdevsim3: set [1, 0] type 2 family 0 port 6081 - 0
nf_conntrack: default automatic helper assignment has been turned off for
security reasons and CT-based firewall rule not found. Use the iptables CT
target to attach helpers instead.
WARNING: possible irq lock inversion dependency detected
5.10.0-rc2-syzkaller #0 Not tainted
syz-executor981/8491 just changed the state of lock:
88801798e638 (>f_owner.lock){.+..}-{2:2}, at: send_sigurg+0x1e/0xac0
fs/fcntl.c:824
but this lock was taken by another, HARDIRQ-safe lock in the past:
(>event_lock){-...}-{2:2}
and interrupts could create inverse lock ordering between them.
other info that might help us debug this:
Chain exists of:
>event_lock --> >fa_lock --> >f_owner.lock
Possible interrupt unsafe locking scenario:
CPU0CPU1
lock(>f_owner.lock);
local_irq_disable();
lock(>event_lock);
lock(>fa_lock);
lock(>event_lock);
*** DEADLOCK ***
1 lock held by syz-executor981/8491:
#0: 888023e118a0 (sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock
include/net/sock.h:1581 [inline]
#0: 888023e118a0 (sk_lock-AF_INET){+.+.}-{0:0}, at: inet_wait_for_connect
net/ipv4/af_inet.c:596 [inline]
#0: 888023e118a0 (sk_lock-AF_INET){+.+.}-{0:0}, at:
__inet_stream_connect+0x596/0xe30 net/ipv4/af_inet.c:686
the shortest dependencies between 2nd lock and 1st lock:
-> (>event_lock){-...}-{2:2} {
IN-HARDIRQ-W at:
lock_acquire kernel/locking/lockdep.c:5436 [inline]
lock_acquire+0x2a3/0x8c0 kernel/locking/lockdep.c:5401
__raw_spin_lock_irqsave
include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x39/0x50
kernel/locking/spinlock.c:159
input_event drivers/input/input.c:440 [inline]
input_event+0x7b/0xb0 drivers/input/input.c:433
input_report_key include/linux/input.h:417 [inline]
psmouse_report_standard_buttons+0x2c/0x80
drivers/input/mouse/psmouse-base.c:123
psmouse_report_standard_packet
drivers/input/mouse/psmouse-base.c:141 [inline]
psmouse_process_byte+0x1e1/0x890
drivers/input/mouse/psmouse-base.c:232
psmouse_handle_byte+0x41/0x1b0
drivers/input/mouse/psmouse-base.c:274
psmouse_interrupt+0x304/0xf00
drivers/input/mouse/psmouse-base.c:426
serio_interrupt+0x88/0x150
drivers/input/serio/serio.c:1002
i8042_interrupt+0x27a/0x520
drivers/input/serio/i8042.c:598
__handle_irq_event_percpu+0x303/0x8f0
kernel/irq/handle.c:156
handle_irq_event_percpu kernel/irq/handle.c:196
[inline]
handle_irq_event+0x102/0x290 kernel/irq/handle.c:213
handle_edge_irq+0x25f/0xd00 kernel/irq/chip.c:819
asm_call_irq_on_stack+0xf/0x20
__run_irq_on_irqstack
arch/x86/include/asm/irq_stack.h:48 [inline]
run_irq_on_irqstack_cond
arch/x86/include/asm/irq_stack.h:101 [inline]
handle_irq arch/x86/kernel/irq.c:230 [inline]
__common_interrupt arch/x86/kernel/irq.c:249 [inline]
common_interrupt+0x120/0x200 arch/x86/kernel/irq.c:239
asm_common_interrupt+0x1e/0x40
arch/x86/include/asm/idtentry.h:622
native_safe_halt arch/x86/include/asm/irqflags.h:60
[inline]
arch_safe_halt arch/x86/include/asm/irqflags.h:103
[inline]
acpi_safe_halt drivers/acpi/processor_idle.c:111
[inline]
possible deadlock in send_sigurg (2)
Hello,
syzbot found the following issue on:
HEAD commit:7cf726a5 Merge tag 'linux-kselftest-kunit-5.10-rc1' of git..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=14ba8ea050
kernel config: https://syzkaller.appspot.com/x/.config?x=80f536b098d66fd2
dashboard link: https://syzkaller.appspot.com/bug?extid=c5e32344981ad9f33750
compiler: gcc (GCC) 10.1.0-syz 20200507
Unfortunately, I don't have any reproducer for this issue yet.
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c5e32344981ad9f33...@syzkaller.appspotmail.com
WARNING: possible irq lock inversion dependency detected
5.9.0-syzkaller #0 Not tainted
syz-executor.4/8709 just changed the state of lock:
8880132a3438 (>f_owner.lock){.+..}-{2:2}, at: send_sigurg+0x1e/0xac0
fs/fcntl.c:824
but this lock was taken by another, HARDIRQ-safe lock in the past:
(>event_lock){-...}-{2:2}
and interrupts could create inverse lock ordering between them.
other info that might help us debug this:
Chain exists of:
>event_lock --> >fa_lock --> >f_owner.lock
Possible interrupt unsafe locking scenario:
CPU0CPU1
lock(>f_owner.lock);
local_irq_disable();
lock(>event_lock);
lock(>fa_lock);
lock(>event_lock);
*** DEADLOCK ***
1 lock held by syz-executor.4/8709:
#0: 88802a83c160 (sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock
include/net/sock.h:1581 [inline]
#0: 88802a83c160 (sk_lock-AF_INET){+.+.}-{0:0}, at: inet_wait_for_connect
net/ipv4/af_inet.c:596 [inline]
#0: 88802a83c160 (sk_lock-AF_INET){+.+.}-{0:0}, at:
__inet_stream_connect+0x596/0xe30 net/ipv4/af_inet.c:686
the shortest dependencies between 2nd lock and 1st lock:
-> (>event_lock){-...}-{2:2} {
IN-HARDIRQ-W at:
lock_acquire kernel/locking/lockdep.c:5442 [inline]
lock_acquire+0x219/0x9d0 kernel/locking/lockdep.c:5407
__raw_spin_lock_irqsave
include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x94/0xd0
kernel/locking/spinlock.c:159
input_event drivers/input/input.c:440 [inline]
input_event+0x7b/0xb0 drivers/input/input.c:433
input_report_key include/linux/input.h:417 [inline]
psmouse_report_standard_buttons+0x2c/0x80
drivers/input/mouse/psmouse-base.c:123
psmouse_report_standard_packet
drivers/input/mouse/psmouse-base.c:141 [inline]
psmouse_process_byte+0x1e1/0x890
drivers/input/mouse/psmouse-base.c:232
psmouse_handle_byte+0x41/0x1b0
drivers/input/mouse/psmouse-base.c:274
psmouse_interrupt+0x304/0xf00
drivers/input/mouse/psmouse-base.c:426
serio_interrupt+0x88/0x150
drivers/input/serio/serio.c:1002
i8042_interrupt+0x27a/0x520
drivers/input/serio/i8042.c:598
__handle_irq_event_percpu+0x20b/0x9e0
kernel/irq/handle.c:156
handle_irq_event_percpu kernel/irq/handle.c:196
[inline]
handle_irq_event+0x102/0x290 kernel/irq/handle.c:213
handle_edge_irq+0x25f/0xd00 kernel/irq/chip.c:819
asm_call_irq_on_stack+0xf/0x20
__run_irq_on_irqstack
arch/x86/include/asm/irq_stack.h:48 [inline]
run_irq_on_irqstack_cond
arch/x86/include/asm/irq_stack.h:101 [inline]
handle_irq arch/x86/kernel/irq.c:230 [inline]
__common_interrupt arch/x86/kernel/irq.c:249 [inline]
common_interrupt+0x120/0x200 arch/x86/kernel/irq.c:239
asm_common_interrupt+0x1e/0x40
arch/x86/include/asm/idtentry.h:622
arch_local_irq_restore
arch/x86/include/asm/paravirt.h:653 [inline]
__raw_spin_unlock_irqrestore
include/linux/spinlock_api_smp.h:160 [inline]
_raw_spin_unlock_irqrestore+0x4d/0x90
kernel/locking/spinlock.c:191
spin_unlock_irqrestore include/linux/spinlock.h:409
[inline]
i8042_command+0x111/0x130
drivers/input/serio/i8042.c:348
i8042_aux_write+0xd7/0x120
drivers/input/serio/i8042.c:383
serio_write include/linux/serio.h:125 [inline]
ps2_do_sendbyte+0x2cc/0x640
drivers/input/serio/libps2.c:40
ps2_sendbyte+0x4b/0x90
