On Fri, Jul 01, 2022 at 03:56:18AM -0600, Vitaliy Makkoveev wrote:
> CVSROOT: /cvs
> Module name: src
> Changes by: m...@cvs.openbsd.org 2022/07/01 03:56:18
>
> Modified files:
> sys/kern : uipc_socket.c uipc_socket2.c uipc_syscalls.c
> uipc_usrreq.c
> sys/miscfs/fifofs: fifo_vnops.c
> sys/sys : socketvar.h unpcb.h
>
> Log message:
> Make fine grained unix(4) domain sockets locking. Use the per-socket
> `so_lock' rwlock(9) instead of global `unp_lock' which locks the whole
> layer.
>
> The PCB of unix(4) sockets are linked to each other and we need to lock
> them both. This introduces the lock ordering problem, because when the
> thread (1) keeps lock on `so1' and trying to lock `so2', the thread (2)
> could hold lock on `so2' and trying to lock `so1'. To solve this we
> always lock sockets in the strict order.
>
> For the sockets which are already accessible from userland, we always
> lock socket with the smallest memory address first. Sometimes we need to
> unlock socket before lock it's peer and lock it again.
>
> We use reference counters for prevent the connected peer destruction
> during to relock. We also handle the case where the peer socket was
> replaced by another socket.
>
> For the newly connected sockets, which are not yet exported to the
> userland by accept(2), we always lock the listening socket `head' first.
> This allows us to avoid unwanted relock within accept(2) syscall.
>
> ok claudio@
syzkaller found a panic, I'm spotting one missed unlock. Feel free to
commit.
diff --git sys/kern/uipc_usrreq.c sys/kern/uipc_usrreq.c
index 0710393d376..7231b16b735 100644
--- sys/kern/uipc_usrreq.c
+++ sys/kern/uipc_usrreq.c
@@ -363,6 +363,7 @@ uipc_usrreq(struct socket *so, int req, struct mbuf *m,
struct mbuf *nam,
control = NULL;
} else {
error = ENOBUFS;
+ sounlock(so2);
break;
}
} else if (so->so_type == SOCK_SEQPACKET)
syzbot has found a reproducer for the following issue on:
HEAD commit: a4a82b864d54 Remove PIPEXCSESSION ioctl(2) call only from ..
git tree: openbsd
console output: https://syzkaller.appspot.com/x/log.txt?x=10cffe24080000
kernel config: https://syzkaller.appspot.com/x/.config?x=7058272de1526588
dashboard link: https://syzkaller.appspot.com/bug?extid=a648408d6a58fd40b59a
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15cc61f4080000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=178e44ec080000
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+a648408d6a58fd40b...@syzkaller.appspotmail.com
witness: userret: returning with the following locks held:
exclusive rwlock solock r = 0 (0xfffffd8073dde420)
#0 witness_lock+0x44d
#1 unp_solock_peer+0x64 sys/kern/uipc_usrreq.c:168
#2 uipc_usrreq+0x7c6 sys/kern/uipc_usrreq.c:350
#3 sosend+0x61b sys/kern/uipc_socket.c:657
#4 sendit+0x65d sys/kern/uipc_syscalls.c:682
#5 sys_sendmsg+0x198 sys/kern/uipc_syscalls.c:589
#6 syscall+0x4c3 mi_syscall sys/sys/syscall_mi.h:101 [inline]
#6 syscall+0x4c3 sys/arch/amd64/amd64/trap.c:585
#7 Xsyscall+0x128
panic: witness_warn
Stopped at db_enter+0x18: addq $0x8,%rsp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
* 35921 8907 0 0x2 0 0 syz-executor3905931047
442241 79361 73 0x1100010 0 1 syslogd
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:437
panic(ffffffff82593e80) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e witness_debugger
sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e
sys/kern/subr_witness.c:1473
userret(ffff800021233268) at userret+0x265 sys/kern/kern_sig.c:2012
syscall(ffff8000212825b0) at syscall+0x57e mi_syscall_return
sys/sys/syscall_mi.h:128 [inline]
syscall(ffff8000212825b0) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7fffff0b10, count: 9
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}>
ddb{0}> set $lines = 0
ddb{0}> set $maxwidth = 0
ddb{0}> show panic
*cpu0: witness_warn
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:437
panic(ffffffff82593e80) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e witness_debugger
sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e
sys/kern/subr_witness.c:1473
userret(ffff800021233268) at userret+0x265 sys/kern/kern_sig.c:2012
syscall(ffff8000212825b0) at syscall+0x57e mi_syscall_return
sys/sys/syscall_mi.h:128 [inline]
syscall(ffff8000212825b0) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7fffff0b10, count: -6
ddb{0}> show registers
rdi 0
rsi 0x1
rbp 0xffff8000212822e0
rbx 0xffffffff82937bb7 cpu_info_full_primary+0x2bb7
rdx 0x3fd
rcx 0
rax 0x14
r8 0x101010101010101
r9 0x8080808080808080
r10 0xb330cdf0a42dbe5
r11 0xab2e9c27cf6e23
r12 0xffffffff829379b8 cpu_info_full_primary+0x29b8
r13 0
r14 0
r15 0x1
rip 0xffffffff81d03c18 db_enter+0x18
cs 0x8
rflags 0x246
rsp 0xffff8000212822d0
ss 0x10
db_enter+0x18: addq $0x8,%rsp
ddb{0}> show proc
PROC (syz-executor3905931047) pid=35921 stat=onproc
flags process=2<EXEC> proc=0
pri=50, usrpri=51, nice=20
forw=0xffffffffffffffff, list=0xffff8000212337a8,0xffffffff82a6ea48
process=0xffff8000ffff10b0 user=0xffff80002127d000,
vmspace=0xfffffd806ee5d458
estcpu=1, cpticks=4, pctcpu=0.0
user=0, sys=4, intr=0
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
* 8907 35921 40417 0 7 0x2
syz-executor3905931047
40417 262142 92423 0 3 0x10008a sigsusp ksh
92423 197898 89384 0 3 0x9a kqread sshd
67007 222332 1 0 3 0x100083 ttyin getty
89384 96099 1 0 3 0x88 kqread sshd
17913 409899 95101 74 3 0x1100092 bpf pflogd
95101 411087 1 0 3 0x80 netio pflogd
79361 442241 8594 73 7 0x1100010 syslogd
8594 202546 1 0 3 0x100082 netio syslogd
78741 379903 1 0 3 0x100080 kqread resolvd
88508 494536 41847 77 3 0x100092 kqread dhcpleased
28115 467895 41847 77 3 0x100092 kqread dhcpleased
41847 358398 1 0 3 0x80 kqread dhcpleased
89451 195928 0 0 3 0x14200 bored smr
22769 378343 0 0 3 0x14200 pgzero zerothread
47500 472157 0 0 3 0x14200 aiodoned aiodoned
62613 471709 0 0 3 0x14200 syncer update
54963 298935 0 0 3 0x14200 cleaner cleaner
8710 227179 0 0 3 0x14200 reaper reaper
34091 23155 0 0 3 0x14200 pgdaemon pagedaemon
51011 495196 0 0 3 0x14200 bored viomb
64329 116108 0 0 3 0x40014200 acpi0 acpi0
3841 219163 0 0 3 0x40014200 idle1
49499 359978 0 0 3 0x14200 bored softnet
95986 369019 0 0 3 0x14200 bored softnet
66787 474740 0 0 3 0x14200 bored softnet
28433 495381 0 0 3 0x14200 bored softnet
92620 35932 0 0 3 0x14200 bored systqmp
94156 372483 0 0 3 0x14200 bored systq
80998 435403 0 0 3 0x40014200 bored softclock
42343 501218 0 0 3 0x40014200 idle0
1 263976 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{0}> show all locks
Process 8907 (syz-executor3905931047) thread 0xffff800021233268 (35921)
exclusive rwlock solock r = 0 (0xfffffd8073dde420)
#0 witness_lock+0x44d
#1 unp_solock_peer+0x64 sys/kern/uipc_usrreq.c:168
#2 uipc_usrreq+0x7c6 sys/kern/uipc_usrreq.c:350
#3 sosend+0x61b sys/kern/uipc_socket.c:657
#4 sendit+0x65d sys/kern/uipc_syscalls.c:682
#5 sys_sendmsg+0x198 sys/kern/uipc_syscalls.c:589
#6 syscall+0x4c3 mi_syscall sys/sys/syscall_mi.h:101 [inline]
#6 syscall+0x4c3 sys/arch/amd64/amd64/trap.c:585
#7 Xsyscall+0x128
Process 79361 (syslogd) thread 0xffff8000ffff7cf0 (442241)
exclusive kernel_lock &kernel_lock r = 0 (0xffffffff82a57e08)
#0 witness_lock+0x44d
#1 __mp_acquire_count+0x48 sys/kern/kern_lock.c:227
#2 mi_switch+0x3bb sys/kern/sched_bsd.c:416
#3 sleep_finish+0x180 sys/kern/kern_synch.c:420
#4 tsleep+0x12c sys/kern/kern_synch.c:158
#5 vwaitforio+0x98 sys/kern/vfs_subr.c:1897
#6 ffs_fsync+0x38e sys/ufs/ffs/ffs_vnops.c:495
#7 VOP_FSYNC+0xce sys/kern/vfs_vops.c:311
#8 sys_fsync+0x109 sys/kern/vfs_syscalls.c:2939
#9 syscall+0x435 mi_syscall sys/sys/syscall_mi.h:101 [inline]
#9 syscall+0x435 sys/arch/amd64/amd64/trap.c:585
#10 Xsyscall+0x128
exclusive rrwlock inode r = 0 (0xfffffd806e7bb5f0)
#0 witness_lock+0x44d
#1 rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2 rrw_enter+0x8b sys/kern/kern_rwlock.c:465
#3 VOP_LOCK+0x87 sys/kern/vfs_vops.c:518
#4 vn_lock+0x84 sys/kern/vfs_vnops.c:567
#5 sys_fsync+0xf5 sys/kern/vfs_syscalls.c:2939
#6 syscall+0x435 mi_syscall sys/sys/syscall_mi.h:101 [inline]
#6 syscall+0x435 sys/arch/amd64/amd64/trap.c:585
#7 Xsyscall+0x128
ddb{0}> show malloc
Type InUse MemUse HighUse Limit Requests Type Lim
devbuf 10152 6455K 6456K 78643K 11242 0
pcb 13 8K 8K 78643K 13 0
rtable 58 1K 2K 78643K 108 0
ifaddr 29 8K 8K 78643K 32 0
counters 40 33K 33K 78643K 40 0
ioctlops 0 0K 4K 78643K 1479 0
mount 1 1K 1K 78643K 1 0
log 0 0K 0K 78643K 5 0
vnodes 1167 73K 73K 78643K 1180 0
UFS quota 1 32K 32K 78643K 1 0
UFS mount 5 36K 36K 78643K 5 0
shm 2 1K 1K 78643K 2 0
VM map 2 1K 1K 78643K 2 0
sem 2 0K 0K 78643K 2 0
dirhash 12 2K 2K 78643K 12 0
ACPI 1697 195K 286K 78643K 12548 0
file desc 1 0K 0K 78643K 1 0
proc 67 91K 91K 78643K 282 0
NFS srvsock 1 0K 0K 78643K 1 0
NFS daemon 1 16K 16K 78643K 1 0
in_multi 11 0K 0K 78643K 11 0
ether_multi 1 0K 0K 78643K 1 0
ISOFS mount 1 32K 32K 78643K 1 0
MSDOSFS mount 1 16K 16K 78643K 1 0
ttys 25 122K 122K 78643K 25 0
exec 0 0K 2K 78643K 432 0
tdb 3 0K 0K 78643K 3 0
pagedep 1 8K 8K 78643K 1 0
inodedep 1 32K 32K 78643K 1 0
newblk 1 0K 0K 78643K 1 0
VM swap 7 26K 26K 78643K 7 0
UVM amap 57 3K 5K 78643K 1548 0
UVM aobj 3 2K 2K 78643K 3 0
memdesc 1 4K 4K 78643K 1 0
crypto data 1 1K 1K 78643K 1 0
NDP 4 0K 0K 78643K 4 0
temp 23 4705K 4769K 78643K 2514 0
kqueue 11 16K 18K 78643K 24 0
SYN cache 2 16K 16K 78643K 2 0
ddb{0}> show all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
plcache 128 22 0 0 1 0 1 1 0 8 0
rtpcb 120 17 0 14 1 0 1 1 0 8 0
rtentry 112 23 0 1 1 0 1 1 0 8 0
unpcb 144 37 0 20 1 0 1 1 0 8 0
syncache 296 5 0 5 2 1 1 1 0 8 1
tcpcb 736 8 0 5 1 0 1 1 0 8 0
arp 120 2 0 0 1 0 1 1 0 8 0
inpcb 320 32 0 26 1 0 1 1 0 8 0
pfosfp 40 1428 0 1005 5 0 5 5 0 8 0
pfosfpen 112 1428 0 714 21 0 21 21 0 8 0
pfstitem 24 9 0 0 1 0 1 1 0 8 0
pfstkey 120 9 0 0 1 0 1 1 0 8 0
pfstate 336 9 0 0 1 0 1 1 0 8 0
pfrule 1360 21 0 16 2 1 1 2 0 8 0
art_heap8 4096 1 0 0 1 0 1 1 0 8 0
art_heap4 256 97 0 0 7 0 7 7 0 8 0
art_table 32 98 0 0 1 0 1 1 0 8 0
art_node 16 22 0 2 1 0 1 1 0 8 0
dirhash 1024 17 0 0 3 0 3 3 0 8 0
dino2pl 256 1425 0 39 87 0 87 87 0 8 0
ffsino 272 1425 0 39 93 0 93 93 0 8 0
nchpl 144 1614 0 49 58 0 58 58 0 8 0
uvmvnodes 80 1435 0 0 30 0 30 30 0 8 0
vnodes 224 1435 0 0 85 0 85 85 0 8 0
namei 1024 4472 0 4472 1 0 1 1 0 8 1
percpumem 16 32 0 0 1 0 1 1 0 8 0
kstatmem 264 8 0 0 1 0 1 1 0 8 0
scxspl 216 3954 0 3953 2 1 1 2 0 8 0
plimitpl 152 16 0 9 1 0 1 1 0 8 0
sigapl 424 323 0 291 5 1 4 5 0 8 0
knotepl 120 51 0 0 2 0 2 2 0 8 0
kqueuepl 224 20 0 13 1 0 1 1 0 8 0
pipepl 336 86 0 83 1 0 1 1 0 8 0
fdescpl 496 306 0 291 3 0 3 3 0 8 0
filepl 152 1184 0 1124 3 0 3 3 0 8 0
lockfpl 104 6 0 4 1 0 1 1 0 8 0
lockfspl 48 4 0 2 1 0 1 1 0 8 0
sessionpl 144 18 0 9 1 0 1 1 0 8 0
pgrppl 48 18 0 9 1 0 1 1 0 8 0
ucredpl 104 69 0 57 1 0 1 1 0 8 0
zombiepl 144 291 0 291 2 1 1 1 0 8 1
processpl 1064 323 0 291 3 0 3 3 0 8 0
procpl 672 323 0 291 4 1 3 4 0 8 0
sockpl 504 86 0 60 4 0 4 4 0 8 0
mcl8k 8192 4 0 0 1 0 1 1 0 8 0
mcl4k 4096 3 0 0 1 0 1 1 0 8 0
mcl2k 2048 69 0 0 8 0 8 8 0 8 0
mtagpl 96 3 0 0 1 0 1 1 0 8 0
mbufpl 256 105 0 0 7 0 7 7 0 8 0
bufpl 288 2014 0 96 137 0 137 137 0 8 0
anonpl 24 38006 0 36011 16 2 14 15 0 186 1
amapchunkpl 152 2803 0 2708 6 2 4 6 0 158 0
amappl16 200 97 0 92 1 0 1 1 0 8 0
amappl15 192 1 0 1 1 1 0 1 0 8 0
amappl13 176 34 0 33 2 1 1 1 0 8 0
amappl12 168 3 0 3 2 1 1 1 0 8 1
amappl11 160 66 0 49 1 0 1 1 0 8 0
amappl10 152 2 0 0 1 0 1 1 0 8 0
amappl9 144 485 0 483 1 0 1 1 0 8 0
amappl8 136 364 0 363 2 1 1 1 0 8 0
amappl7 128 52 0 49 1 0 1 1 0 8 0
amappl6 120 105 0 95 1 0 1 1 0 8 0
amappl5 112 92 0 78 1 0 1 1 0 8 0
amappl4 104 631 0 609 1 0 1 1 0 8 0
amappl3 96 403 0 379 1 0 1 1 0 8 0
amappl2 88 454 0 416 1 0 1 1 0 8 0
amappl1 80 9479 0 9071 11 1 10 10 0 8 1
amappl 88 1267 0 1228 2 1 1 2 0 92 0
dma4096 4096 1 0 1 1 1 0 1 0 8 0
dma1024 1024 1 0 0 1 0 1 1 0 8 0
dma256 256 6 0 6 1 1 0 1 0 8 0
dma128 128 253 0 253 1 1 0 1 0 8 0
dma64 64 6 0 6 1 1 0 1 0 8 0
dma32 32 7 0 7 1 1 0 1 0 8 0
dma16 16 18 0 17 1 0 1 1 0 8 0
aobjpl 72 2 0 0 1 0 1 1 0 8 0
uaddrrnd 24 306 0 291 1 0 1 1 0 8 0
uaddrbest 32 2 0 0 1 0 1 1 0 8 0
uaddr 24 306 0 291 1 0 1 1 0 8 0
vmmpekpl 168 6300 0 6288 1 0 1 1 0 8 0
vmmpepl 168 27039 0 26178 43 1 42 42 0 357 0
vmsppl 368 305 0 291 2 0 2 2 0 8 0
rwobjpl 56 9720 0 7711 30 0 30 30 0 8 0
pdppl 4096 619 0 582 52 15 37 45 0 8 0
pvpl 32 126743 0 122778 37 1 36 36 0 265 3
pmappl 248 305 0 291 2 0 2 2 0 8 0
extentpl 40 58 0 38 1 0 1 1 0 8 0
phpool 112 412 0 18 12 0 12 12 0 8 0
ddb{0}> machine ddbcpu 0
Invalid cpu 0
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:437
panic(ffffffff82593e80) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e witness_debugger
sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff82620d13) at witness_warn+0x65e
sys/kern/subr_witness.c:1473
userret(ffff800021233268) at userret+0x265 sys/kern/kern_sig.c:2012
syscall(ffff8000212825b0) at syscall+0x57e mi_syscall_return
sys/sys/syscall_mi.h:128 [inline]
syscall(ffff8000212825b0) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7fffff0b10, count: -6
ddb{0}> machine ddbcpu 1
Stopped at x86_ipi_db+0x1a: addq $0x8,%rsp
x86_ipi_db(ffff800020dd8ff0) at x86_ipi_db+0x1a
sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_acquire_count(ffffffff82a57c00,ffffffff82a57c00) at
__mp_acquire_count+0x10 sys/kern/kern_lock.c:226
tsleep(fffffd8079e5e610,11,ffffffff825b249b,0) at tsleep+0x179
vwaitforio(fffffd8079e5e590,0,ffffffff825b249b,ffffffffffffffff) at
vwaitforio+0x98 sys/kern/vfs_subr.c:1897
ffs_fsync(ffff8000212b8c70) at ffs_fsync+0x38e sys/ufs/ffs/ffs_vnops.c:495
VOP_FSYNC(fffffd8079e5e590,fffffd807f7d79c0,1,ffff8000ffff7cf0) at
VOP_FSYNC+0xce sys/kern/vfs_vops.c:311
sys_fsync(ffff8000ffff7cf0,ffff8000212b8d60,ffff8000212b8db0) at
sys_fsync+0x109 sys/kern/vfs_syscalls.c:2939
syscall(ffff8000212b8e30) at syscall+0x435 mi_syscall sys/sys/syscall_mi.h:101
[inline]
syscall(ffff8000212b8e30) at syscall+0x435 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffda750, count: 4
ddb{1}> trace
x86_ipi_db(ffff800020dd8ff0) at x86_ipi_db+0x1a
sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_acquire_count(ffffffff82a57c00,ffffffff82a57c00) at
__mp_acquire_count+0x10 sys/kern/kern_lock.c:226
tsleep(fffffd8079e5e610,11,ffffffff825b249b,0) at tsleep+0x179
vwaitforio(fffffd8079e5e590,0,ffffffff825b249b,ffffffffffffffff) at
vwaitforio+0x98 sys/kern/vfs_subr.c:1897
ffs_fsync(ffff8000212b8c70) at ffs_fsync+0x38e sys/ufs/ffs/ffs_vnops.c:495
VOP_FSYNC(fffffd8079e5e590,fffffd807f7d79c0,1,ffff8000ffff7cf0) at
VOP_FSYNC+0xce sys/kern/vfs_vops.c:311
sys_fsync(ffff8000ffff7cf0,ffff8000212b8d60,ffff8000212b8db0) at
sys_fsync+0x109 sys/kern/vfs_syscalls.c:2939
syscall(ffff8000212b8e30) at syscall+0x435 mi_syscall sys/sys/syscall_mi.h:101
[inline]
syscall(ffff8000212b8e30) at syscall+0x435 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffda750, count: -11
ddb{1}>
