>Synopsis: ssh with ControlPath on NFS causes netlock kernel panic
>Category: kernel
>Environment:
System : OpenBSD 6.0
Details : OpenBSD 6.0-current (GENERIC.MP) #73: Mon Dec 26
23:25:48 MST 2016
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
First noticed this problem with my workstation (/home is on NFS),
after upgrading to the Dec 23 snapshot, using ssh caused a kernel panic:
panic: rw_enter: netlock locking against myself
I waited for the Dec 27 snapshot
(Build date: 1482820394 - Tue Dec 27 06:33:14 UTC 2016)
and tried again, same result. After disabling SSH connection
multiplexing, using ssh didn't cause a panic.
To better diagnose the problem, I upgraded my laptop
(local /home) to the same snapshot, ssh with ControlPath on the local
/home worked fine, but as soon as I put the ControlPath on a NFS share,
exiting ssh caused a kernel panic.
If I try ssh again, after rebooting with the multiplexing socket
still there, I get an immediate panic, if I delete the socket first,
it will again panic when exiting ssh.
This didn't happen with the last snapshot I installed, unfortunately
that was back in October.
even boot reboot in ddb won't do anything, I have to use the
reset button/power cycle the machine
relevant nfs mount/ddb output/ssh -vvv/dmesg/usbdevs following, if you
need any more info or tests (other architectures are also possible),
I'll do my best.
I hope I/Thunderbird didn't mess up the sendbug output too much.
Thank you very much for your help and OpenBSD!
nfs mount (domain/username changed):
nfs.DOMAIN.net:/freigabe1 on /home/USERNAME/nfs type nfs (nodev, nosuid,
v3, udp, timeo=100, retrans=101)
ddb output from serial console:
panic: rw_enter: netlock locking against myself
Stopped at Debugger+0x9: leave
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*478420 98645 1000 0x3 0 1 ssh
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
sosend() at sosend+0x114
nfs_send() at nfs_send+0x60
nfs_request() at nfs_request+0x408
nfs_access() at nfs_access+0x178
VOP_ACCESS() at VOP_ACCESS+0x3c
unp_connect() at unp_connect+0x13c
uipc_usrreq() at uipc_usrreq+0x26f
soconnect() at soconnect+0xbb
sys_connect() at sys_connect+0x1c2
syscall() at syscall+0x27b
--- syscall (number 98) ---
end of kernel
end trace frame: 0x6a102eee2c0, count: 2
0x6a19d0f7eea:
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{1}> trace
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
sosend() at sosend+0x114
nfs_send() at nfs_send+0x60
nfs_request() at nfs_request+0x408
nfs_access() at nfs_access+0x178
VOP_ACCESS() at VOP_ACCESS+0x3c
unp_connect() at unp_connect+0x13c
uipc_usrreq() at uipc_usrreq+0x26f
soconnect() at soconnect+0xbb
sys_connect() at sys_connect+0x1c2
syscall() at syscall+0x27b
--- syscall (number 98) ---
end of kernel
end trace frame: 0x6a102eee2c0, count: -13
0x6a19d0f7eea:
ddb{1}> mach ddbcpu 0
Stopped at Debugger+0x9: leave
Debugger() at Debugger+0x9
x86_ipi_handler() at x86_ipi_handler+0x76
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
__mp_lock() at __mp_lock+0x40
softintr_dispatch() at softintr_dispatch+0x43
Xsoftclock() at Xsoftclock+0x1f
--- interrupt ---
end trace frame: 0x0, count: 9
0x8:
ddb{0}> trace
Debugger() at Debugger+0x9
x86_ipi_handler() at x86_ipi_handler+0x76
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
__mp_lock() at __mp_lock+0x40
softintr_dispatch() at softintr_dispatch+0x43
Xsoftclock() at Xsoftclock+0x1f
--- interrupt ---
end trace frame: 0x0, count: -6
0x8:
ddb{0}> mach ddbcpu 1
Stopped at Debugger+0x9: leave
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
sosend() at sosend+0x114
nfs_send() at nfs_send+0x60
nfs_request() at nfs_request+0x408
nfs_access() at nfs_access+0x178
VOP_ACCESS() at VOP_ACCESS+0x3c
unp_connect() at unp_connect+0x13c
uipc_usrreq() at uipc_usrreq+0x26f
soconnect() at soconnect+0xbb
sys_connect() at sys_connect+0x1c2
syscall() at syscall+0x27b
--- syscall (number 98) ---
end of kernel
end trace frame: 0x6a102eee2c0, count: 2
0x6a19d0f7eea:
ddb{1}> trace
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
sosend() at sosend+0x114
nfs_send() at nfs_send+0x60
nfs_request() at nfs_request+0x408
nfs_access() at nfs_access+0x178
VOP_ACCESS() at VOP_ACCESS+0x3c
unp_connect() at unp_connect+0x13c
uipc_usrreq() at uipc_usrreq+0x26f
soconnect() at soconnect+0xbb
sys_connect() at sys_connect+0x1c2
syscall() at syscall+0x27b
--- syscall (number 98) ---
end of kernel
end trace frame: 0x6a102eee2c0, count: -13
0x6a19d0f7eea:
ddb{1}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
*98645 47125 98645 1000 7 0x3 ssh
50889 0 0 0 3 0x14280 nfsidl nfsio
63573 0 0 0 3 0x14280 nfsidl nfsio
11386 0 0 0 3 0x14280 nfsidl nfsio
68742 0 0 0 3 0x14280 nfsidl nfsio
47125 23105 47125 1000 3 0x10008b pause ksh
23105 1 23105 1000 3 0x1000b2 select xterm
49998 1 49998 1000 3 0x80 poll dbus-daemon
42804 1 39499 1000 3 0x80 select dbus-launch
48601 39499 39499 1000 3 0x82 poll awesome
48601 39499 39499 1000 3 0x4000082 poll awesome
39499 36294 39499 1000 3 0x10008a pause sh
88238 1 38332 35 3 0x100090 poll xconsole
23972 1 38332 0 3 0x80 netio xconsole
36294 35061 36294 0 3 0x80 wait xdm
52028 14088 14088 0 3 0x1000b0 netio Xorg
72921 1 72921 0 3 0x100083 ttyin getty
65745 1 65745 0 3 0x100083 ttyin getty
71789 1 71789 0 3 0x100083 ttyin getty
33528 1 33528 0 3 0x100083 ttyin getty
72375 1 72375 0 3 0x100083 ttyin getty
14088 35061 14088 35 3 0xb2 select Xorg
35061 1 35061 0 3 0x88 pause xdm
7318 1 7318 0 3 0x100098 poll cron
98636 1 98636 0 3 0x80 kqread apmd
45198 1 45198 99 3 0x100090 poll sndiod
33071 1 33071 110 3 0x100090 poll sndiod
23480 39004 39004 95 3 0x100092 kqread smtpd
91857 39004 39004 103 3 0x100092 kqread smtpd
57941 39004 39004 95 3 0x100092 kqread smtpd
38559 39004 39004 95 3 0x100092 kqread smtpd
69839 39004 39004 95 3 0x100092 kqread smtpd
78235 39004 39004 95 3 0x100092 kqread smtpd
39004 1 39004 0 3 0x100080 kqread smtpd
18080 1 18080 0 3 0x80 select sshd
5415 62918 83714 83 3 0x100092 poll ntpd
62918 83714 83714 83 3 0x100092 poll ntpd
83714 1 83714 0 3 0x100080 poll ntpd
78835 92198 92198 74 3 0x100090 bpf pflogd
92198 1 92198 0 3 0x80 netio pflogd
58417 74759 74759 73 3 0x100090 kqread syslogd
74759 1 74759 0 3 0x100082 netio syslogd
95763 1 95763 77 3 0x100090 poll dhclient
56079 1 56079 0 3 0x80 poll dhclient
99814 0 0 0 3 0x14200 pgzero zerothread
18309 0 0 0 3 0x14200 aiodoned aiodoned
71170 0 0 0 3 0x14200 syncer update
8712 0 0 0 3 0x14200 cleaner cleaner
70310 0 0 0 3 0x14200 reaper reaper
87311 0 0 0 3 0x14200 pgdaemon pagedaemon
25023 0 0 0 3 0x14200 bored crynlk
22169 0 0 0 3 0x14200 bored crypto
36153 0 0 0 3 0x14200 pftm pfpurge
64902 0 0 0 3 0x14200 mmctsk sdmmc0
78312 0 0 0 3 0x14200 usbtsk usbtask
11358 0 0 0 3 0x14200 usbatsk usbatsk
15331 0 0 0 3 0x14200 bored i915
36247 0 0 0 3 0x40014200 acpi0 acpi0
14944 0 0 0 3 0x40014200 idle1
23017 0 0 0 3 0x14200 bored sensors
47311 0 0 0 3 0x14200 bored softnet
31150 0 0 0 3 0x14200 bored systqmp
91648 0 0 0 3 0x14200 bored systq
83241 0 0 0 3 0x40014200 bored softclock
38600 0 0 0 7 0x40014200 idle0
90688 0 0 0 3 0x14200 bored sbar
1 0 1 0 3 0x82 wait init
0 -1 0 0 3 0x10200 scheduler swapper
ddb{1}> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
987369 VM pages: 21751 active, 1922 inactive, 8702 wired, 922449 free
(115310
zero)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
pages 0 anon, 0 vnode, 0 vtext
freemin=32912, free-target=43882, inactive-target=0, wired-max=329123
faults=881539, traps=134199, intrs=64384, ctxswitch=170562 fpuswitch=114
softint=14220, syscalls=513707, kmapent=19
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=7756(7756), anget(retries)=796606(0), amapcopy=52869
neighbor anon/obj pg=3373/48491, gets(lock/unlock)=23790/7756
cases: anon=788246, anoncow=8360, obj=20643, prcopy=3147, przero=58846
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0
swpages=1082800, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xffffffff8192ace0
ddb{1}> show bcstats
Current Buffer Cache status:
numbufs 4326 busymapped 0, delwri 15
kvaslots 6553 avail kva slots 6553
bufpages 17363, dirtypages 68
pendingreads 0, pendingwrites 0
ddb{1}> show panic
rw_enter: netlock locking against myself
ddb{1}> boot reboot
panic: rw_enter: netlock locking against myself
Stopped at Debugger+0x9: leave
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
if_downall() at if_downall+0x26
boot() at boot+0xe4
reboot() at reboot+0x30
db_boot_reboot_cmd() at db_boot_reboot_cmd+0x12
db_command() at db_command+0x13a
db_command_loop() at db_command_loop+0xb0
db_trap() at db_trap+0xe9
db_ktrap() at db_ktrap+0xc4
trap() at trap+0x15d
--- trap (number 1) ---
Debugger() at Debugger+0x9
panic() at panic+0xfe
end trace frame: 0xffff800032a7b990, count: 0
ddb{1}> boot reboot
panic: rw_enter: netlock locking against myself
Stopped at Debugger+0x9: leave
Debugger() at Debugger+0x9
panic() at panic+0xfe
rw_enter() at rw_enter+0x1f8
if_downall() at if_downall+0x26
boot() at boot+0xe4
reboot() at reboot+0x30
db_boot_reboot_cmd() at db_boot_reboot_cmd+0x12
db_command() at db_command+0x13a
db_command_loop() at db_command_loop+0xb0
db_trap() at db_trap+0xe9
db_ktrap() at db_ktrap+0xc4
trap() at trap+0x15d
--- trap (number 1) ---
Debugger() at Debugger+0x9
panic() at panic+0xfe
end trace frame: 0xffff800032a7b3f0, count: 0
ddb{1}>
ssh -vvv without left over socket (host-/usernames and subnet changed):
XXX ~$ ssh -vvv YYY
OpenSSH_7.4, LibreSSL 2.5.1
debug1: Reading configuration data /home/USERNAME/.ssh/config
debug1: /home/USERNAME/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket
"/home/USERNAME/nfs/home/USERNAME/XXX/ctl-USERNAME-YYY-22" does not exist
debug2: resolving "YYY" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to YYY [123.456.789.163] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/USERNAME/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to YYY:22 as 'USERNAME'
debug3: hostkeys_foreach: reading file "/home/USERNAME/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file
/home/USERNAME/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from YYY
debug3: order_hostkeyalgs: prefer hostkeyalgs:
[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms:
[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos:
[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc:
[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos:
[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos:
[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc:
[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos:
[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC:
<implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC:
<implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256
SHA256:FK0SA7s6R0fd7unDsmUny9TaKsYrwx8etN4gmjSgNtY
debug3: hostkeys_foreach: reading file "/home/USERNAME/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file
/home/USERNAME/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from YYY
debug3: hostkeys_foreach: reading file "/home/USERNAME/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file
/home/USERNAME/.ssh/known_hosts:5
debug3: load_hostkeys: loaded 1 keys from 123.456.789.163
debug1: Host 'YYY' is known and matches the ECDSA host key.
debug1: Found key in /home/USERNAME/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/USERNAME/.ssh/id_rsa (0x0)
debug2: key: /home/USERNAME/.ssh/id_dsa (0x0)
debug2: key: /home/USERNAME/.ssh/id_ecdsa (0x0)
debug2: key: /home/USERNAME/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info:
server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue:
publickey,password,keyboard-interactive
debug3: start over, passed a different list
publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/USERNAME/.ssh/id_rsa
debug3: no such identity: /home/USERNAME/.ssh/id_rsa: No such file or
directory
debug1: Trying private key: /home/USERNAME/.ssh/id_dsa
debug3: no such identity: /home/USERNAME/.ssh/id_dsa: No such file or
directory
debug1: Trying private key: /home/USERNAME/.ssh/id_ecdsa
debug3: no such identity: /home/USERNAME/.ssh/id_ecdsa: No such file or
directory
debug1: Trying private key: /home/USERNAME/.ssh/id_ed25519
debug3: no such identity: /home/USERNAME/.ssh/id_ed25519: No such file
or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue:
publickey,password,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred:
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
USERNAME@YYY's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to YYY ([123.456.789.163]:22).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path
/home/USERNAME/nfs/home/USERNAME/XXX/ctl-USERNAME-YYY-22.wwwWD9fq8Qhjt42S
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new
[/home/USERNAME/nfs/home/USERNAME/XXX/ctl-USERNAME-YYY-22]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug1: channel 1: new [client-session]
debug3: ssh_session2_open: channel_new: 1
debug2: channel 1: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: id
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected]
want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 1
debug2: channel 1: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 1: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 1
Last login: Tue Dec 27 14:53:23 2016 from 123.456.789.154
OpenBSD 6.0-current (GENERIC.MP) #73: Mon Dec 26 23:25:48 MST 2016
YYY ~$
YYY ~$ ^D
debug3: receive packet: type 96
debug2: channel 1: rcvd eof
debug2: channel 1: output open -> drain
debug2: channel 1: obuf empty
debug2: channel 1: close_write
debug2: channel 1: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 1 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 1 rtype [email protected] reply 0
debug2: channel 1: rcvd eow
debug2: channel 1: close_read
debug2: channel 1: input open -> closed
debug3: receive packet: type 97
debug2: channel 1: rcvd close
debug3: channel 1: will not send data after close
debug2: channel 1: almost dead
debug2: channel 1: gc: notify user
debug2: channel 1: gc: user detached
debug2: channel 1: send close
debug3: send packet: type 97
debug2: channel 1: is dead
debug2: channel 1: garbage collecting
debug1: channel 1: free: client-session, nchannels 2
debug3: channel 1: status: The following connections are open:
#1 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
debug3: send packet: type 1
debug1: channel 0: free:
/home/USERNAME/nfs/home/USERNAME/XXX/ctl-USERNAME-YYY-22, nchannels 1
ssh -vvv with left over socket:
XXX ~$ ssh -vvv YYY
OpenSSH_7.4, LibreSSL 2.5.1
debug1: Reading configuration data /home/USERNAME/.ssh/config
debug1: /home/USERNAME/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
>How-To-Repeat:
mount a NFS share
put this in your .ssh/config:
Host *
ControlPath /path/to/nfs/mount/ctl-%r-%h-%p
ControlMaster auto
ssh to any host
logout
panic
reboot, don't delete socket
ssh to any host
panic
>Fix:
<how to correct or work around the problem, if known (multiple lines)>
dmesg:
booting hd0a:/bsd: 6979304+2212872+258624+0+765952
[72+710280+477696]=0xae2350
entry point at 0x1001000 [7205c766, 34000004, 24448b12, 1240a304]
[ using 1188688 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2016 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 6.0-current (GENERIC.MP) #73: Mon Dec 26 23:25:48 MST 2016
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4166717440 (3973MB)
avail mem = 4035776512 (3848MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (80 entries)
bios0: vendor LENOVO version "6FET93WW (3.23 )" date 10/12/2012
bios0: LENOVO 40613VG
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT ECDT APIC MCFG HPET SLIC BOOT ASF! SSDT
TCPA DMAR SSDT SSDT SSDT
acpi0: wakeup devices LID_(S3) SLPB(S3) UART(S3) IGBE(S4) EXP0(S4)
EXP1(S4) EXP2(S4) EXP3(S4) EXP4(S4) PCI1(S4) USB0(S3) USB3(S3) USB5(S3)
EHC0(S3) EHC1(S3) HDEF(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpiec0 at acpi0
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM)2 Duo CPU T9800 @ 2.93GHz, 2926.39 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
cpu0: 6MB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 7 var ranges, 88 fixed ranges
cpu0: apic clock running at 265MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2.1.3, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM)2 Duo CPU T9800 @ 2.93GHz, 2925.99 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
cpu1: 6MB 64b/line 16-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (AGP_)
acpiprt2 at acpi0: bus 2 (EXP0)
acpiprt3 at acpi0: bus 3 (EXP1)
acpiprt4 at acpi0: bus 4 (EXP2)
acpiprt5 at acpi0: bus 5 (EXP3)
acpiprt6 at acpi0: bus 13 (EXP4)
acpiprt7 at acpi0: bus 21 (PCI1)
acpicpu0 at acpi0: !C3(100@57 mwait.3@0x30), !C2(500@1 mwait.1@0x10),
C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: !C3(100@57 mwait.3@0x30), !C2(500@1 mwait.1@0x10),
C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PUBS, resource for USB0, USB3, USB5, EHC0, EHC1
acpitz0 at acpi0: critical temperature is 127 degC
acpitz1 at acpi0: critical temperature is 100 degC
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: SLPB
"PNP0303" at acpi0 not configured
"IBM0057" at acpi0 not configured
"PNP0501" at acpi0 not configured
"PNP0400" at acpi0 not configured
"INTC0102" at acpi0 not configured
acpibat0 at acpi0: BAT0 model "42T4619" serial 7164 type LION oem "SANYO"
acpibat1 at acpi0: BAT1 model "43R8891" serial 59 type LiP oem "SONY"
acpiac0 at acpi0: AC unit online
acpithinkpad0 at acpi0
"IBM0079" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpidock0 at acpi0: GDCK docked (15)
acpivideo0 at acpi0: VID_
acpivout0 at acpivideo0: LCD0
acpivideo1 at acpi0: VID_
cpu0: Enhanced SpeedStep 2926 MHz: speeds: 2934, 2933, 2133, 1600, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel GM45 Host" rev 0x07
inteldrm0 at pci0 dev 2 function 0 "Intel GM45 Video" rev 0x07
drm0 at inteldrm0
intagp0 at inteldrm0
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0: msi
inteldrm0: 1920x1200
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel GM45 Video" rev 0x07 at pci0 dev 2 function 1 not configured
"Intel GM45 HECI" rev 0x07 at pci0 dev 3 function 0 not configured
em0 at pci0 dev 25 function 0 "Intel ICH9 IGP M AMT" rev 0x03: msi,
address 00:22:68:15:76:5d
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 1 int 20
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 1 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 1 int 22
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 1 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi
azalia0: codecs: Conexant CX20561, Conexant/0x2c06, using Conexant CX20561
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x03: msi
pci1 at ppb0 bus 2
ppb1 at pci0 dev 28 function 1 "Intel 82801I PCIE" rev 0x03: msi
pci2 at ppb1 bus 3
iwn0 at pci2 dev 0 function 0 "Intel WiFi Link 5300" rev 0x00: msi, MIMO
3T3R, MoW, address 00:21:6a:6f:cd:da
ppb2 at pci0 dev 28 function 2 "Intel 82801I PCIE" rev 0x03: msi
pci3 at ppb2 bus 4
"Intel Turbo Memory" rev 0x11 at pci3 dev 0 function 0 not configured
ppb3 at pci0 dev 28 function 3 "Intel 82801I PCIE" rev 0x03: msi
pci4 at ppb3 bus 5
ppb4 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x03: msi
pci5 at ppb4 bus 13
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 1 int 16
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 1 int 17
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 1 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 1 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0x93
pci6 at ppb5 bus 21
cbb0 at pci6 dev 0 function 0 "Ricoh 5C476 CardBus" rev 0xba: apic 1 int 16
"Ricoh 5C832 Firewire" rev 0x04 at pci6 dev 0 function 1 not configured
sdhc0 at pci6 dev 0 function 2 "Ricoh 5C822 SD/MMC" rev 0x21: apic 1 int 18
sdhc0: SDHC 1.0, 33 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed
"Ricoh 5C843 MMC" rev 0x11 at pci6 dev 0 function 3 not configured
"Ricoh 5C592 Memory Stick" rev 0x11 at pci6 dev 0 function 4 not configured
"Ricoh 5C852 xD" rev 0x11 at pci6 dev 0 function 5 not configured
cardslot0 at cbb0 slot 0 flags 0
cardbus0 at cardslot0: bus 22 device 0 cacheline 0x10, lattimer 0xb0
pcmcia0 at cardslot0
pcib0 at pci0 dev 31 function 0 "Intel 82801IEM LPC" rev 0x03
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x03: msi, AHCI 1.2
ahci0: port 0: 3.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, Crucial_CT500MX2, MU02> SCSI3
0/direct fixed naa.500a07511100fc1e
sd0: 476940MB, 512 bytes/sector, 976773168 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x03: apic 1
int 23
iic0 at ichiic0
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
aps0 at isa0 port 0x1600/31
vmm0 at mainbus0: VMX
uhub8 at uhub0 port 5 configuration 1 interface 0 "IBM product 0x4486"
rev 2.00/0.01 addr 2
uvideo0 at uhub0 port 6 configuration 1 interface 0 "Lenovo product
0x4807" rev 2.00/31.34 addr 3
video0 at uvideo0
ugen0 at uhub3 port 1 "AuthenTec Fingerprint Sensor" rev 2.00/17.03 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (0f12031d647a9d55.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
usbdevs:
Controller /dev/usb0:
addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
port 3 powered
port 4 powered
port 5 addr 2: high speed, self powered, config 1, product
0x4486(0x4486), IBM(0x04b3), rev 0.01
port 1 powered
port 2 powered
port 3 powered
port 4 powered
port 5 powered
port 6 powered
port 7 powered
port 6 addr 3: high speed, power 100 mA, config 1, product
0x4807(0x4807), Lenovo(0x17ef), rev 31.34
Controller /dev/usb1:
addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
port 3 powered
port 4 powered
port 5 powered
port 6 powered
Controller /dev/usb2:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
Controller /dev/usb3:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 addr 2: full speed, power 100 mA, config 1, Fingerprint
Sensor(0x2810), AuthenTec(0x08ff), rev 17.03
port 2 powered
Controller /dev/usb4:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
Controller /dev/usb5:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
Controller /dev/usb6:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
Controller /dev/usb7:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
port 1 powered
port 2 powered
