in6_mcast: in6_joingroup attempts to acquire IN6_MULTI_LOCK when sleeping prohibited
I have seen this on boot of my laptop. It appears that in6_joingroup() was called in netisr_dispatch_src codepath, and it tried to acquire IN6_MULTI_LOCK(), which happened to sleep because we failed to acquire the sx, thus triggered the panic. === panic: sleepq_add: td 0xf8000ecd6000 to sleep on wchan 0x81dedfe0 with sleeping prohibited #1 0x80bbff90 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:479 #2 0x80bc03e6 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:908 #3 0x80bc0143 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:835 #4 0x80c1a2bf in sleepq_add (wchan=0x81dedfe0, lock=0x0, wmesg=0x8110f331 "in6_multi_sx", flags=3, queue=0) at /usr/src/sys/kern/subr_sleepqueue.c:318 #5 0x80bc9ce4 in _sx_xlock_hard (sx=0x81dedfe0, x=18446735277856440320, opts=, file=, line=) at /usr/src/sys/kern/kern_sx.c:841 #6 0x80bc983f in _sx_xlock (sx=0x81dedfe0, opts=0, file=0x8113a568 "/usr/src/sys/netinet6/in6_mcast.c", line=1185) at /usr/src/sys/kern/kern_sx.c:325 #7 0x80e17dd1 in in6_joingroup (ifp=0xf80003b99800, mcaddr=0xfe00e1612e58, imf=, pinm=0xf80019e17300, delay=2) at /usr/src/sys/netinet6/in6_mcast.c:1185 #8 0x80e0fa72 in in6_update_ifa (ifp=0xf80003b99800, ifra=, ia=, flags=) at /usr/src/sys/netinet6/in6.c:752 #9 0x80e374c5 in nd6_ra_input (m=0xf800191d4a00, off=, icmp6len=) at /usr/src/sys/netinet6/nd6_rtr.c:2274 #10 0x80e096d5 in icmp6_input (mp=, offp=0xfe00e161335c, proto=) at /usr/src/sys/netinet6/icmp6.c:767 #11 0x80e22dff in ip6_input (m=0xf800191d4a00) at /usr/src/sys/netinet6/ip6_input.c:963 #12 0x80ceff11 in netisr_dispatch_src (proto=6, source=0, m=0xf800191d4a00) at /usr/src/sys/net/netisr.c:1127 #13 0x80cd399e in ether_demux (ifp=0xf80003b99800, m=) at /usr/src/sys/net/if_ethersubr.c:916 #14 0x80cd4f88 in ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:705 #15 0x80ceff11 in netisr_dispatch_src (proto=5, source=0, m=0xf800191d4a00) at /usr/src/sys/net/netisr.c:1127 #16 0x80cd3e8d in ether_input (ifp=0xf8000397a800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:824 #17 0x80d429f0 in sta_input (ni=, m=0xf800191d4a00, rxs=, rssi=, nf=) at /usr/src/sys/net80211/ieee80211_sta.c:891 #18 0x80d1ec2a in ieee80211_input_mimo (ni=0xfe0106c66000, m=0xf800191d4a00) at /usr/src/sys/net80211/ieee80211_input.c:101 #19 0x848e55a2 in iwm_mvm_rx_rx_mpdu (sc=0xfe00e2c0, m=0xf800191d4a00, offset=, stolen=) at /usr/src/sys/dev/iwm/if_iwm.c:3245 #20 0x848e3fee in iwm_intr (arg=) at /usr/src/sys/dev/iwm/if_iwm.c:5151 signature.asc Description: OpenPGP digital signature
RE: panic on invalid ifp pointer in iflib drivers
> -Original Message- > From: John Baldwin > Sent: Thursday, October 17, 2019 4:34 PM > To: Keller, Jacob E ; freebsd-net@freebsd.org > Cc: sh...@llnw.com; Joyner, Eric > Subject: Re: panic on invalid ifp pointer in iflib drivers > > On 10/17/19 4:22 PM, Keller, Jacob E wrote: > > Hmm.. now that I look at that more closely I think it's a separate issue. > > This may just be the rcvif issue where we don't reference count the ifp we > store in rcvif in mbufs? That was my reaction to your first e-mail except > that you said it wasn't reproducible on some other drivers. I wonder if > other drivers would also provoke this if you just ran them in a detach/attach > loop long enough. > This is almost certainly the rcvif issue based on where it panics. I never saw this panic before. I 've also only reproduced it on a kernel with INVARIANTS and memguard set to ifnet. Thanks, Jake ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: panic on invalid ifp pointer in iflib drivers
On 10/17/19 4:22 PM, Keller, Jacob E wrote: > Hmm.. now that I look at that more closely I think it's a separate issue. This may just be the rcvif issue where we don't reference count the ifp we store in rcvif in mbufs? That was my reaction to your first e-mail except that you said it wasn't reproducible on some other drivers. I wonder if other drivers would also provoke this if you just ran them in a detach/attach loop long enough. -- John Baldwin ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: panic on invalid ifp pointer in iflib drivers
> -Original Message- > From: Keller, Jacob E > Sent: Thursday, October 17, 2019 4:22 PM > To: John Baldwin ; freebsd-net@freebsd.org > Cc: sh...@llnw.com; Joyner, Eric > Subject: RE: panic on invalid ifp pointer in iflib drivers > > > -Original Message- > > From: John Baldwin > > Sent: Thursday, October 17, 2019 9:31 AM > > To: Keller, Jacob E ; freebsd-net@freebsd.org > > Cc: sh...@llnw.com; Joyner, Eric > > Subject: Re: panic on invalid ifp pointer in iflib drivers > > > > Nominally, ifnet drivers should call ether_ifdetach first to remove public > > references to the ifnet and only call their stop routine after that has > > returned. > > This ensures any open if_ioctl invocations have completed, etc. before the > > stop routine is invoked. Otherwise you are open to a race where the > > inteface > > can be upped via an ioctl after you have stopped the hardware. > > > > Any other references to the ifnet via eventhandlers, etc. should also be > > deregistered before calling the stop routine. > > Looks like iflib moved this much later when we refactored to add a shared > function to deregister VLAN handlers... > > > > > After the hardware is stopped, interrupt handlers should be torn down and > > callouts > > and tasks drained to ensure there are no other references to the ifp > > outside of > > the thread running detach. > > > > After that you can release device resources, destroy mutexes, free the ifp, > > etc. > > Note that drivers have to be prepared for ether_ifdetach to invoke if_ioctl > > (e.g. > > when detaching bpf), but of the drivers I've looked at this has generally > > been a > > non-issue. > > > > It sounds like iflib should be doing the detach before calling iflib_stop. > > > > I tested a patch that moved ether_ifdetach above the call to iflib_stop. > > This seems to have made the issue significantly harder to reproduce, but after > multiple attach/detach cycles with IPv6 traffic: (INVARIANTS and WITNESS are > enabled, as well as meguard protecting ifnet) > > Unread portion of the kernel message buffer: > Kernel page fault with the following non-sleepable locks held: > exclusive sleep mutex ip6qlock (ip6qlock) r = 0 (0xfe7aa848) locked @ > /usr/src/sys/netinet6/frag6.c:849 > shared rw vnet_rwlock (vnet_rwlock) r = 0 (0x820be700) locked @ > /usr/src/sys/netinet6/frag6.c:845 > stack backtrace: > #0 0x80bb6f83 at witness_debugger+0x73 > #1 0x80bb7fa2 at witness_warn+0x442 > #2 0x8108a0f3 at trap_pfault+0x53 > #3 0x810896e4 at trap+0x2b4 > #4 0x8106201c at calltrap+0x8 > #5 0x80d8c07a at icmp6_error+0x4aa > #6 0x80d8b30e at frag6_freef+0x10e > #7 0x80d8b551 at frag6_slowtimo+0x111 > #8 0x80bdcda4 at pfslowtimo+0x54 > #9 0x80b65bdf at softclock_call_cc+0x13f > #10 0x80b65f9c at softclock+0x7c > #11 0x80b0f857 at ithread_loop+0x187 > #12 0x80b0c4a4 at fork_exit+0x84 > #13 0x8106305e at fork_trampoline+0xe > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xfe825dd8 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0x80d8c5b2 > stack pointer = 0x28:0xfe1fc28c6ff0 > frame pointer = 0x28:0xfe1fc28c7090 > code segment= base 0x0, limit 0xf, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags= interrupt enabled, resume, IOPL = 0 > current process = 12 (swi4: clock (0)) > trap number = 12 > panic: page fault > cpuid = 0 > time = 1571354026 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfe1fc28c6cb0 > vpanic() at vpanic+0x19d/frame 0xfe1fc28c6d00 > panic() at panic+0x43/frame 0xfe1fc28c6d60 > trap_fatal() at trap_fatal+0x39c/frame 0xfe1fc28c6dc0 > trap_pfault() at trap_pfault+0x62/frame 0xfe1fc28c6e10 > trap() at trap+0x2b4/frame 0xfe1fc28c6f20 > calltrap() at calltrap+0x8/frame 0xfe1fc28c6f20 > --- trap 0xc, rip = 0x80d8c5b2, rsp = 0xfe1fc28c6ff0, rbp = > 0xfe1fc28c7090 --- > icmp6_reflect() at icmp6_reflect+0x242/frame 0xfe1fc28c7090 > icmp6_error() at icmp6_error+0x4aa/frame 0xfe1fc28c70e0 > frag6_freef() at frag6_freef+0x10e/frame 0xfe1fc28c7130 > frag6_slowtimo() at frag6_slowtimo+0x111/frame 0xfe1fc28c7180 > pfslowtimo() at pfslowtimo+0x54/frame 0xfe1fc28c71b0 > softclock_call_cc() at softclock_call_cc+0x13f/frame 0xfe1fc28c7260 > softclock() at softclock+0x7c/frame 0xfe1fc28c7290 > ithread_loop() at ithread_loop+0x187/frame 0xfe1fc28c72f0 > fork_exit() at fork_exit+0x84/frame 0xfe1fc28c7330 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe1fc28c7330 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > > > Hmm.. now that I look at that more closely I think it's a
RE: panic on invalid ifp pointer in iflib drivers
> -Original Message- > From: John Baldwin > Sent: Thursday, October 17, 2019 9:31 AM > To: Keller, Jacob E ; freebsd-net@freebsd.org > Cc: sh...@llnw.com; Joyner, Eric > Subject: Re: panic on invalid ifp pointer in iflib drivers > > Nominally, ifnet drivers should call ether_ifdetach first to remove public > references to the ifnet and only call their stop routine after that has > returned. > This ensures any open if_ioctl invocations have completed, etc. before the > stop routine is invoked. Otherwise you are open to a race where the inteface > can be upped via an ioctl after you have stopped the hardware. > > Any other references to the ifnet via eventhandlers, etc. should also be > deregistered before calling the stop routine. Looks like iflib moved this much later when we refactored to add a shared function to deregister VLAN handlers... > > After the hardware is stopped, interrupt handlers should be torn down and > callouts > and tasks drained to ensure there are no other references to the ifp outside > of > the thread running detach. > > After that you can release device resources, destroy mutexes, free the ifp, > etc. > Note that drivers have to be prepared for ether_ifdetach to invoke if_ioctl > (e.g. > when detaching bpf), but of the drivers I've looked at this has generally > been a > non-issue. > > It sounds like iflib should be doing the detach before calling iflib_stop. > I tested a patch that moved ether_ifdetach above the call to iflib_stop. This seems to have made the issue significantly harder to reproduce, but after multiple attach/detach cycles with IPv6 traffic: (INVARIANTS and WITNESS are enabled, as well as meguard protecting ifnet) Unread portion of the kernel message buffer: Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ip6qlock (ip6qlock) r = 0 (0xfe7aa848) locked @ /usr/src/sys/netinet6/frag6.c:849 shared rw vnet_rwlock (vnet_rwlock) r = 0 (0x820be700) locked @ /usr/src/sys/netinet6/frag6.c:845 stack backtrace: #0 0x80bb6f83 at witness_debugger+0x73 #1 0x80bb7fa2 at witness_warn+0x442 #2 0x8108a0f3 at trap_pfault+0x53 #3 0x810896e4 at trap+0x2b4 #4 0x8106201c at calltrap+0x8 #5 0x80d8c07a at icmp6_error+0x4aa #6 0x80d8b30e at frag6_freef+0x10e #7 0x80d8b551 at frag6_slowtimo+0x111 #8 0x80bdcda4 at pfslowtimo+0x54 #9 0x80b65bdf at softclock_call_cc+0x13f #10 0x80b65f9c at softclock+0x7c #11 0x80b0f857 at ithread_loop+0x187 #12 0x80b0c4a4 at fork_exit+0x84 #13 0x8106305e at fork_trampoline+0xe Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xfe825dd8 fault code = supervisor read data, page not present instruction pointer = 0x20:0x80d8c5b2 stack pointer = 0x28:0xfe1fc28c6ff0 frame pointer = 0x28:0xfe1fc28c7090 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock (0)) trap number = 12 panic: page fault cpuid = 0 time = 1571354026 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe1fc28c6cb0 vpanic() at vpanic+0x19d/frame 0xfe1fc28c6d00 panic() at panic+0x43/frame 0xfe1fc28c6d60 trap_fatal() at trap_fatal+0x39c/frame 0xfe1fc28c6dc0 trap_pfault() at trap_pfault+0x62/frame 0xfe1fc28c6e10 trap() at trap+0x2b4/frame 0xfe1fc28c6f20 calltrap() at calltrap+0x8/frame 0xfe1fc28c6f20 --- trap 0xc, rip = 0x80d8c5b2, rsp = 0xfe1fc28c6ff0, rbp = 0xfe1fc28c7090 --- icmp6_reflect() at icmp6_reflect+0x242/frame 0xfe1fc28c7090 icmp6_error() at icmp6_error+0x4aa/frame 0xfe1fc28c70e0 frag6_freef() at frag6_freef+0x10e/frame 0xfe1fc28c7130 frag6_slowtimo() at frag6_slowtimo+0x111/frame 0xfe1fc28c7180 pfslowtimo() at pfslowtimo+0x54/frame 0xfe1fc28c71b0 softclock_call_cc() at softclock_call_cc+0x13f/frame 0xfe1fc28c7260 softclock() at softclock+0x7c/frame 0xfe1fc28c7290 ithread_loop() at ithread_loop+0x187/frame 0xfe1fc28c72f0 fork_exit() at fork_exit+0x84/frame 0xfe1fc28c7330 fork_trampoline() at fork_trampoline+0xe/frame 0xfe1fc28c7330 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic Hmm.. now that I look at that more closely I think it's a separate issue. > -- > John Baldwin ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: panic on invalid ifp pointer in iflib drivers
> -Original Message- > From: John Baldwin > Sent: Thursday, October 17, 2019 9:31 AM > To: Keller, Jacob E ; freebsd-net@freebsd.org > Cc: sh...@llnw.com; Joyner, Eric > Subject: Re: panic on invalid ifp pointer in iflib drivers > > On 10/16/19 2:16 PM, Keller, Jacob E wrote: > > Hi, > > > > I'm investigating an issue on the iflib ixl driver in 11.3-RELEASE as well > > as 12- > RELEASE. We found a panic in that occurs if SCTP/IPv6 traffic is being > transmitted > while the device is detached: > > > > Fatal trap 12: page fault while in kernel mode > > cpuid = 0; apic id = 00 > > fault virtual address = 0xfe411e38 > > fault code = supervisor read data, page not present > > instruction pointer = 0x20:0x80c84700 > > stack pointer = 0x28:0xfe2f4351b600 > > frame pointer = 0x28:0xfe2f4351b650 > > code segment= base 0x0, limit 0xf, type 0x1b > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags= interrupt enabled, resume, IOPL = 0 > > current process = 12 (swi4: clock (0)) > > trap number = 12 > > panic: page fault > > cpuid = 0 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfe2f4351b2c0 > > vpanic() at vpanic+0x17e/frame 0xfe2f4351b320 > > panic() at panic+0x43/frame 0xfe2f4351b380 > > trap_fatal() at trap_fatal+0x369/frame 0xfe2f4351b3d0 > > trap_pfault() at trap_pfault+0x62/frame 0xfe2f4351b420 > > trap() at trap+0x2b3/frame 0xfe2f4351b530 > > calltrap() at calltrap+0x8/frame 0xfe2f4351b530 > > --- trap 0xc, rip = 0x80c84700, rsp = 0xfe2f4351b600, rbp = > 0xfe2f4351b650 --- > > in6_selecthlim() at in6_selecthlim+0x20/frame 0xfe2f4351b650 > > sctp_lowlevel_chunk_output() at sctp_lowlevel_chunk_output+0xeb2/frame > 0xfe2f4351b790 > > sctp_chunk_output() at sctp_chunk_output+0x68c/frame 0xfe2f4351c110 > > sctp_timeout_handler() at sctp_timeout_handler+0x2d8/frame > 0xfe2f4351c180 > > softclock_call_cc() at softclock_call_cc+0x15b/frame 0xfe2f4351c230 > > softclock() at softclock+0x7c/frame 0xfe2f4351c260 > > intr_event_execute_handlers() at intr_event_execute_handlers+0x9a/frame > 0xfe2f4351c2a0 > > ithread_loop() at ithread_loop+0xb7/frame 0xfe2f4351c2f0 > > fork_exit() at fork_exit+0x84/frame 0xfe2f4351c330 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe2f4351c330 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > KDB: enter: panic > > > > > > From what I've gathered so far, it appears that the issue is a > > use-after-free > where the SCTP stack gets an ifp pointer that's no longer valid. We've > reproduced > this issue on multiple iflib-based drivers, including ixl and the recently > published > ice driver code (available on phabricator). > > > > Additionally, we cannot reproduce it on legacy-stack drivers for ixl, or a > mellanox 100G board we have. This leads me to believe that it's an issue in > iflib > rather than in the specific device drivers. > > > > I am not sure exactly what's going wrong here... anyone have suggestions? I > thought it might be an issue of when ether_ifdetach is called. That function > is > supposed to clear all of the pre-existing routes from the route entry list. > I'm > thinking maybe somehow a route gets added after ether_ifdetach is called. > > > > In the iflib_device_deregister function, ether_ifdetach is called just after > iflib_stop, (which would call a device's if_stop routine), and then the task > queues > are shutdown, a driver's ifdi_detach handler is called, and the ifp is free'd > at the > end. In the ixl legacy driver, ether_ifdetach is called prior to the stop > routine. > However, in the mlx5 driver, it's called after a call to close_locked()... > > > > So I'm really not sure exactly what could cause a stale ifp pointer to get > > into the > route entry list. > > Nominally, ifnet drivers should call ether_ifdetach first to remove public > references to the ifnet and only call their stop routine after that has > returned. > This ensures any open if_ioctl invocations have completed, etc. before the > stop routine is invoked. Otherwise you are open to a race where the inteface > can be upped via an ioctl after you have stopped the hardware. > So we should (a) move ether_ifdetach before the stop, and... > Any other references to the ifnet via eventhandlers, etc. should also be > deregistered before calling the stop routine. > (b) make sure all of the event handlers are moved before the stop too. > After the hardware is stopped, interrupt handlers should be torn down and > callouts > and tasks drained to ensure there are no other references to the ifp outside > of > the thread running detach. > > After that you can release device resources, destroy mutexes, free the ifp, > etc. > Note that drivers have to be prepared for ether_ifdetach to invoke if_ioctl >
Re: panic on invalid ifp pointer in iflib drivers
On 10/16/19 2:16 PM, Keller, Jacob E wrote: > Hi, > > I'm investigating an issue on the iflib ixl driver in 11.3-RELEASE as well as > 12-RELEASE. We found a panic in that occurs if SCTP/IPv6 traffic is being > transmitted while the device is detached: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xfe411e38 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0x80c84700 > stack pointer = 0x28:0xfe2f4351b600 > frame pointer = 0x28:0xfe2f4351b650 > code segment= base 0x0, limit 0xf, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags= interrupt enabled, resume, IOPL = 0 > current process = 12 (swi4: clock (0)) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe2f4351b2c0 > vpanic() at vpanic+0x17e/frame 0xfe2f4351b320 > panic() at panic+0x43/frame 0xfe2f4351b380 > trap_fatal() at trap_fatal+0x369/frame 0xfe2f4351b3d0 > trap_pfault() at trap_pfault+0x62/frame 0xfe2f4351b420 > trap() at trap+0x2b3/frame 0xfe2f4351b530 > calltrap() at calltrap+0x8/frame 0xfe2f4351b530 > --- trap 0xc, rip = 0x80c84700, rsp = 0xfe2f4351b600, rbp = > 0xfe2f4351b650 --- > in6_selecthlim() at in6_selecthlim+0x20/frame 0xfe2f4351b650 > sctp_lowlevel_chunk_output() at sctp_lowlevel_chunk_output+0xeb2/frame > 0xfe2f4351b790 > sctp_chunk_output() at sctp_chunk_output+0x68c/frame 0xfe2f4351c110 > sctp_timeout_handler() at sctp_timeout_handler+0x2d8/frame 0xfe2f4351c180 > softclock_call_cc() at softclock_call_cc+0x15b/frame 0xfe2f4351c230 > softclock() at softclock+0x7c/frame 0xfe2f4351c260 > intr_event_execute_handlers() at intr_event_execute_handlers+0x9a/frame > 0xfe2f4351c2a0 > ithread_loop() at ithread_loop+0xb7/frame 0xfe2f4351c2f0 > fork_exit() at fork_exit+0x84/frame 0xfe2f4351c330 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe2f4351c330 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > > > From what I've gathered so far, it appears that the issue is a use-after-free > where the SCTP stack gets an ifp pointer that's no longer valid. We've > reproduced this issue on multiple iflib-based drivers, including ixl and the > recently published ice driver code (available on phabricator). > > Additionally, we cannot reproduce it on legacy-stack drivers for ixl, or a > mellanox 100G board we have. This leads me to believe that it's an issue in > iflib rather than in the specific device drivers. > > I am not sure exactly what's going wrong here... anyone have suggestions? I > thought it might be an issue of when ether_ifdetach is called. That function > is supposed to clear all of the pre-existing routes from the route entry > list. I'm thinking maybe somehow a route gets added after ether_ifdetach is > called. > > In the iflib_device_deregister function, ether_ifdetach is called just after > iflib_stop, (which would call a device's if_stop routine), and then the task > queues are shutdown, a driver's ifdi_detach handler is called, and the ifp is > free'd at the end. In the ixl legacy driver, ether_ifdetach is called prior > to the stop routine. However, in the mlx5 driver, it's called after a call to > close_locked()... > > So I'm really not sure exactly what could cause a stale ifp pointer to get > into the route entry list. Nominally, ifnet drivers should call ether_ifdetach first to remove public references to the ifnet and only call their stop routine after that has returned. This ensures any open if_ioctl invocations have completed, etc. before the stop routine is invoked. Otherwise you are open to a race where the inteface can be upped via an ioctl after you have stopped the hardware. Any other references to the ifnet via eventhandlers, etc. should also be deregistered before calling the stop routine. After the hardware is stopped, interrupt handlers should be torn down and callouts and tasks drained to ensure there are no other references to the ifp outside of the thread running detach. After that you can release device resources, destroy mutexes, free the ifp, etc. Note that drivers have to be prepared for ether_ifdetach to invoke if_ioctl (e.g. when detaching bpf), but of the drivers I've looked at this has generally been a non-issue. It sounds like iflib should be doing the detach before calling iflib_stop. -- John Baldwin ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ipsec on multicore VM
09.10.2019 2:05, Victor Gamov wrote: > I have FreeBSD 11.2-STABLE #0 r343863 VM with 2 CPU and vxnet3 NIC. This host > uses many if_ipsec and strongswan-5.7.2 to make site-to-site ipsec > connections. > > When I use `tcpdump -nn -i src and esp` then I got > many reordered IPsec packets. > > Does tcpdump give me a real picture and I have reordering somewhere "on the > wire" or packets may be reordered due more then one CPU read packets from NIC > ? You may easily verify your suspiction disabling SMP inside the guest system temporary: nextboot -k kernel echo kern.smp.disabled=1 >> /boot/nextboot.conf shutdown -r now This way, the system will perform one-time boot with all cores but one disabled. Should it experience any problems booting this way, another reset of the VM will boot it normally, otherwise try running tcpdump while single CPU is used by kernel. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
