[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Vincenzo Maffione  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|In Progress |Closed

--- Comment #10 from Vincenzo Maffione  ---
This has been fixed by r343344.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Vincenzo Maffione  changed:

   What|Removed |Added

 Status|Open|In Progress

--- Comment #9 from Vincenzo Maffione  ---
Hi Aleksandr,
  Thanks a lot for the debugging and the fix.
Your patch looks the correct solution to this problem.
I tried to make it more compact here https://reviews.freebsd.org/D18876
What do you think? Could you test it?

Thanks,
  Vincenzo

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Mark Linimon  changed:

   What|Removed |Added

   Keywords||patch

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

--- Comment #8 from Harald Schmalzbauer  ---
Just a quick note that I'm unfortunately busy with completely different things
:-(
And for the records, here's Vincenzo's approach which I usesd (successfully as
far as I remember) back in 11.2:
+++ src/sys/dev/netmap/netmap_freebsd.c 2017-06-17 11:22:53.685116000 +0200
@@ -1374,7 +1374,7 @@
kn->kn_fop = (ev == EVFILT_WRITE) ?
_wfiltops : _rfiltops;
kn->kn_hook = priv;
-   knlist_add(>si.si_note, kn, 1);
+   knlist_add(>si.si_note, kn, 0);
// XXX unlock(priv)
ND("register %p %s td %p priv %p kn %p np_nifp %p kn_fp/fpop %s",
na, na->ifp->if_xname, curthread, priv, kn,

No skills to contribute and no time to test at the moment, sorry.

Thanks for taking care, will get back to it :-)

-harry

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

--- Comment #7 from Aleksandr Fedorov  ---
bhyve trace with userstack:

dtrace: script './nm_os_selwakeup.d' matched 1 probe
CPU IDFUNCTION:NAME
 20  39521nm_os_selwakeup:entry
  kernel`netmap_notify+0x1d
  kernel`nm_vale_flush+0xb70
  kernel`netmap_vale_vp_txsync+0x51d
  kernel`netmap_ioctl+0x4e1
  kernel`freebsd_netmap_ioctl+0x88
  kernel`devfs_ioctl+0xca
  kernel`VOP_IOCTL_APV+0x63
  kernel`vn_ioctl+0x124
  kernel`devfs_ioctl_f+0x1f
  kernel`kern_ioctl+0x29b
  kernel`sys_ioctl+0x15d
  kernel`amd64_syscall+0x276
  kernel`0x8107a81d

  libc.so.7`ioctl+0xa
  bhyve`pci_vtnet_netmap_tx+0x91
  bhyve`pci_vtnet_proctx+0x1a4
  bhyve`pci_vtnet_tx_thread+0x1ce
  libthr.so.3`0x800689756


 19  39521nm_os_selwakeup:entry
  kernel`netmap_notify+0x1d
  kernel`netmap_poll+0x5e6
  kernel`netmap_knrw+0x6f
  kernel`kqueue_kevent+0x56d
  kernel`kern_kevent_fp+0x97
  kernel`kern_kevent+0x9f
  kernel`kern_kevent_generic+0x70
  kernel`sys_kevent+0x61
  kernel`amd64_syscall+0x276
  kernel`0x8107a81d

  libc.so.7`_kevent+0xa
  bhyve`mevent_dispatch+0x2d4
  bhyve`main+0x848
  bhyve`_start+0x11b
  ld-elf.so.1`0x80029c000

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

--- Comment #6 from Aleksandr Fedorov  ---
This patch fix the panic:

Index: sys/dev/netmap/netmap_freebsd.c
===
--- sys/dev/netmap/netmap_freebsd.c (revision 342550)
+++ sys/dev/netmap/netmap_freebsd.c (working copy)
@@ -1321,7 +1321,12 @@
/* use a non-zero hint to tell the notification from the
 * call done in kqueue_scan() which uses 0
 */
-   KNOTE_UNLOCKED(>si.si_note, 0x100 /* notification */);
+
+   if (mtx_owned(>m)) {
+   KNOTE_LOCKED(>si.si_note, 0x100 /* notification */);
+   } else {
+   KNOTE_UNLOCKED(>si.si_note, 0x100 /* notification */);
+   }
 }

 void

It seems there are several paths to call the function nm_os_selwakeup():

Trace from bhyve:
dtrace: script './nm_os_selwakeup.d' matched 1 probe
CPU IDFUNCTION:NAME
 18  39521nm_os_selwakeup:entry
  kernel`netmap_notify+0x1d
  kernel`nm_vale_flush+0xb70
  kernel`netmap_vale_vp_txsync+0x51d
  kernel`netmap_ioctl+0x4e1
  kernel`freebsd_netmap_ioctl+0x88
  kernel`devfs_ioctl+0xca
  kernel`VOP_IOCTL_APV+0x63
  kernel`vn_ioctl+0x124
  kernel`devfs_ioctl_f+0x1f
  kernel`kern_ioctl+0x29b
  kernel`sys_ioctl+0x15d
  kernel`amd64_syscall+0x276
  kernel`0x8107a81d


 20  39521nm_os_selwakeup:entry
  kernel`netmap_notify+0x1d
  kernel`netmap_poll+0x5e6
  kernel`netmap_knrw+0x6f
  kernel`kqueue_kevent+0x56d
  kernel`kern_kevent_fp+0x97
  kernel`kern_kevent+0x9f
  kernel`kern_kevent_generic+0x70
  kernel`sys_kevent+0x61
  kernel`amd64_syscall+0x276
  kernel`0x8107a81d

The first does not hold the mutex ("nm_kn_lock"), the second does.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Aleksandr Fedorov  changed:

   What|Removed |Added

 CC||aleksandr.fedorov@itglobal.
   ||com

--- Comment #4 from Aleksandr Fedorov  ---
Created attachment 201116
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=201116=edit
kernel trace

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Vincenzo Maffione  changed:

   What|Removed |Added

 Status|New |Open
 CC||vmaffi...@freebsd.org

--- Comment #3 from Vincenzo Maffione  ---
I still have to setup a test environment for this bug, but please note that now
stable/11 contains the latest netmap code (as well as stable/12 and HEAD).

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

--- Comment #2 from Harald Schmalzbauer  ---
Like kib@ suggested in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206053, I already tried to
replacing 1 with 0 in the sys/dev/netmap/netmap_freebsd.c:netmap_kqfilter() at
the call to knlist_add(), which does indeed prevent the initial panic (when the
guest boots), but ends up in another panic short afterwards:

(quoting my github report)

panic: mutex nm_kn_lock owned at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:2176
cpuid = 6
KDB: stack backtrace:
#0 0x805cb1a7 at kdb_backtrace+0x67
#1 0x8058b0e6 at vpanic+0x186
#2 0x8058b163 at panic+0x43
#3 0x8056c216 at __mtx_assert+0xc6
now:
#4 0x80545219 at knote+0x39
#5 0x8041ebc7 at nm_os_selwakeup+0x87
#6 0x8041c94d at netmap_notify+0x1d
#7 0x8041c701 at netmap_poll+0x821
#8 0x8041f63c at netmap_knrw+0x6c
#9 0x805443d7 at kqueue_kevent+0x397

#10 0x80543fd6 at kern_kevent_fp+0x96
#11 0x80543eef at kern_kevent+0x9f
#12 0x80543cf8 at sys_kevent+0x138
#13 0x80881b5a at amd64_syscall+0x57a
#14 0x8086563b at Xfast_syscall+0xfb

#0  doadump (textdump=) at pcpu.h:222
#1  0x8058ab60 in kern_reboot (howto=260) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
#2  0x8058b120 in vpanic (fmt=, ap=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
#3  0x8058b163 in panic (fmt=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
#4  0x8056c216 in __mtx_assert (c=,
what=, file=, line=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_mutex.c:1013
#5  0x80545219 in knote (list=, hint=256,
lockflags=0) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:2034
#6  0x8041ebc7 in nm_os_selwakeup (si=0xfe00088b94c0) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/dev/netmap/netmap_freebsd.c:1259
#7  0x8041c94d in netmap_notify (kring=,
flags=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/dev/netmap/netmap.c:2777
#8  0x8041c701 in netmap_poll (priv=,
events=, sr=0x0)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/dev/netmap/netmap.c:2735
#9  0x8041f63c in netmap_knrw (kn=,
hint=, events=1)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/dev/netmap/netmap_freebsd.c:1313
#10 0x805443d7 in kqueue_kevent (kq=0xf8001aa08600,
td=0xf80141470560, nchanges=, nevents=,
k_ops=0xfe045b5028a0, timeout=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:1700
#11 0x80543fd6 in kern_kevent_fp (td=0xf80141470560,
fp=, nchanges=0, nevents=,
k_ops=, timeout=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:1050
#12 0x80543eef in kern_kevent (td=0xf80141470560, fd=6,
nchanges=0, nevents=64, k_ops=0xfe045b5028a0, timeout=0x0)
at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:993
#13 0x80543cf8 in sys_kevent (td=0xf80141470560,
uap=0xfe045b502a30) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_event.c:925
#14 0x80881b5a in amd64_syscall (td=0xf80141470560,
traced=0) at subr_syscall.c:135
#15 0x8086563b in Xfast_syscall () at
/usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:396
#16 0x00080122813a in ?? ()

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Harald Schmalzbauer  changed:

   What|Removed |Added

   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=2060
   ||53

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 219846] [panic] [kevent] mutex nm_kn_lock not owned, netmap with INVARIANTS+WITNESS

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219846

Harald Schmalzbauer  changed:

   What|Removed |Added

Summary|[panic] mutex nm_kn_lock|[panic] [kevent] mutex
   |not owned, netmap LOR   |nm_kn_lock not owned,
   ||netmap with
   ||INVARIANTS+WITNESS

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"