On 28.11.2022. 17:07, Alexandr Nedvedicky wrote: > diff below should avoid panic above (and similar panics in pfsync_q_del(). > It also prints some 'error' system message buffer (a.k.a. dmesg) > > We panic because we attempt to remove state from psync queue which is > already empty. the pfsync_q_del() must be acting on some stale information > kept in `st` argument (state). > > the information we print into dmesg buffer should help us understand > what's going on. At the moment I can not explain how does it come > there is a state which claims its presence on state queue, while the > queue in question is empty. > > I'd like to ask you to give a try diff below and repeat your test. > Let it run for some time and collect 'dmesg' output for me after usual > uptime-to-panic elapses during a test run.
Hi, here's panic with WITESS, this diff and this one https://www.mail-archive.com/tech@openbsd.org/msg72582.html I will leave box in ddb ... wsmouse1 at ums1 mux 0 <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 107228145326 32609891)nosync: no unlinked: no timeout: PFTM_TCP_OPENING <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 132937190089 77519715)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124393720902 11468387)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182347109632 42042467)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <4>com1: 1 silo overflow, 0 ibuf overflows <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 142245292777 58899299)nosync: yes unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 475207607577 1004003)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 159470263971 51003747)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 134712868555 08329571)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 790090587538 2371427)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 543286072470 0808291)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 924950197971 1276131)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 139134763774 04146787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 930062379002 4590435)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 125019156702 63792739)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 134887042707 43536739)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 155369141176 78236771)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 100393240929 61031267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 909085788793 5661155)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 909085788793 5661155)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 909085788793 5661155)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 909085788793 5661155)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 909085788793 5661155)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182376159588 61972579)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 159637401507 14238051)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 242206588258 3172195)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105537141411 11420003)nosync: no unlinked: no timeout: PFTM_TCP_OPENING <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 152424297451 57538915)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 498918805181 3434467)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 129428869418 65968739)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 901279038052 0178787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 154248628865 1158627)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 363889982485 9186275)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 444543668810 4588387)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 832369897278 2134371)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 669530136561 9917923)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 147745314982 15638115)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175941839919 65418595)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 791695884393 2148835)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 124396976702 07341667)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 148499614134 9504099)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 164966778776 93449315)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 844576237712 6151267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 140047293399 29068643)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 861558964891 2516195)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175682795237 84819811)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172280851285 92786531)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 881095528157 2201571)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 154058293278 02500195)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 125817183229 92022627)nosync: yes unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 159074870270 08275555)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 409443636135 4249315)nosync: no unlinked: no timeout: PFTM_TCP_OPENING <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 140563877462 76476003)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 104590012087 67415395)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 631463072618 3978083)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 107170862832 10327139)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED state (id 108799382532 30081123)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 108799382532 30081123)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 108799382532 30081123)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 108799382532 30081123)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 108799382532 30081123)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 542828583291 1291491)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 128407006461 67839843)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 171466267788 56850531)nosync: no unlinked: no timeout: PFTM_TCP_OPENING <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 116917011081 27204451)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 845142386987 1752291)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 447177231946 2098019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 747028786726 0339299)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 898129282553 8102371)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 137650304277 74846051)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 118650754387 84513123)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 873453274233 6119907)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 797539363493 0820195)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 500499810130 7917411)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 802910698126 5900643)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 636900935006 8307043)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172549468621 9928675)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172549468621 9928675)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172549468621 9928675)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172549468621 9928675)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 172549468621 9928675)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 297299728914 9629539)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 264684255793 2668003)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 597498840179 3205347)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 177289018431 37234019)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 128019484576 31122531)nosync: no unlinked: no timeout: PFTM_UDP_MULTIPLE <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 169645202642 90698339)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 191599499710 6574435)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 191599499710 6574435)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 127652093846 90549859)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 135791052406 05951075)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 236598113018 5147491)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 236598113018 5147491)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 144705288296 26942563)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 149916379675 64866659)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 175217560643 72393059)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 201229243737 5353955)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 715299725157 762147)nosync: no unlinked: no timeout: PFTM_TCP_OPENING <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 290510247619 9232611)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 290510247619 9232611)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 290510247619 9232611)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 290510247619 9232611)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 830386711043 671139)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 182810051358 25372259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105833242297 65596259)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 313440013290 8745827)nosync: yes unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 104719118203 29649251)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 165056314113 31826787)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 378583768691 6809827)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 378583768691 6809827)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 378583768691 6809827)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 105996211997 02426723)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 305608522678 7079267)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 185241136924 8687203)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 185241136924 8687203)nosync: no unlinked: no timeout: PFTM_TCP_ESTABLISHED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 185241136924 8687203)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 185241136924 8687203)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 185241136924 8687203)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 555202170611 0305379)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 267181688104 584291)nosync: no unlinked: no timeout: PFTM_TCP_CLOSED <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 111188974646 41750115)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 995760977917 412451)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT <3>pfsync: pfsync_q_del: stale queue (PFSYNC_S_UPD_C) in state (id 146580945568 63956067)nosync: no unlinked: no timeout: PFTM_TCP_FIN_WAIT r620-1# panic: acquiring blockable sleep lock with spinlock or critical section held ( kernel_lock) &kernel_lock Stopped at db_enter+0x10: popq %rbp TID PID UID PRFLAGS PFLAGS CPU COMMAND *451557 8367 0 0x14000 0x200 3 softnet db_enter() at db_enter+0x10 panic(ffffffff81f58683) at panic+0xbf witness_checkorder(ffffffff82323008,9,0) at witness_checkorder+0xb61 __mp_lock(ffffffff82322e00) at __mp_lock+0x5f kpageflttrap(ffff800022d65370,17) at kpageflttrap+0x1b2 kerntrap(ffff800022d65370) at kerntrap+0x91 alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b pfsync_q_del(fffffd83ad2657f0) at pfsync_q_del+0xbc pf_remove_state(fffffd83ad2657f0) at pf_remove_state+0x14e pfsync_in_del_c(fffffd800829b538,c,79,2) at pfsync_in_del_c+0x6f pfsync_input(ffff800022d65628,ffff800022d65634,f0,2) at pfsync_input+0x33c ip_deliver(ffff800022d65628,ffff800022d65634,f0,2) at ip_deliver+0x113 ipintr() at ipintr+0x69 if_netisr(0) at if_netisr+0xea end trace frame: 0xffff800022d65710, count: 0 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{3}> ddb{3}> show panic *cpu3: acquiring blockable sleep lock with spinlock or critical section held (k ernel_lock) &kernel_lock ddb{3}> show reg rdi 0 rsi 0x14 rbp 0xffff800022d65160 rbx 0xffffffff822d9af8 lock_class_kernel_lock rdx 0xc000000000000000 rcx 0x286 rax 0x68 r8 0x101010101010101 r9 0 r10 0x2660b6d186ee32d3 r11 0xd7c06341fd499f9e r12 0xffff80002251c990 r13 0xffffffff8237fef8 w_locklistdata+0x43e68 r14 0 r15 0xffffffff81f58683 cmd0646_9_tim_udma+0x44d97 rip 0xffffffff81daf890 db_enter+0x10 cs 0x8 rflags 0x282 rsp 0xffff800022d65160 ss 0x10 db_enter+0x10: popq %rbp ddb{3}> ddb{3}> ps PID TID PPID UID S FLAGS WAIT COMMAND 27770 386238 1 0 3 0x100083 ttyin ksh 37185 459272 1 0 3 0x100098 kqread cron 85256 378255 33267 95 3 0x1100092 kqread smtpd 44487 99537 33267 103 3 0x1100092 kqread smtpd 69524 32660 33267 95 3 0x1100092 kqread smtpd 1195 379710 33267 95 3 0x100092 kqread smtpd 42719 465371 33267 95 3 0x1100092 kqread smtpd 16343 7460 33267 95 3 0x1100092 kqread smtpd 33267 181901 1 0 3 0x100080 kqread smtpd 75506 220238 1 0 3 0x88 kqread sshd 15004 258054 1 0 3 0x100080 kqread ntpd 61684 156740 12982 83 3 0x100092 kqread ntpd 12982 102627 1 83 3 0x1100092 kqread ntpd 88449 174316 17350 74 3 0x1100092 bpf pflogd 17350 93458 1 0 3 0x80 netio pflogd 46983 283727 98623 73 3 0x1100090 kqread syslogd 98623 154118 1 0 3 0x100082 netio syslogd 59714 468004 0 0 3 0x14200 bored smr 18593 231333 0 0 3 0x14200 pgzero zerothread 60734 251071 0 0 3 0x14200 aiodoned aiodoned 81548 489394 0 0 3 0x14200 syncer update 50476 309309 0 0 3 0x14200 cleaner cleaner 53868 504426 0 0 3 0x14200 reaper reaper 39154 438682 0 0 3 0x14200 pgdaemon pagedaemon 11478 327461 0 0 3 0x14200 usbtsk usbtask 7839 136957 0 0 3 0x14200 usbatsk usbatsk 59823 447112 0 0 3 0x40014200 acpi0 acpi0 12966 349531 0 0 7 0x40014200 idle5 98398 298451 0 0 7 0x40014200 idle4 55161 51088 0 0 3 0x40014200 idle3 99660 476058 0 0 7 0x40014200 idle2 76107 10345 0 0 7 0x40014200 idle1 59204 205984 0 0 3 0x14200 bored sensors 68970 250617 0 0 3 0x14200 netlock softnet 98798 223371 0 0 3 0x14200 netlock softnet 5537 137474 0 0 3 0x14200 netlock softnet 28444 195546 0 0 3 0x14200 netlock softnet 21146 7946 0 0 3 0x14200 netlock softnet * 8367 451557 0 0 7 0x14200 softnet 33996 258966 0 0 3 0x14200 bored systqmp 52172 299137 0 0 3 0x14200 bored systq 4105 108060 0 0 3 0x40014200 netlock softclock 33568 402366 0 0 7 0x40014200 idle0 1 254187 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper ddb{3}> ddb{3}> mach ddbcpu 0 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffffffff822ceff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffffffff822ceff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{0}> mach ddbcpu 1 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022509ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022509ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{1}> mach ddbcpu 2 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022512ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022512ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{2}> mach ddbcpu 3 Stopped at db_enter+0x10: popq %rbp db_enter() at db_enter+0x10 panic(ffffffff81f58683) at panic+0xbf witness_checkorder(ffffffff82323008,9,0) at witness_checkorder+0xb61 __mp_lock(ffffffff82322e00) at __mp_lock+0x5f kpageflttrap(ffff800022d65370,17) at kpageflttrap+0x1b2 kerntrap(ffff800022d65370) at kerntrap+0x91 alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b pfsync_q_del(fffffd83ad2657f0) at pfsync_q_del+0xbc pf_remove_state(fffffd83ad2657f0) at pf_remove_state+0x14e pfsync_in_del_c(fffffd800829b538,c,79,2) at pfsync_in_del_c+0x6f pfsync_input(ffff800022d65628,ffff800022d65634,f0,2) at pfsync_input+0x33c ip_deliver(ffff800022d65628,ffff800022d65634,f0,2) at ip_deliver+0x113 ipintr() at ipintr+0x69 if_netisr(0) at if_netisr+0xea end trace frame: 0xffff800022d65710, count: 0 ddb{3}> mach ddbcpu 4 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff800022524ff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff800022524ff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{4}> mach ddbcpu 5 Stopped at x86_ipi_db+0x12: leave x86_ipi_db(ffff80002252dff0) at x86_ipi_db+0x12 x86_ipi_handler() at x86_ipi_handler+0x80 Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23 acpicpu_idle() at acpicpu_idle+0x281 sched_idle(ffff80002252dff0) at sched_idle+0x280 end trace frame: 0x0, count: 10 ddb{5}>
