Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-10 Thread Michael Butler
On 2019-05-09 14:38, Gleb Smirnoff wrote:
> On Thu, May 09, 2019 at 02:32:44PM -0400, Michael Butler wrote:
> M> (kgdb) frame 8
> M> #8  0x80a15377 in ip_output (m=, opt= M> optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
> M> /usr/src/sys/netinet/ip_output.c:362
> M> 362 IFP_TO_IA(ifp, ia, _ifa_tracker);
> M> (kgdb) print imo
> M> $1 = (struct ip_moptions *) 0xfe0072b14780
> M> (kgdb) print ifp
> M> $2 = (struct ifnet *) 0xf8000411
> M> (kgdb) print ia
> M> $3 = 
> ...
> 
> This all looks good.
> 
> Can you please traverse the 'in_ifaddrhead' linked list?

To close the loop on this - it's now fixed by SVN r347466,

Thanks!

imb


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Gleb Smirnoff
On Thu, May 09, 2019 at 02:32:44PM -0400, Michael Butler wrote:
M> (kgdb) frame 8
M> #8  0x80a15377 in ip_output (m=, opt= optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
M> /usr/src/sys/netinet/ip_output.c:362
M> 362 IFP_TO_IA(ifp, ia, _ifa_tracker);
M> (kgdb) print imo
M> $1 = (struct ip_moptions *) 0xfe0072b14780
M> (kgdb) print ifp
M> $2 = (struct ifnet *) 0xf8000411
M> (kgdb) print ia
M> $3 = 
...

This all looks good.

Can you please traverse the 'in_ifaddrhead' linked list?

-- 
Gleb Smirnoff
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Michael Butler
On 2019-05-09 14:22, Gleb Smirnoff wrote:
>   Michael,
> 
> On Thu, May 09, 2019 at 10:25:37AM -0500, Kyle Evans wrote:
> K> > #0  doadump () at src/sys/amd64/include/pcpu.h:241
> K> > #1  0x808393c8 in kern_reboot (howto=260) at
> K> > /usr/src/sys/kern/kern_shutdown.c:470
> K> > #2  0x80839826 in vpanic (fmt=, ap= K> > optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896
> K> > #3  0x80839663 in panic (fmt=) at
> K> > /usr/src/sys/kern/kern_shutdown.c:823
> K> > #4  0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156)
> K> > at /usr/src/sys/amd64/amd64/trap.c:946
> K> > #5  0x80c31c59 in trap_pfault (frame=0xfe0072b14510,
> K> > usermode=0) at src/sys/amd64/include/cpufunc.h:423
> K> > #6  0x80c30fde in trap (frame=0xfe0072b14510) at
> K> > /usr/src/sys/amd64/amd64/trap.c:441
> K> > #7  0x80c0d4b5 in calltrap () at
> K> > /usr/src/sys/amd64/amd64/exception.S:232
> K> > #8  0x80a15377 in ip_output (m=, opt= K> > optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
> K> > /usr/src/sys/netinet/ip_output.c:362
> K> > #9  0x809ffea4 in igmp_intr (m=) at
> K> > /usr/src/sys/netinet/igmp.c:3455
> K> > #10 0x80975a0f in netisr_dispatch_src (proto=2, source= K> > optimized out>, m=) at 
> /usr/src/sys/net/netisr.c:1122
> K> > #11 0x809fe07a in igmp_fasttimo () at
> K> > /usr/src/sys/netinet/igmp.c:496
> K> > #12 0x808c5854 in pffasttimo (arg=) at
> K> > /usr/src/sys/kern/uipc_domain.c:521
> K> > #13 0x80853df3 in softclock_call_cc (c=0x813f7f48,
> K> > cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
> K> > #14 0x808542b9 in softclock (arg=0x814c9ac0) at
> K> > /usr/src/sys/kern/kern_timeout.c:869
> K> > #15 0x807fd0c4 in ithread_loop (arg=) at
> K> > /usr/src/sys/kern/kern_intr.c:1129
> K> > #16 0x807f9f33 in fork_exit (callout=0x807fcef0
> K> > , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at
> K> > /usr/src/sys/kern/kern_fork.c:1058
> K> > #17 0x80c0e4ae in fork_trampoline () at
> K> > /usr/src/sys/amd64/amd64/exception.S:995
> K> > #18 0x in ?? ()
> K> >
> K> 
> K> Ah, I misread your backtrace (and forgot the proper tap detachment
> K> from my previous patch, so that's fixed/committed anyways). CC'ing
> K> Gleb for further triage as committer of r347375 that touched things in
> K> this path.
> 
> Michael, can you please dump a core and look at it in kgdb? Line 362 in
> ip_output() really belongs to part that had minimal change with r347375.
> So I need more details. Can you please print out in kgdb the following
> variables: imo, ifp, ia?
> 

This was a backtrace from kgdb. From frame 8, I see ..

(kgdb) frame 8
#8  0x80a15377 in ip_output (m=, opt=, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
/usr/src/sys/netinet/ip_output.c:362
362 IFP_TO_IA(ifp, ia, _ifa_tracker);
(kgdb) print imo
$1 = (struct ip_moptions *) 0xfe0072b14780
(kgdb) print ifp
$2 = (struct ifnet *) 0xf8000411
(kgdb) print ia
$3 = 

(kgdb) print *imo
$4 = {imo_multicast_ifp = 0xf8000411, imo_multicast_addr =
{s_addr = 1924220944}, imo_multicast_vif = 18446744073709551615,
imo_multicast_ttl = 1 '\001', imo_multicast_loop = 0 '\0',
  imo_num_memberships = 15350, imo_max_memberships = 63489,
imo_membership = 0xf80002c10d00, imo_mfilters = 0xf80002c10d00,
imo_epoch_ctx = {data = 0xfe0072b147b0}}
(kgdb) print *ifp
$5 = {if_link = {cstqe_next = 0x0}, if_clones = {le_next = 0x0, le_prev
= 0xf800040f9728}, if_groups = {cstqh_first = 0xf80002878d60,
cstqh_last = 0xf80002878d38},
  if_alloctype = 6 '\006', if_numa_domain = 255 '�', if_softc =
0xf80004197300, if_llsoftc = 0x0, if_l2com = 0xf800040fe800,
if_dname = 0x80e0bd14 "tap", if_dunit = 0,
  if_index = 4, if_index_reserved = 0, if_xname = 0xf80004110058
"tap0", if_description = 0x0, if_flags = 34818, if_drv_flags = 0,
if_capabilities = 524288, if_capenable = 524288,
  if_linkmib = 0x0, if_linkmiblen = 0, if_refcount = 1, if_type = 6
'\006', if_addrlen = 6 '\006', if_hdrlen = 14 '\016', if_link_state = 1
'\001', if_mtu = 1500, if_metric = 0,
  if_baudrate = 1000, if_hwassist = 0, if_epoch = 10, if_lastchange
= {tv_sec = 1557408022, tv_usec = 929504}, if_snd = {ifq_head = 0x0,
ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 50,
ifq_mtx = {lock_object = {lo_name = 0xf80004110058 "tap0",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0},
ifq_drv_head = 0x0, ifq_drv_tail = 0x0,
ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0,
altq_disc = 0x0, altq_ifp = 0xf8000411, altq_enqueue = 0,
altq_dequeue = 0, altq_request = 0,
altq_clfier = 0x0, altq_classify = 0, altq_tbr = 0x0, altq_cdnr =
0x0}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
ta_priority = 0,
ta_func = 0x809494e0 , ta_context 

Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Gleb Smirnoff
  Michael,

On Thu, May 09, 2019 at 10:25:37AM -0500, Kyle Evans wrote:
K> > #0  doadump () at src/sys/amd64/include/pcpu.h:241
K> > #1  0x808393c8 in kern_reboot (howto=260) at
K> > /usr/src/sys/kern/kern_shutdown.c:470
K> > #2  0x80839826 in vpanic (fmt=, ap= > optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896
K> > #3  0x80839663 in panic (fmt=) at
K> > /usr/src/sys/kern/kern_shutdown.c:823
K> > #4  0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156)
K> > at /usr/src/sys/amd64/amd64/trap.c:946
K> > #5  0x80c31c59 in trap_pfault (frame=0xfe0072b14510,
K> > usermode=0) at src/sys/amd64/include/cpufunc.h:423
K> > #6  0x80c30fde in trap (frame=0xfe0072b14510) at
K> > /usr/src/sys/amd64/amd64/trap.c:441
K> > #7  0x80c0d4b5 in calltrap () at
K> > /usr/src/sys/amd64/amd64/exception.S:232
K> > #8  0x80a15377 in ip_output (m=, opt= > optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
K> > /usr/src/sys/netinet/ip_output.c:362
K> > #9  0x809ffea4 in igmp_intr (m=) at
K> > /usr/src/sys/netinet/igmp.c:3455
K> > #10 0x80975a0f in netisr_dispatch_src (proto=2, source= > optimized out>, m=) at /usr/src/sys/net/netisr.c:1122
K> > #11 0x809fe07a in igmp_fasttimo () at
K> > /usr/src/sys/netinet/igmp.c:496
K> > #12 0x808c5854 in pffasttimo (arg=) at
K> > /usr/src/sys/kern/uipc_domain.c:521
K> > #13 0x80853df3 in softclock_call_cc (c=0x813f7f48,
K> > cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
K> > #14 0x808542b9 in softclock (arg=0x814c9ac0) at
K> > /usr/src/sys/kern/kern_timeout.c:869
K> > #15 0x807fd0c4 in ithread_loop (arg=) at
K> > /usr/src/sys/kern/kern_intr.c:1129
K> > #16 0x807f9f33 in fork_exit (callout=0x807fcef0
K> > , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at
K> > /usr/src/sys/kern/kern_fork.c:1058
K> > #17 0x80c0e4ae in fork_trampoline () at
K> > /usr/src/sys/amd64/amd64/exception.S:995
K> > #18 0x in ?? ()
K> >
K> 
K> Ah, I misread your backtrace (and forgot the proper tap detachment
K> from my previous patch, so that's fixed/committed anyways). CC'ing
K> Gleb for further triage as committer of r347375 that touched things in
K> this path.

Michael, can you please dump a core and look at it in kgdb? Line 362 in
ip_output() really belongs to part that had minimal change with r347375.
So I need more details. Can you please print out in kgdb the following
variables: imo, ifp, ia?

-- 
Gleb Smirnoff
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Kyle Evans
On Thu, May 9, 2019 at 8:27 AM Michael Butler
 wrote:
>
> On 2019-05-09 09:07, Kyle Evans wrote:
> > On Thu, May 9, 2019 at 7:24 AM Michael Butler
> >  wrote:
> >>
> >> Seems there's a lock or tuntap issue after recent changes. Restarting
> >> openvpn (configured to use /dev/tap) yields a panic as follows:
> >>
> >
> > Ah, I knew I was forgetting something. =( Give this a shot:
> > https://people.freebsd.org/~kevans/etherdetach.diff
>
> Sorry - same result,
>
> (kgdb) bt
> #0  doadump () at src/sys/amd64/include/pcpu.h:241
> #1  0x808393c8 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:470
> #2  0x80839826 in vpanic (fmt=, ap= optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896
> #3  0x80839663 in panic (fmt=) at
> /usr/src/sys/kern/kern_shutdown.c:823
> #4  0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156)
> at /usr/src/sys/amd64/amd64/trap.c:946
> #5  0x80c31c59 in trap_pfault (frame=0xfe0072b14510,
> usermode=0) at src/sys/amd64/include/cpufunc.h:423
> #6  0x80c30fde in trap (frame=0xfe0072b14510) at
> /usr/src/sys/amd64/amd64/trap.c:441
> #7  0x80c0d4b5 in calltrap () at
> /usr/src/sys/amd64/amd64/exception.S:232
> #8  0x80a15377 in ip_output (m=, opt= optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
> /usr/src/sys/netinet/ip_output.c:362
> #9  0x809ffea4 in igmp_intr (m=) at
> /usr/src/sys/netinet/igmp.c:3455
> #10 0x80975a0f in netisr_dispatch_src (proto=2, source= optimized out>, m=) at /usr/src/sys/net/netisr.c:1122
> #11 0x809fe07a in igmp_fasttimo () at
> /usr/src/sys/netinet/igmp.c:496
> #12 0x808c5854 in pffasttimo (arg=) at
> /usr/src/sys/kern/uipc_domain.c:521
> #13 0x80853df3 in softclock_call_cc (c=0x813f7f48,
> cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
> #14 0x808542b9 in softclock (arg=0x814c9ac0) at
> /usr/src/sys/kern/kern_timeout.c:869
> #15 0x807fd0c4 in ithread_loop (arg=) at
> /usr/src/sys/kern/kern_intr.c:1129
> #16 0x807f9f33 in fork_exit (callout=0x807fcef0
> , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at
> /usr/src/sys/kern/kern_fork.c:1058
> #17 0x80c0e4ae in fork_trampoline () at
> /usr/src/sys/amd64/amd64/exception.S:995
> #18 0x in ?? ()
>

Ah, I misread your backtrace (and forgot the proper tap detachment
from my previous patch, so that's fixed/committed anyways). CC'ing
Gleb for further triage as committer of r347375 that touched things in
this path.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Michael Butler
On 2019-05-09 09:07, Kyle Evans wrote:
> On Thu, May 9, 2019 at 7:24 AM Michael Butler
>  wrote:
>>
>> Seems there's a lock or tuntap issue after recent changes. Restarting
>> openvpn (configured to use /dev/tap) yields a panic as follows:
>>
> 
> Ah, I knew I was forgetting something. =( Give this a shot:
> https://people.freebsd.org/~kevans/etherdetach.diff

Sorry - same result,

(kgdb) bt
#0  doadump () at src/sys/amd64/include/pcpu.h:241
#1  0x808393c8 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:470
#2  0x80839826 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:896
#3  0x80839663 in panic (fmt=) at
/usr/src/sys/kern/kern_shutdown.c:823
#4  0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156)
at /usr/src/sys/amd64/amd64/trap.c:946
#5  0x80c31c59 in trap_pfault (frame=0xfe0072b14510,
usermode=0) at src/sys/amd64/include/cpufunc.h:423
#6  0x80c30fde in trap (frame=0xfe0072b14510) at
/usr/src/sys/amd64/amd64/trap.c:441
#7  0x80c0d4b5 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:232
#8  0x80a15377 in ip_output (m=, opt=, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
/usr/src/sys/netinet/ip_output.c:362
#9  0x809ffea4 in igmp_intr (m=) at
/usr/src/sys/netinet/igmp.c:3455
#10 0x80975a0f in netisr_dispatch_src (proto=2, source=, m=) at /usr/src/sys/net/netisr.c:1122
#11 0x809fe07a in igmp_fasttimo () at
/usr/src/sys/netinet/igmp.c:496
#12 0x808c5854 in pffasttimo (arg=) at
/usr/src/sys/kern/uipc_domain.c:521
#13 0x80853df3 in softclock_call_cc (c=0x813f7f48,
cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
#14 0x808542b9 in softclock (arg=0x814c9ac0) at
/usr/src/sys/kern/kern_timeout.c:869
#15 0x807fd0c4 in ithread_loop (arg=) at
/usr/src/sys/kern/kern_intr.c:1129
#16 0x807f9f33 in fork_exit (callout=0x807fcef0
, arg=0xf800025f10a0, frame=0xfe0072b14ac0) at
/usr/src/sys/kern/kern_fork.c:1058
#17 0x80c0e4ae in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:995
#18 0x in ?? ()

imb

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: at SVN r347375, terminating/restarting openvpn on tap causes panic

2019-05-09 Thread Kyle Evans
On Thu, May 9, 2019 at 7:24 AM Michael Butler
 wrote:
>
> Seems there's a lock or tuntap issue after recent changes. Restarting
> openvpn (configured to use /dev/tap) yields a panic as follows:
>

Ah, I knew I was forgetting something. =( Give this a shot:
https://people.freebsd.org/~kevans/etherdetach.diff

Thanks,

Kyle Evans
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"