Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Brian Somers]

> > Interfaces in promiscuous mode will always result in a reboot.  I 
> > *usually* get away with ejecting an active card if it's not in 
> > promiscuous mode.
> 
> A while back I committed patches to use bpf_detach(), which elminated the
> struct ifnet pointer in the bpf described at detach time.  This removed
> most of the panics I experience on ejecting pccards.  This should be in
> 5.0-CURRENT and 4.1-STABLE.  If you're still experiencing panics, we
> should track it down some more as presumably there is another reference
> (it could also be a race condition, or in-use mbuf during detach?)

I guess with a promiscuous interface, it's a whole lot more likely 
that there's an in-use mbuf around somewhere.  That's probably what 
panics things I would think.

>   Robert N M Watson 
> 
> [EMAIL PROTECTED]  http://www.watson.org/~robert/
> PGP key fingerprint: AF B5 5F FF A6 4A 79 37  ED 5F 55 E9 58 04 6A B1
> TIS Labs at Network Associates, Safeport Network Services

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Wes Peters]

Warner Losh wrote:
> 
> In message <[EMAIL PROTECTED]> Wes Peters writes:
> : state the code is in now, and if someone wants it to be better, we await
> : their patches.  As always.  ;^)
> 
> Tanimura-san did contribute patches.  This problem isn't a race at the
> eject, but rather the network layer incompletely cleaning up after a
> device has gone away.

Robert Watson and I had a nice discussion about how to approach that
problem a while back.  I've since gotten busy and forgotten about it,
as has he (apparently).

The quick (-ish) fix I came up with is to double all those cache ifp's
all over the code, and always check the first pointer for a null
reference before diving off through it.  The disadvantage is the check
and dereference on every access, the advantage is being able to
nullify one pointer in the interface take-down and automagically have
all the cached references die.  You would leak a single pointer for
every interface detach, unless you did reference counting or something
like that.

The full solution would be to implement reference counting in the if
objects, and to always check the state of the interface before trying
to exercise an associated function.  It's an ugly problem with no real 
simple solutions (in C).

-- 
"Where am I, and what am I doing in this handbasket?"

Wes Peters Softweyr LLC
[EMAIL PROTECTED]   http://softweyr.com/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Wes Peters]

Seigo Tanimura wrote:
> 
> In ip_output(), imo->imo_multicast_ifp points to the removed
> interface, which is passed to IN_LOOKUP_MULTI() to cause a panic. This
> problem also occurs when we attempt to delete a multicast address from
> a removed interface. if_delmulti() derefers an ifp to the removed
> interface, ending up with a panic. The problem does not occur for
> unicast.
> 
> http://people.FreeBSD.org/~tanimura/patches/mcastif.diff.gz is a
> workround patch. The idea is to track all of the active ifps, confirm
> an ifp to be active prior to dereferencing it, and free orphaned
> ifmultiaddrs attached to a removed ifp. It would be much more elegant
> if we could clean up the multicast information related to a removed
> interface, though.

Warner Losh pointed out to me:

> Tanimura-san did contribute patches.  This problem isn't a race at the
> eject, but rather the network layer incompletely cleaning up after a
> device has gone away.

This is, of course, exactly the problem you're looking at.  There are 
several of these cached ifp's hanging around, all causing problems.

Robert Watson and I had a nice discussion about how to approach that
problem a while back.  I've since gotten busy and forgotten about it,
as has he (apparently).

The quick (-ish) fix I came up with is to double all those cache ifp's
all over the code, and always check the first pointer for a null
reference before diving off through it.  The disadvantage is the check
and dereference on every access, the advantage is being able to
nullify one pointer in the interface take-down and automagically have
all the cached references die.  You would leak a single pointer for
every interface detach, unless you did reference counting or something
like that.

The full solution would be to implement ifs a full objects, and to 
always check the state of the interface before trying to exercise an 
associated function.  It's an ugly problem with no real simple solutions 
(in C).

-- 
"Where am I, and what am I doing in this handbasket?"

Wes Peters Softweyr LLC
[EMAIL PROTECTED]   http://softweyr.com/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Robert Watson]

On Sat, 9 Sep 2000, Brian Somers wrote:

> > Seigo Tanimura wrote:
> > > 
> > > I have been suffering from this problem for almost 2 months. When I
> > > remove a pcmcia ethernet card from my laptop PC, routed(8) announces
> > > updated routing information by multicast, leading to a kernel
> > > panic.
> > 
> > Ejecting an interface configured up will do that.  ifconfig the interface
> > `down' and then `delete' before ejecting it.
> 
> Interfaces in promiscuous mode will always result in a reboot.  I 
> *usually* get away with ejecting an active card if it's not in 
> promiscuous mode.

A while back I committed patches to use bpf_detach(), which elminated the
struct ifnet pointer in the bpf described at detach time.  This removed
most of the panics I experience on ejecting pccards.  This should be in
5.0-CURRENT and 4.1-STABLE.  If you're still experiencing panics, we
should track it down some more as presumably there is another reference
(it could also be a race condition, or in-use mbuf during detach?)


  Robert N M Watson 

[EMAIL PROTECTED]  http://www.watson.org/~robert/
PGP key fingerprint: AF B5 5F FF A6 4A 79 37  ED 5F 55 E9 58 04 6A B1
TIS Labs at Network Associates, Safeport Network Services



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Brian Somers]

> Seigo Tanimura wrote:
> > 
> > I have been suffering from this problem for almost 2 months. When I
> > remove a pcmcia ethernet card from my laptop PC, routed(8) announces
> > updated routing information by multicast, leading to a kernel
> > panic.
> 
> Ejecting an interface configured up will do that.  ifconfig the interface
> `down' and then `delete' before ejecting it.

Interfaces in promiscuous mode will always result in a reboot.  I 
*usually* get away with ejecting an active card if it's not in 
promiscuous mode.

> -- 
> "Where am I, and what am I doing in this handbasket?"
> 
> Wes Peters Softweyr LLC
> [EMAIL PROTECTED]   http://softweyr.com/

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Julian Elischer]

Warner Losh wrote:
>
> NetBSD has done some interesting things in this area with reference
> counting and such that I'd love to bring in as soon as I get newcard
> done.


A few years ago I did some work on rationalising the reference counting
in
the kernel WRT network routes and ifps.
Some of it was actually checked in on a branch, however it 
should probably be done from scratch since so much has changed.

the basic things I did were:

ALL references to ifps were counted 
  (mbufs too, though that was never checked in)
References to ifnets in many structures were changed to be 
references to ifaddrs attached to the ifnets instead.
Ifaddrs and ifnet structs had a 'VALID' flag added to them.

Whenever a new pointer to an ifnet/ifaddr was followed, the VALID flag
was checked.
if the VALID flag was not set then the operation was aborted and the
reference being followed was dropped. (on reaching 0 the target was
freed).
If you freed an interface, then the structures would hang around as long
as they had refering pointers outstanding. This could possibly lead to a
leak,
but that would by definition also be a leak of mbufs or routes. The
ifaddrs
also had references to the ifps. However on boing made invalid,
they dropped those references immediatly. so theoretically,
since many structs pointed to ifaddrs instead of ifnets, it would be
quite possible for the ifnet structs to be freed long before an ifaddr
that was once attached to it. (this was safe because the VALID flag
was not set in the ifaddr, so nothing would follow the pointer.
(which was NULLED out anyhow).

I had a set of small methods to do these dereferences and stuff,
but the  hardest part was deciding in the CALLING routines (e.g.
ipoutput)
what to do when you discovered that the ifnet you were going to access
came back as INVALID).

Note that this didn't totally close the window for disaster
as it is possible that an interface might be disabled between the 
time that you followed the ifp and the time that you used the pointer.
it took it down to millisecconds however as opposed to seconds. 
I think it would be possible using sufficiently paranoid coding
to completely close the window, but the question is raised
"how much time do you want to waste in normal operation
being paranoid?"

With SMP the windows widen again, so we need to think a lot about this.
Maybe all Interface removals are not done directly but handled in a 
serialised manner by some kernel thread? This feeds into what I believe 
should be done in the general case of device removal, which is the 
creation of a devd kernel thread, that responds to requests from drivers
to 
instatiate and deinstanciate system linkages. having this done by the 
drivers themselves raises so many complications WRT locking and 
such that I think it would be a win overall.
The drivers would supply code to be run by the devd, using methods 
supplied by the devd.

-- 
  __--_|\  Julian Elischer
 /   \ [EMAIL PROTECTED]
(   OZ) World tour 2000
---> X_.---._/  presently in:  Perth
v


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Warner Losh]

In message <[EMAIL PROTECTED]> Robert Watson 
writes:
: This is all made harder by the fact that struct mbuf has a struct ifnet
: pointer in it, so if for any reason there is an outstanding mbuf
: originating from that interface, it is possible that the struct ifnet *
: will be dereferenced.  For example, if it hits an ipfw rule that dummynets
: it, then hits an interface-based rule.

Yes.  That's true.  There's a race.

: This has been raised as an issue before, and is a good reason to ifconfig
: down the interface, and wait a second or two before ejecting.  You could
: imagine code-based solutions, including scanning mbufs (?) for pointers
: that are undesirable, refcounting the struct ifnet so it isn't freed until
: all mbufs are free'd, etc.  Whatever the case, you want to make sure that
: locking in the line of fire is avoided (i.e., attempting to lock struct
: ifnet during packet handling in the interrupt).

No body waits :-(.  This is made worse by pccard's detaching the
device when a suspend happens via acpi or apm.

NetBSD has done some interesting things in this area with reference
counting and such that I'd love to bring in as soon as I get newcard
done.

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Robert Watson]

On Tue, 5 Sep 2000, Warner Losh wrote:

> In message <[EMAIL PROTECTED]> Wes Peters writes:
> : Ejecting an interface configured up will do that.  ifconfig the interface
> : `down' and then `delete' before ejecting it.
> 
> At best this is an unsatisfactory workaround.  if_detach should cause
> the right thing to happen.

This is all made harder by the fact that struct mbuf has a struct ifnet
pointer in it, so if for any reason there is an outstanding mbuf
originating from that interface, it is possible that the struct ifnet *
will be dereferenced.  For example, if it hits an ipfw rule that dummynets
it, then hits an interface-based rule.

This has been raised as an issue before, and is a good reason to ifconfig
down the interface, and wait a second or two before ejecting.  You could
imagine code-based solutions, including scanning mbufs (?) for pointers
that are undesirable, refcounting the struct ifnet so it isn't freed until
all mbufs are free'd, etc.  Whatever the case, you want to make sure that
locking in the line of fire is avoided (i.e., attempting to lock struct
ifnet during packet handling in the interrupt).

  Robert N M Watson 

[EMAIL PROTECTED]  http://www.watson.org/~robert/
PGP key fingerprint: AF B5 5F FF A6 4A 79 37  ED 5F 55 E9 58 04 6A B1
TIS Labs at Network Associates, Safeport Network Services




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Warner Losh]

In message <[EMAIL PROTECTED]> Wes Peters writes:
: state the code is in now, and if someone wants it to be better, we await
: their patches.  As always.  ;^)

Tanimura-san did contribute patches.  This problem isn't a race at the
eject, but rather the network layer incompletely cleaning up after a
device has gone away.

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Wes Peters]

Warner Losh wrote:
> 
> In message <[EMAIL PROTECTED]> Wes Peters writes:
> : Ejecting an interface configured up will do that.  ifconfig the interface
> : `down' and then `delete' before ejecting it.
> 
> At best this is an unsatisfactory workaround.  if_detach should cause
> the right thing to happen.

Right, but as we've discussed before, that may not be possible with PCCard
and CardBus, or anything else short of CPCI.  In the meantime, this is the
state the code is in now, and if someone wants it to be better, we await
their patches.  As always.  ;^)

-- 
"Where am I, and what am I doing in this handbasket?"

Wes Peters Softweyr LLC
[EMAIL PROTECTED]   http://softweyr.com/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Warner Losh]

In message <[EMAIL PROTECTED]> Wes Peters writes:
: Ejecting an interface configured up will do that.  ifconfig the interface
: `down' and then `delete' before ejecting it.

At best this is an unsatisfactory workaround.  if_detach should cause
the right thing to happen.

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Wes Peters]

Seigo Tanimura wrote:
> 
> I have been suffering from this problem for almost 2 months. When I
> remove a pcmcia ethernet card from my laptop PC, routed(8) announces
> updated routing information by multicast, leading to a kernel
> panic.

Ejecting an interface configured up will do that.  ifconfig the interface
`down' and then `delete' before ejecting it.

-- 
"Where am I, and what am I doing in this handbasket?"

Wes Peters Softweyr LLC
[EMAIL PROTECTED]   http://softweyr.com/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

the ifp to a removed pcmcia ethernet card is left in struct ip_moptions and struct ifmultiaddr

[Seigo Tanimura]

I have been suffering from this problem for almost 2 months. When I
remove a pcmcia ethernet card from my laptop PC, routed(8) announces
updated routing information by multicast, leading to a kernel
panic. The stack trace looks like this:

IdlePTD 4136960
initial pcb at 352840
panicstr: page fault
panic messages:
---
Syntax error: Unterminated quoted string
---
#0  boot (howto=256) at ../../kern/kern_shutdown.c:310
310 dumppcb.pcb_cr3 = rcr3();
(kgdb) bt
#0  boot (howto=256) at ../../kern/kern_shutdown.c:310
#1  0xc018f3e1 in panic (fmt=0xc02faf4f "page fault") at ../../kern/kern_shutdown.c:560
#2  0xc02b5043 in trap_fatal (frame=0xcb9b0d7c, eva=213493) at 
../../i386/i386/trap.c:951
#3  0xc02b4ce5 in trap_pfault (frame=0xcb9b0d7c, usermode=0, eva=213493) at 
../../i386/i386/trap.c:844
#4  0xc02b4877 in trap (frame={tf_fs = 16, tf_es = -969998320, tf_ds = 16, tf_edi = 
-1058687232, tf_esi = -1058687744, 
  tf_ebp = -879030772, tf_isp = -879030872, tf_ebx = -1058661888, tf_edx = 
-1067786268, tf_ecx = 213485, tf_eax = -1058661888, 
  tf_trapno = 12, tf_err = 0, tf_eip = -1071746146, tf_cs = 8, tf_eflags = 66054, 
tf_esp = -1067786496, tf_ss = -879030720})
at ../../i386/i386/trap.c:443
#5  0xc01e739e in ip_output (m0=0xc05adf00, opt=0x0, ro=0xcb424dc8, flags=32, 
imo=0xc0e5b700) at ../../netinet/ip_output.c:326
#6  0xc01e8e26 in rip_output (m=0xc05adf00, so=0xcae549c0, dst=33554656) at 
../../netinet/raw_ip.c:234
#7  0xc01e9627 in rip_send (so=0xcae549c0, flags=0, m=0xc05adf00, nam=0xc0da7950, 
control=0x0, p=0xc62f9f60) at ../../netinet/raw_ip.c:566
#8  0xc01b281f in sosend (so=0xcae549c0, addr=0xc0da7950, uio=0xcb9b0ed4, 
top=0xc05adf00, control=0x0, flags=0, p=0xc62f9f60)
at ../../kern/uipc_socket.c:614
#9  0xc01b63be in sendit (p=0xc62f9f60, s=5, mp=0xcb9b0f14, flags=0) at 
../../kern/uipc_syscalls.c:519
#10 0xc01b7998 in sendto (p=0xc62f9f60, uap=0xcb9b0f80) at 
../../kern/uipc_syscalls.c:571
#11 0xc02b51ba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 0, 
tf_esi = 134727225, tf_ebp = -1077937408, 
  tf_isp = -879030316, tf_ebx = 134844928, tf_edx = 2, tf_ecx = -1, tf_eax = 133, 
tf_trapno = 7, tf_err = 2, tf_eip = 134573992, 
  tf_cs = 31, tf_eflags = 659, tf_esp = -1077937484, tf_ss = 47}) at 
../../i386/i386/trap.c:1150
#12 0xc02a8875 in Xint0x80_syscall ()
#13 0x8051b10 in ?? ()
#14 0x8050ff4 in ?? ()
#15 0x804c212 in ?? ()
#16 0x804813f in ?? ()

In ip_output(), imo->imo_multicast_ifp points to the removed
interface, which is passed to IN_LOOKUP_MULTI() to cause a panic. This
problem also occurs when we attempt to delete a multicast address from
a removed interface. if_delmulti() derefers an ifp to the removed
interface, ending up with a panic. The problem does not occur for
unicast.

http://people.FreeBSD.org/~tanimura/patches/mcastif.diff.gz is a
workround patch. The idea is to track all of the active ifps, confirm
an ifp to be active prior to dereferencing it, and free orphaned
ifmultiaddrs attached to a removed ifp. It would be much more elegant
if we could clean up the multicast information related to a removed
interface, though.

-- 
Seigo Tanimura <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message