Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Tue, Sep 23, 2008 at 04:46:38PM -0400, Rodney Dunn wrote: Seems they are not planning a special rebuild for this unfortunately. Mmmh, bad news. We are trying to get them to build a engineering special generally available for TAC if you have a SR open they should be able to get it. That would work fine for us, though. Thanks, gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Gert, Seems they are not planning a special rebuild for this unfortunately. We are trying to get them to build a engineering special generally available for TAC if you have a SR open they should be able to get it. Sorry... Rodney On Mon, Sep 22, 2008 at 09:04:45AM -0400, Rodney Dunn wrote: They are Gert. Let me check on it... On Sat, Sep 20, 2008 at 09:29:53PM +0200, Gert Doering wrote: Hi, On Fri, Sep 19, 2008 at 07:23:36PM +0200, Peter Rathlev wrote: CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Indeed... makes me wonder why they are not doing an SXH rebuild on their own, instead of making us wait 4-6 weeks for a bugfix for a *catastrophic* (!!) bug. (No news from our case yet regarding an interim rebuild) thanks, gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
On Fri, Sep 19, 2008 at 12:46:47PM -0500, Winders, Timothy A wrote: On 9/19/08 12:23 PM, Peter Rathlev [EMAIL PROTECTED] wrote: On Fri, 2008-09-19 at 18:51 +0200, Gert Doering wrote: On Thu, Sep 18, 2008 at 08:36:43PM -0400, Jared Mauch wrote: Your bug (CSCsu59917) should also be listed on CCO. cut What does CCO say about it, right now? (Don't want to check - $very expensive GPRS link...) Probably not totally legal to post this, but here goes. :-) CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Status: Fixed. Fixed-In 12.2(18)SXF15 12.2(33.3.11)SXH 12.2(32.8.11)SX206 I don't understand. How can this show up in SXF15 and be fixed in SXF15? Because when we pull the label there are a few more test cycles that run pre-CCO post. If they find something catastrophic at the last minute they will fix it if at all possible. That appears to be what happened here with the SXF15 build and the bug that caused it. They are pushing for a faster rebuild on SXH to get the fix also. Rodney Or, am I reading this wrong? Tim Winders | Associate Dean of Information Technology | South Plains College ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
They are Gert. Let me check on it... On Sat, Sep 20, 2008 at 09:29:53PM +0200, Gert Doering wrote: Hi, On Fri, Sep 19, 2008 at 07:23:36PM +0200, Peter Rathlev wrote: CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Indeed... makes me wonder why they are not doing an SXH rebuild on their own, instead of making us wait 4-6 weeks for a bugfix for a *catastrophic* (!!) bug. (No news from our case yet regarding an interim rebuild) thanks, gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
On 9/22/08 8:04 AM, Rodney Dunn [EMAIL PROTECTED] wrote: On Fri, Sep 19, 2008 at 12:46:47PM -0500, Winders, Timothy A wrote: On 9/19/08 12:23 PM, Peter Rathlev [EMAIL PROTECTED] wrote: On Fri, 2008-09-19 at 18:51 +0200, Gert Doering wrote: On Thu, Sep 18, 2008 at 08:36:43PM -0400, Jared Mauch wrote: Your bug (CSCsu59917) should also be listed on CCO. cut What does CCO say about it, right now? (Don't want to check - $very expensive GPRS link...) Probably not totally legal to post this, but here goes. :-) CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Status: Fixed. Fixed-In 12.2(18)SXF15 12.2(33.3.11)SXH 12.2(32.8.11)SX206 I don't understand. How can this show up in SXF15 and be fixed in SXF15? Because when we pull the label there are a few more test cycles that run pre-CCO post. If they find something catastrophic at the last minute they will fix it if at all possible. That appears to be what happened here with the SXF15 build and the bug that caused it. They are pushing for a faster rebuild on SXH to get the fix also. Thanks for the answer Rodney. So, it was found in SXF15, but corrected before SXF15 was pushed out the door. Gotcha. Tim Winders | Associate Dean of Information Technology | South Plains College ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Mon, Sep 22, 2008 at 09:04:03AM -0400, Rodney Dunn wrote: They are pushing for a faster rebuild on SXH to get the fix also. Cool! Thank you very much (if you have been involved in this - if not, at least for giving us some more background info). gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Thu, Sep 18, 2008 at 08:36:43PM -0400, Jared Mauch wrote: On Thu, Sep 18, 2008 at 10:13:16PM +0200, Gert Doering wrote: On Tue, Sep 16, 2008 at 06:58:49PM +0200, Gert Doering wrote: I've got no bug ID, but it's on case SR 609537689. Thanks. I'll forward this - and let's see what will happen. Updates. I now have a bug ID, CSCsu59917, which turns out to be a duplicate of an internal bug ID (ID not known because it's internal), which is supposed to be fixed in SXH4 to be released in 4-5 weeks. If it's experienced in a customer network, they should change the bug to found-in: customer-use and make the bug available on CCO. I agree... If they do not, escalate the case. ... but this is a bit difficult right now - I'm on vacation, technically, and thus I'm a bit handicapped regarding escalation things. So this will have to wait a few days. Hopefully they will at least be able to do a SXH3 interim rebuild in the mean time. (I'll check with the developers). We'll raise a stink about this since it's seriously impacting our routing and I'm not really willing to wait for 4 weeks of daily routing loops... Your bug (CSCsu59917) should also be listed on CCO. What does CCO say about it, right now? (Don't want to check - $very expensive GPRS link...) gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
On Fri, 2008-09-19 at 18:51 +0200, Gert Doering wrote: On Thu, Sep 18, 2008 at 08:36:43PM -0400, Jared Mauch wrote: Your bug (CSCsu59917) should also be listed on CCO. cut What does CCO say about it, right now? (Don't want to check - $very expensive GPRS link...) Probably not totally legal to post this, but here goes. :-) CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Status: Fixed. Fixed-In 12.2(18)SXF15 12.2(33.3.11)SXH 12.2(32.8.11)SX206 Regards, Peter (And then a 256-line .sig...) ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
On 9/19/08 12:23 PM, Peter Rathlev [EMAIL PROTECTED] wrote: On Fri, 2008-09-19 at 18:51 +0200, Gert Doering wrote: On Thu, Sep 18, 2008 at 08:36:43PM -0400, Jared Mauch wrote: Your bug (CSCsu59917) should also be listed on CCO. cut What does CCO say about it, right now? (Don't want to check - $very expensive GPRS link...) Probably not totally legal to post this, but here goes. :-) CSCsu59917 SXF15: IPv4/v6 BGP routes not cleared when source routes is gone Severity: 1 - catastrophic. Status: Fixed. Fixed-In 12.2(18)SXF15 12.2(33.3.11)SXH 12.2(32.8.11)SX206 I don't understand. How can this show up in SXF15 and be fixed in SXF15? Or, am I reading this wrong? Tim Winders | Associate Dean of Information Technology | South Plains College ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Tue, Sep 16, 2008 at 06:58:49PM +0200, Gert Doering wrote: I've got no bug ID, but it's on case SR 609537689. Thanks. I'll forward this - and let's see what will happen. Updates. I now have a bug ID, CSCsu59917, which turns out to be a duplicate of an internal bug ID (ID not known because it's internal), which is supposed to be fixed in SXH4 to be released in 4-5 weeks. We'll raise a stink about this since it's seriously impacting our routing and I'm not really willing to wait for 4 weeks of daily routing loops... gert -- Gert Doering Mobile communications ... right now writing from * Sardegna, Italy * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Mon, Sep 15, 2008 at 12:56:08PM -0700, Christopher McCrory wrote: I'm curious, is bgp dampening on or off? BGP dampening is off. gert -- Gert Doering Mobile communications ... right now writing from * Munich Airport * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Mon, Sep 15, 2008 at 09:11:14PM +0100, Peter Taphouse wrote: Just to second (or third?) this bug. We've got four 7600s on SXH3 which are afflicted by this - they were upgraded from 2a on tac's advise (to avoid netflow bug related spontaneous reloads) - and we don't use dampening. It doesn't seem to matter if the prefixes that get withdrawn are i or ebgp, they get still ghosted to other ibgp peers. For iBGP-iBGP ghosts, our current setup is not suitable (read: no route-reflector setup, so iBGP-iBGP announcements would not take place), thus I have no evidence on whether this would also trigger the bug for us. But I think it's quite likely indeed, given that this seems to happen on sending *out* the withdraw... I've got a case open with tac, Would you mind sharing the case number with me? I could forward this to our TAC engineer so they know this is not just us. Do you have a bug ID? but it's causing us enough grief that I'm moving back to SXF until things calm down. *grumble* - I would love to do that, given that we're quite happy with SXF since about two years now. But we were unlucky/stupid enough to get Sup720-10Gs for these new boxes, and they only run SHX... Would love the new netflow stuff in SXH if it gets stable enough... We're quite happy with the SHX3 netflow. No crashes (yet, knock wood) and the load on everything is indeed much lower. There are some other funnies in SXH3, but these are just annoyances and not service impacting (we're not using scp). gert -- Gert Doering Mobile communications ... right now writing from * Munich Airport * ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Would you mind sharing the case number with me? I could forward this to our TAC engineer so they know this is not just us. Do you have a bug ID? I've got no bug ID, but it's on case SR 609537689. SXH3 introduced another new bgp bug too - the output of show ip bgp neigh xx.xxx.xx.xx advertised-routes produced badly wrong output. For example it showed us announcing zero prefixes to one of our transit providers, even though their looking glass showed them receiving them just fine :-/ That's why I opened the case originally, the ghosting bug I noticed afterwards and then I quickly moved to SXF since it was causing too much grief. but it's causing us enough grief that I'm moving back to SXF until things calm down. *grumble* - I would love to do that, given that we're quite happy with SXF since about two years now. But we were unlucky/stupid enough to get Sup720-10Gs for these new boxes, and they only run SHX... Just to make you feel better, the 7604 I reloaded yesterday with SXF15 just spontaneously reloaded... - -- Peter Taphouse Bytemark Hosting http://www.bytemark.co.uk/ tel. +44 (0) 845 004 3 004 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIz8GDIAZ7OKeBB58RAi3zAKCaUsJbYjy6yRwx4796Yv9ko+hXTQCePYEB UaQrHjlsOaFCeXKrjz7yTag= =WgJ0 -END PGP SIGNATURE- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, following up on this: On Tue, Sep 02, 2008 at 08:33:08PM +0300, Adrian Minta wrote: Gert Doering wrote: On Tue, Sep 02, 2008 at 07:02:08PM +0300, Adrian Minta wrote: - the BGP ghost bug is back :-( I have now managed to open a TAC case on this - in case you want to open your own case and attach to it, it's SR 609533003. We have no BugID yet, TAC is trying to reproduce it. In our network, reproducing the problem is fairly straightforward (and I have demonstrated it to the TAC engineer, who then mumbled something like I think you have found a bug here - surprise, surprise :) ): Host H, AS 65500 -- Router A, AS 5539 Router B, AS 5539 Host H announces a single prefix via eBGP to Router A. Router A has a bog standard iBGP session to Router B. No(!) filters of any kind between A and B. Now inject instabilities - tear down the BGP session H-A, bring it up again, wait a few minutes, tear it down again, and so on. After somewhat between 2 and 10 session down, the following happens: - on Router A, the prefix completely drops from the BGP table Cisco-A#sh ip b 193.31.7.1/32 % Network not in table - on Router B, the prefix is still visible, via A: Cisco-B#sh ip b 193.31.7.1/32 BGP routing table entry for 193.31.7.1/32, version 16726560 Paths: (1 available, best #1, table Default-IP-Routing-Table) Flag: 0x820 Advertised to update-groups: 4 65500 195.30.H.H (metric 3584) from 193.149.A.A (194.97.A.A) Origin IGP, metric 0, localpref 100, valid, internal, best it doesn't matter whether next-hop-self is used or whether B is a normal iBGP peer or a route-reflector-client (or whether a mixture of normal peers and RR clients exists). A just forgets - occasionally - to send withdraws if it doesn't have the prefix any longer. Of course A and B have full BGP tables - and as there is instability and withdraws out there, we see this happen to about 5-20 prefixes per day. It might have to do with the amount of normal updates going on in parallel - if there is lots of updates, then there is a propability that things will get lost. But maybe not. (I have not yet tested in a pure lab environment, with no other BGP updates between A and B). gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany [EMAIL PROTECTED] fax: +49-89-35655025[EMAIL PROTECTED] pgp223xLafzZU.pgp Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hi, On Mon, Sep 15, 2008 at 04:50:13PM +0200, Gert Doering wrote: Cisco-A#sh ip b 193.31.7.1/32 % Network not in table Cisco-B#sh ip b 193.31.7.1/32 BGP routing table entry for 193.31.7.1/32, version 16726560 Paths: (1 available, best #1, table Default-IP-Routing-Table) Flag: 0x820 Advertised to update-groups: 4 65500 195.30.H.H (metric 3584) from 193.149.A.A (194.97.A.A) Origin IGP, metric 0, localpref 100, valid, internal, best Oh, and I've found a workaround what to do in this case - since doing a clear ip bgp * soft doesn't work, and a and hard clear of all the iBGP sessions is a bit disturbative... What you need to do is: - insert a static route for that prefix which gets distributed into BGP - remove static route for us, this is straightfoward - we are redistributing static routes - bgp based on route tags, so it's just ip route $prefix $mask tag and remove the route again, 5 seconds later. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany [EMAIL PROTECTED] fax: +49-89-35655025[EMAIL PROTECTED] pgpPpgr1BETdc.pgp Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hello... I'm curious, is bgp dampening on or off? On Mon, 2008-09-15 at 16:50 +0200, Gert Doering wrote: Hi, following up on this: On Tue, Sep 02, 2008 at 08:33:08PM +0300, Adrian Minta wrote: Gert Doering wrote: On Tue, Sep 02, 2008 at 07:02:08PM +0300, Adrian Minta wrote: - the BGP ghost bug is back :-( I have now managed to open a TAC case on this - in case you want to open your own case and attach to it, it's SR 609533003. We have no BugID yet, TAC is trying to reproduce it. In our network, reproducing the problem is fairly straightforward (and I have demonstrated it to the TAC engineer, who then mumbled something like I think you have found a bug here - surprise, surprise :) ): Host H, AS 65500 -- Router A, AS 5539 Router B, AS 5539 Host H announces a single prefix via eBGP to Router A. Router A has a bog standard iBGP session to Router B. No(!) filters of any kind between A and B. Now inject instabilities - tear down the BGP session H-A, bring it up again, wait a few minutes, tear it down again, and so on. After somewhat between 2 and 10 session down, the following happens: - on Router A, the prefix completely drops from the BGP table Cisco-A#sh ip b 193.31.7.1/32 % Network not in table - on Router B, the prefix is still visible, via A: Cisco-B#sh ip b 193.31.7.1/32 BGP routing table entry for 193.31.7.1/32, version 16726560 Paths: (1 available, best #1, table Default-IP-Routing-Table) Flag: 0x820 Advertised to update-groups: 4 65500 195.30.H.H (metric 3584) from 193.149.A.A (194.97.A.A) Origin IGP, metric 0, localpref 100, valid, internal, best it doesn't matter whether next-hop-self is used or whether B is a normal iBGP peer or a route-reflector-client (or whether a mixture of normal peers and RR clients exists). A just forgets - occasionally - to send withdraws if it doesn't have the prefix any longer. Of course A and B have full BGP tables - and as there is instability and withdraws out there, we see this happen to about 5-20 prefixes per day. It might have to do with the amount of normal updates going on in parallel - if there is lots of updates, then there is a propability that things will get lost. But maybe not. (I have not yet tested in a pure lab environment, with no other BGP updates between A and B). gert ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ -- Christopher McCrory The guy that keeps the servers running To the optimist, the glass is half full. To the pessimist, the glass is half empty. To the engineer, the glass is twice as big as it needs to be. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
Hello, I'm curious, is bgp dampening on or off? Just to second (or third?) this bug. We've got four 7600s on SXH3 which are afflicted by this - they were upgraded from 2a on tac's advise (to avoid netflow bug related spontaneous reloads) - and we don't use dampening. It doesn't seem to matter if the prefixes that get withdrawn are i or ebgp, they get still ghosted to other ibgp peers. I don't have any evidence whether or not the prefixes get withdrawn to ebgp peers as we don't transit that many. I've got a case open with tac, but it's causing us enough grief that I'm moving back to SXF until things calm down. Would love the new netflow stuff in SXH if it gets stable enough... -- Peter Taphouse Bytemark Hosting http://www.bytemark-hosting.co.uk tel. +44 (0) 845 004 3 004 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] SXH3 ghost bugs - more details
From what I hear from our account people, SXF is considered a 'dead' train, and you should move to SXH or SXI. We've got a serious NAT bug in SXF14 that they're claiming won't be fixed in SXF. Sucks for our huge Sup2 base. Chuck -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Taphouse Sent: Monday, September 15, 2008 4:11 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] SXH3 ghost bugs - more details Hello, I'm curious, is bgp dampening on or off? Just to second (or third?) this bug. We've got four 7600s on SXH3 which are afflicted by this - they were upgraded from 2a on tac's advise (to avoid netflow bug related spontaneous reloads) - and we don't use dampening. It doesn't seem to matter if the prefixes that get withdrawn are i or ebgp, they get still ghosted to other ibgp peers. I don't have any evidence whether or not the prefixes get withdrawn to ebgp peers as we don't transit that many. I've got a case open with tac, but it's causing us enough grief that I'm moving back to SXF until things calm down. Would love the new netflow stuff in SXH if it gets stable enough... -- Peter Taphouse Bytemark Hosting http://www.bytemark-hosting.co.uk tel. +44 (0) 845 004 3 004 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
