Re: [c-nsp] 2651xm packet loss with 200 ft ethernet link
Just a quick followup, as I got this resolved this morning (after seeing the odd behavior continue on additional fastethernet interfaces on a replacement router). Turns out, the issue was ultimately with the wiring between the Level3 fuji ports and the jack panel where they officially hand us off the circuits. We bypassed the panel, terminating directly into their mux card, and the packet loss vanished. Still doesn't tell me why the 7200 interface was able to power it's way through whatever was wrong...it's an older PA-FE-TX so I'm guessing the old DEC chips are cranked up all the way to deal with older crappy wiring, and it had enough juice to power through whatever is faulty in the jackpanel. Oh well. Fixed now. Thanks for all the replies on and off list suggesting avenues for narrowing down the problem. Andy On Fri, 28 Dec 2012, Andy Dills wrote: I'm having a weird problem that I can't quite seem to wrap my head around. I have customer who has a 2651XM that we're hooking up to a Level3 (copper) ethernet circuit. Because of the distance between our telco handoff and his rack, he has about 200 feet. Which shouldn't be a problem. However, we were seeing tremendous packet loss. After determining the errors were coming from this end, and testing and replacing everything in the path, we finally tried moving the router into the telco handoff room so we could try to eliminate a lot of variables. With a short patch cable, it worked fine, no packet loss. So, we made a 200 ft cable to simulate the issue, and hooked it up to the router while it sat next to the Level3 handoff...major packet loss again. We tried the other fastethernet interface on the 2651...same problem. However, if I take that same cable and terminate it into one of our 7200s, we get zero packet loss. In sum: This 2651xm with long cable, major packet loss This 2651xm with short cable, works fine Our 7200 with the same long cable, works fine What's going on here? Is the controller in the 2651xm failing? Perhaps not producing enough voltage? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] 2651xm packet loss with 200 ft ethernet link
I'm having a weird problem that I can't quite seem to wrap my head around. I have customer who has a 2651XM that we're hooking up to a Level3 (copper) ethernet circuit. Because of the distance between our telco handoff and his rack, he has about 200 feet. Which shouldn't be a problem. However, we were seeing tremendous packet loss. After determining the errors were coming from this end, and testing and replacing everything in the path, we finally tried moving the router into the telco handoff room so we could try to eliminate a lot of variables. With a short patch cable, it worked fine, no packet loss. So, we made a 200 ft cable to simulate the issue, and hooked it up to the router while it sat next to the Level3 handoff...major packet loss again. We tried the other fastethernet interface on the 2651...same problem. However, if I take that same cable and terminate it into one of our 7200s, we get zero packet loss. In sum: This 2651xm with long cable, major packet loss This 2651xm with short cable, works fine Our 7200 with the same long cable, works fine What's going on here? Is the controller in the 2651xm failing? Perhaps not producing enough voltage? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 2651xm packet loss with 200 ft ethernet link
On Fri, 28 Dec 2012, Pete Lumbis wrote: Are you terminating to the onboard ethernet port or a WIC? What code version? Both onboard ports, no WICs installed. It's running 12.4(25d). Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550-12 interrupts out of control, possibly hardware?
Just to follow up on this on the off chance somebody runs into this in the future...a reload of the switch fixed the issue, and the traffic for the L3 port stopped hitting vlan 1. Andy On Thu, 16 Aug 2012, Andy Dills wrote: Thanks, I appreciate those suggestions. I verified both the SDM and VTP configs are identical. Did you see my followup from earlier? I identified that for some reason unknown to me, the traffic was hitting the vlan1 interface before exiting via the L3 interface facing that network, which was forcing all of the traffic to get process switched. I have no idea why, though, and would love suggestions. My best guess is that because they configured the port for L3 mode before they enabled ip routing on the failover 3550-12, something didn't happen right and perhaps a reload would have fixed it. I do know that in the past when I have done ip routing on a live 3550, it goes unresponsive for about 10-15 seconds, so I have to assume a lot goes on behind the scenes. And I do know from the transcript of their changes that they configured the port for L3 mode before realizing ip routing had never been enabled on that switch. Given the illogical (in quotes because perhaps there is some logic that is escaping me) nature of the behavior observed, I have to assume it was some sort of quirk of bug like this. For what it's worth, they're both running c3550-ipservices-mz.122-44.SE6. Thanks, Andy On Fri, 17 Aug 2012, Tóth András wrote: Hi Andy, One idea is different SDM templates being used. The SDM template is not showing up in running-config, and changing it requires a reload as well. I would compare them with 'sh sdm prefer' command. You might be running out of IPv4 routes, which causes rest of routes to be applied in software, so packets are software switched by the CPU which can cause high utilization. http://www.cisco.com/en/US/products/hw/switches/ps646/products_tech_note09186a0080094bc6.shtml http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.2_44_se/configuration/guide/swadmin.html#wp1235565 Best regards, Andras On Thu, Aug 16, 2012 at 6:36 PM, Andy Dills a...@xecu.net wrote: I've got a customer with a weird situation. They have a pretty straightforward setup, two 7200s fronting two cisco 3550-12s, distributing to a series of 48 port 3550s. It's a bit dated, but works very well for their needs. They have one special network attached to (only) one of the copper gige ports on (one of) the 3550-12s which gets a decent amount of traffic (~100mbps or so). It's a layer 3 connection. Well, one of their 3550-12s died, taking down that network. They moved the IP configuration of the port and moved the cable immediately, restoring service, and racked/configured a replacement switch, but left that network on the second 3550-12, as it seemed fine. However, once it began to come under load this morning, the CPU pegged (80-99%, normally at 1-2%), causing packet drops and latency. At that point I got involved, and for the life of me I can't figure out why this happened. Clearly it's interrupts, as there were no processes in the sh proc cpu that had more than 1% of CPU. However, cef was working fine, everything looked normal in terms of the traditional interrupt-based troubleshooting. So, after scratching our heads for a bit, I had them move the connection back to the original, newly-replaced switch. Note that these switches are configured 100% identically with the exception of IP address and hostname. Same IOS versions. I mean literally, if you diff the two in rancid, those are the only config changes. Zero problems from the point they moved the connection off of the switch in question, both switches now have 1-2% CPU and things are humming along fine. So, my question is: What could be the possible causes of this? Could this be a symptom of failing hardware, perhaps some bad memory requiring constant CPU corrections? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] 3550-12 interrupts out of control, possibly hardware?
I've got a customer with a weird situation. They have a pretty straightforward setup, two 7200s fronting two cisco 3550-12s, distributing to a series of 48 port 3550s. It's a bit dated, but works very well for their needs. They have one special network attached to (only) one of the copper gige ports on (one of) the 3550-12s which gets a decent amount of traffic (~100mbps or so). It's a layer 3 connection. Well, one of their 3550-12s died, taking down that network. They moved the IP configuration of the port and moved the cable immediately, restoring service, and racked/configured a replacement switch, but left that network on the second 3550-12, as it seemed fine. However, once it began to come under load this morning, the CPU pegged (80-99%, normally at 1-2%), causing packet drops and latency. At that point I got involved, and for the life of me I can't figure out why this happened. Clearly it's interrupts, as there were no processes in the sh proc cpu that had more than 1% of CPU. However, cef was working fine, everything looked normal in terms of the traditional interrupt-based troubleshooting. So, after scratching our heads for a bit, I had them move the connection back to the original, newly-replaced switch. Note that these switches are configured 100% identically with the exception of IP address and hostname. Same IOS versions. I mean literally, if you diff the two in rancid, those are the only config changes. Zero problems from the point they moved the connection off of the switch in question, both switches now have 1-2% CPU and things are humming along fine. So, my question is: What could be the possible causes of this? Could this be a symptom of failing hardware, perhaps some bad memory requiring constant CPU corrections? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550-12 interrupts out of control, possibly hardware?
In doing further investigation, looking at traffic graphs, I see that once they moved the network to the other switch, all of a sudden vlan1 started seeing all of the traffic that was being routed to that network. Typically, the only traffic the switches see on vlan1 is traffic actually destined for the switch (config, ICMP, etc). And the switch they are currently on does not see any traffic on the vlan1, and once I had them move the connection, neither switch sees the big traffic spike on vlan1 any longer. This is quite odd, because as I mentioned, the two switches are configured the same...can anybody suggest an explanation or potential course of determining why the traffic that I'm wondering if it's some odd software bug relating to them enabling ip routing on that second switch last night, but not booting fresh after doing so. I just can't puzzle out in my head why traffic destined for an L3 port would transit the VLAN like that. Thanks, Andy On Thu, 16 Aug 2012, Andy Dills wrote: I've got a customer with a weird situation. They have a pretty straightforward setup, two 7200s fronting two cisco 3550-12s, distributing to a series of 48 port 3550s. It's a bit dated, but works very well for their needs. They have one special network attached to (only) one of the copper gige ports on (one of) the 3550-12s which gets a decent amount of traffic (~100mbps or so). It's a layer 3 connection. Well, one of their 3550-12s died, taking down that network. They moved the IP configuration of the port and moved the cable immediately, restoring service, and racked/configured a replacement switch, but left that network on the second 3550-12, as it seemed fine. However, once it began to come under load this morning, the CPU pegged (80-99%, normally at 1-2%), causing packet drops and latency. At that point I got involved, and for the life of me I can't figure out why this happened. Clearly it's interrupts, as there were no processes in the sh proc cpu that had more than 1% of CPU. However, cef was working fine, everything looked normal in terms of the traditional interrupt-based troubleshooting. So, after scratching our heads for a bit, I had them move the connection back to the original, newly-replaced switch. Note that these switches are configured 100% identically with the exception of IP address and hostname. Same IOS versions. I mean literally, if you diff the two in rancid, those are the only config changes. Zero problems from the point they moved the connection off of the switch in question, both switches now have 1-2% CPU and things are humming along fine. So, my question is: What could be the possible causes of this? Could this be a symptom of failing hardware, perhaps some bad memory requiring constant CPU corrections? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550-12 interrupts out of control, possibly hardware?
Thanks, I appreciate those suggestions. I verified both the SDM and VTP configs are identical. Did you see my followup from earlier? I identified that for some reason unknown to me, the traffic was hitting the vlan1 interface before exiting via the L3 interface facing that network, which was forcing all of the traffic to get process switched. I have no idea why, though, and would love suggestions. My best guess is that because they configured the port for L3 mode before they enabled ip routing on the failover 3550-12, something didn't happen right and perhaps a reload would have fixed it. I do know that in the past when I have done ip routing on a live 3550, it goes unresponsive for about 10-15 seconds, so I have to assume a lot goes on behind the scenes. And I do know from the transcript of their changes that they configured the port for L3 mode before realizing ip routing had never been enabled on that switch. Given the illogical (in quotes because perhaps there is some logic that is escaping me) nature of the behavior observed, I have to assume it was some sort of quirk of bug like this. For what it's worth, they're both running c3550-ipservices-mz.122-44.SE6. Thanks, Andy On Fri, 17 Aug 2012, Tóth András wrote: Hi Andy, One idea is different SDM templates being used. The SDM template is not showing up in running-config, and changing it requires a reload as well. I would compare them with 'sh sdm prefer' command. You might be running out of IPv4 routes, which causes rest of routes to be applied in software, so packets are software switched by the CPU which can cause high utilization. http://www.cisco.com/en/US/products/hw/switches/ps646/products_tech_note09186a0080094bc6.shtml http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.2_44_se/configuration/guide/swadmin.html#wp1235565 Best regards, Andras On Thu, Aug 16, 2012 at 6:36 PM, Andy Dills a...@xecu.net wrote: I've got a customer with a weird situation. They have a pretty straightforward setup, two 7200s fronting two cisco 3550-12s, distributing to a series of 48 port 3550s. It's a bit dated, but works very well for their needs. They have one special network attached to (only) one of the copper gige ports on (one of) the 3550-12s which gets a decent amount of traffic (~100mbps or so). It's a layer 3 connection. Well, one of their 3550-12s died, taking down that network. They moved the IP configuration of the port and moved the cable immediately, restoring service, and racked/configured a replacement switch, but left that network on the second 3550-12, as it seemed fine. However, once it began to come under load this morning, the CPU pegged (80-99%, normally at 1-2%), causing packet drops and latency. At that point I got involved, and for the life of me I can't figure out why this happened. Clearly it's interrupts, as there were no processes in the sh proc cpu that had more than 1% of CPU. However, cef was working fine, everything looked normal in terms of the traditional interrupt-based troubleshooting. So, after scratching our heads for a bit, I had them move the connection back to the original, newly-replaced switch. Note that these switches are configured 100% identically with the exception of IP address and hostname. Same IOS versions. I mean literally, if you diff the two in rancid, those are the only config changes. Zero problems from the point they moved the connection off of the switch in question, both switches now have 1-2% CPU and things are humming along fine. So, my question is: What could be the possible causes of this? Could this be a symptom of failing hardware, perhaps some bad memory requiring constant CPU corrections? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Possible to make NAT decisions based on source address, on ASA?
Hi there, I'm looking to do something along the following, and while in my head it should be doable, I can't seem to find a mechanism to enable me to configure it: I want users, accessing a public address on the ASA (let's call it 5.5.5.5) from subnet A out on the public side, let's say from a specific remote office (20.0.0.0/24), to get NATed to private server 10.0.0.100, where everybody else gets NATed to 10.0.0.200. So: 20.0.0.0/24 - 5.5.5.5 gets NATed to 10.0.0.100 0.0.0.0/0 - 5.5.5.5 gets NATed to 10.0.0.200 So, in essence, I want to consider source address when determining which server on the private network the traffic is NATed to. Is this possible? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Possible to make NAT decisions based on source address, on ASA?
On Thu, 17 May 2012, Peter Rathlev wrote: On Thu, 2012-05-17 at 14:36 -0400, Andy Dills wrote: So, in essence, I want to consider source address when determining which server on the private network the traffic is NATed to. Is this possible? No problem. Take a look at Configuring Dynamic NAT or Dynamic PAT: http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/nat_dynamic.html#wp1081940 This is for 8.2 and earlier with the old NAT configuration style. With version 8.3 or later the commands are different. Quick example: ! Policy NAT 20.0.0.0/24 towards 5.5.5.5 access-list PolicyNAT-example permit ip 20.0.0.0 255.255.255.0 host 5.5.5.5 nat (inside) 1 access-list PolicyNAT-example global (outside) 1 10.0.0.100 ! Regular NAT everything else nat (inside) 2 0.0.0.0 0.0.0.0 global (outside) 2 10.0.0.200 Yeah, I had looked at that, and it's not quite what I'm trying to accomplish. What I want is to take a single public IP and NAT it to two seperate private IPs, based on source address of the incoming request. As best I can tell policy NAT is used in situations (such as what you describe above) where you're trying to dynamically control the source of queries after translation... Thanks for your input, and for any other suggestions. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] ASA SSL VPN client communicating across IPsec tunnel
I have a customer who has a couple of ASA 5510s connected with a typical IPsec tunnel, and on one of them he has a 10 seat Anyconnect SSL license. He'd like for the Anyconnect VPN users to be able to communicate with the network on the other side of IPsec tunnel. In theory that would work, but I've found the ASAs to sometimes ignore theory. I updated the NAT exemption ACL (to include traffic from the VPN users to the remote network and vice versa), the split-tunnel ACL (to have it advertise the remote network in addition to the local), and the crypto map ACL (so that the VPN users are included in the ipsec sa). It didn't seem to work...I didn't have good access to test, but before I arrange for better access to really work with it, is this indeed possible? Any configuration tips? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] DS3 Nubie
On Sat, 25 Sep 2010, Christopher J. Wargaski wrote: Hey Jeff-- This year I installed a video WAN comprised of several 3845 routers with the NM-1T3/E3 for point to point DS3s (that is all, nothing else). The 3845 routers list at $13,000 and the network module at $8,500. You should do yourself and your firm a favor and look at business class Ethernet. DS3s are so expensive and sometimes a major pain in the butt. ...or you can buy a 7204 with a DS3 adapter on the secondary market for a tenth of that cost. That said, ethernet is getting cheaper and more widely deployed. We were very pleased to discover that Verizon was finally offering this service in our LATA. NxT1 now upgrades to ethernet instead of frac DS3 for our customers. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Migrating to a 7200 for DSL aggregation
For the last decade, we've been using a Redback SMS 500 to terminate our DSL customers, delivered via ATM DS3. Recently, however, a few customers (as well as the partners of the company) have expressed a desire to bond 2 DSL circuits. Unfortunately, we've discovered that the customers who have Comtrend modems cannot negotiate LCP with the Redback when ppp multilink is enabled...after working through this with Comtrend, they demonstrated that the Redback is violating RFC 1990, and because of this and the fact that the Comtrend modem didn't support MP, it was unable to negotiate LCP. At the same time, we're starting to get a little nervous about relying on Redback as a platform, given that all the equipment we currently use is EOL and Redback has long since been acquired. We have spares, but clearly it's a deadend. So, we're investigating migrating to a 7200(VXR) platform for DSL aggregation. I was curious about a couple of things: 1) Given that Verizon delivers all of the DSL connections on a region-based series of UBR PVCs, what is the best way to bond two DSL circuits, given that they would not be on private PVCs? Is MLPPP possible in that configuration? Or would I need to do CEF per-packet? Given the archives on this list, I'm leaning toward per-packet CEF via radius assigned routes. I know we won't get ideal performance, but it should be an improvement over a single DSL connection nonetheless. 2) What IOS do people suggest for ATM DSL aggregation...the router won't be doing anything else other than OSPF. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] problems migrating to a 3550
I'm migrating a network from an old HP Procurve switch to a Cisco 3550. Simple setup, public and private vlans. Setup a port to be tagged on both vlans on the HP side, and on the cisco end set it to be in trunking mode. The cisco sees the vlans. I'm getting the full table from 'show mac address-table', with the appropriate vlans attached to the appropriate mac addresses. Things in vlan2 on the HP switch can reach the IP address of the 3550 on vlan2 just fine, vlan2 is solid. However, things in vlan1 on the HP switch cannot reach the IP of the 3550 on vlan1, and anything attached to 3550 on vlan1 ports cannot reach anything on vlan1 on the HP switch. Both switches have all of the correct mac addresses in their layer 2 forwarding table. However, whereas things on vlan2 are consistently reachable and populate the arp table, on vlan1 some things will show up in the arp table, most will not, and none will be pingable. Another symptom I'm noticing is that nothing on vlan1 on the HP switch can see the mac address for the vlan1 interface on the 3550, or of anything attached to vlan1 of the 3550. However, these mac addresses will be in both switches forwarding tables. And likewise, there will be addresses in the forwarding table of the 3550, but somehow the server is unable to get arp resolution for any of those very hosts. Config: ! interface FastEthernet0/48 switchport trunk encapsulation dot1q switchport trunk allowed vlan 1,2 switchport mode trunk ! ! interface Vlan1 description Public ip address 10.0.0.126 255.255.255.128 ! interface Vlan2 description Private ip address 10.0.0.254 255.255.255.128 ! ip route 0.0.0.0 0.0.0.0 10.0.0.1 public#sh interfaces trunk PortMode Encapsulation StatusNative vlan Fa0/48 on 802.1q trunking 1 PortVlans allowed on trunk Fa0/48 1-2 PortVlans allowed and active in management domain Fa0/48 1-2 PortVlans in spanning tree forwarding state and not pruned Fa0/48 1-2 Running Version 12.2(44)SE6. Any suggestions? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] IOS Version for 7206VXR
We just picked up a 7206VXR w/NPE-G2. It currently has 12.4(15)T1, which looks pretty old. This won't be used for anything crazy...bgp, ospf, access lists. It's a box full of ethernet interfaces that will push packets. The router its replacing runs 12.0(31)S, to give you an idea of the goals involved: Stability, PPS, stability, stability. What would people suggest for a nice, stable IOS? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Netflow analyzer suggestions
Hi there, Apologies for the non-cisco specific post, but I couldn't think of a better list to post to off of the top of my head. A friend of mine is looking to get some better visibility into their network usage (low bandwidth, T1 type scenario). I got them setup with an IOS that supports netflow v9, and since they had a freebsd box being barely used, I built ntop to be their collector/analyzer. It works reasonably well, but it's perhaps not quite...user friendly enough for them. What they're looking for is a graphical representation at the host level. In essence, they're looking for something like cricket/mrtg but with each IP having its own graph (rather than each interface). Additional features, such as protocol breakdown and so forth are helpful, but the primary desire is to be able to see how much bandwidth a given IP address is using currently and was using in the past. They're open to commercial solutions, but would prefer to keep costs down. Host operating system requirements for the collector/analyzer aren't important. Does anybody have any suggestions they could pass along? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] CRS-1 too complicated?
On Sat, 26 Jan 2008, Frank Bulk wrote: It sounds like they want to guarantee that each of these sales turns into a successful installation. It's relationship and satisfaction protection. If that's the goal, why not price it into the product and tell them the install is free? The cost of installation must be a very small percentage of the total cost, it's not like price is a primary factor for the target market. Then, not only do you guarantee the customer's satisfaction, you remove the issue of some customers feeling forced to buy something they aren't completely sure they need and replace it with appreciation. Basically, if it's mandatory, why make it a line item... Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] telecommuting support jobs for BGP guys?
On Fri, 25 Jan 2008, neal rauhauser wrote: There are thriving markets for remote workers in the software field but its far less common for infrastructure support. Regarding accessibility in the event of BGP troubles all of my customers have a little collection of static /32s on their border routers coming back to a couple of different hosts - troubles do arise, but I've never been completely shut out. If it were genuinely that serious there would be a dial in line to the facility in place, or better yet DSL from a totally unrelated carrier. I guess the problem might be a shortage of networks who utilize BGP yet don't have somebody on staff or an existing relationship with a consultant to manage their BGP. Besides, correct me if I'm wrong, but don't the vast majority of networks that use outside help with BGP have relatively static configurations? Best of luck regardless. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Strange DS3 Problems
On Mon, 7 Jan 2008, Adam Piasecki wrote: 1) The max download speed i've ever gotten is 700kb/s, this doesn't seem right at all for 10/10mb dedicated, 45mb burstable. Is there anything you can download directly on the other end of the DS3, to confirm it's an issue with your DS3 and not with upstream capacity? Also, sorry to be pedantic, but you're sure it's 700kbps and not 700kBps? 2) the Ds3 is bouncing. It goes into a down/down state and i'll have a FERF alarm light on the router. This will clear in about 20-30secs without me doing anything. It seems to happen more when the Ds3 is under load, though it has happend without any load. What does the provider see on their interface stats? What I can do, 1) I can ping the living daylight out of the ISP serial side IP, 2 pings 3700byte packets with 0% lost, 3ms max. In past this to me, would mean a clean working ds3. Just to be safe, try doing a range sweep with 0x (alternating 1s and 0s). DS3s are a lot more sensitive to alternating bits, and will help expose seemingly solid circuits. Might turn out that you need to increase attentuation, wouldn't be the first time. 1) What is the rx FEBE since last clear counter mean? I notice this is the only sort of error i'm getting now. It's an indicator that the far end received a c-bit parity error. It's possible you need to attenutate the tx from your router to the fiber converter, in addition to the rx. 2) Is a 7206 non-VXR NPE 200 strong enough for a full DS3, if i'm just doing a default route, and a very small ACL. Definitely. 3) Is it possible that i have a BAD PA-T3, I did have it connected without the Attunator for sometime, Have you done loopback testing with a short hard loop cable? It generally eliminates the hardware. See: http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a008034418d.shtml 4) Has anyone used Fiber to DS3 converters in the past? Good/Bad? I have not, so in my mind that's the most suspicious issue, especially in the absence of detected errors on the ISP side. cablelength 10 Try making that 50, see what happens. Probably nothing, but just in case. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] OT: Pingsta spam
On Wed, 2 Jan 2008, Robert Blayzor wrote: Pingsta is clearly breaking federal laws. http://www.ftc.gov/bcp/conline/pubs/buspubs/canspam.shtm It's obvious they are harvesting email addresses from this list. Regardless if someone was carefully selected to receive their mailings, if someone didn't opt in, it's spam. I have no idea who pingsta is, nor do I like spam, but I don't think you can quite say it's obvious they are harvesting email addresses and accuse them of breaking a federal law without posting some evidence to support that. To me, it's not obvious because I've been posting to cisco-nsp for years and can't remember getting an email from them. That doesn't mean you're not correct, but you have to admit you're being a little over the top. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cisco ASA static tcp forwarding question
On Sat, 29 Dec 2007, Michael Smith wrote: Hello All: Is it possible to have a scenario where traffic coming in to a server on either port 443 or port 80 is sent to the inside host only on port 443? Something like: static (inside,outside) tcp x.x.x.x https 192.168.1.1 https static (inside,outside) tcp x.x.x.x http 192.168.1.1 https The above commands don't work because a previous entry is seen when trying to add the second so I'm curious if anyone has gotten this working and how? This configuration is on a 5510 running 8.0(3). You have to have one-to-one correspondence, the IP/port outside/inside addresses cannot have multiple static bindings. Otherwise, the ASA wont know which rule to use on the reply packets. Bind an additional IP to the server at 192.168.1.1, say 192.168.1.2, and then: static (inside,outside) tcp x.x.x.x https 192.168.1.1 https static (inside,outside) tcp x.x.x.x http 192.168.1.2 https Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] OT: How do you fight spam in your enterprise? I needhelp
On Thu, 20 Dec 2007, Ted Mittelstaedt wrote: The expensive commercial spamfiltering solutions only make sense for mid-tier ISPs, that is, the ISPs that have networks too big for a single admin to do everything, but are not large enough to be capitalized to the extent that they can hire a programming team to just chase spam. They have enough money to pay a commercial firm to do it, but not enough money to hire a warm body and put them on staff to do it. Our solution: FreeBSD boxes running postfix interfacing with amavisd-new, which scans the mail with ClamAV (with the additional 3rd party dbs), and also with spamassassin (with DCC, RAZOR, FuzzyOCR). L4 switch on the front, MySQL and NFS on the back...private DCC as well as DNS mirroring of the RBLs. Custom web interface for the customers to enable individual management of filter settings and white/black lists. Tools to monitor the queue sizes. I would consider this a very commonly used solution, it's not like we're doing anything special. While installing, configuring, and tweaking everything from scratch does take every bit of 5 hours, perhaps several days if you aren't familiar with the process, implementing additional servers to accomodate the increasing load takes us less than 30 minutes, as they are implemented by booting the FreeBSD install disk, going into a fixit shell, mounting a fileserver, and restoring from a dump (changing a couple of config files). Takes about 30 minutes total, most of which is waiting for the restore to complete. I don't think the amount of time required to manage the actual mail infrastructure (the abuse mail being a seperate issue) scales with volume, unless you implement a solution that doesn't scale. I would assume most of the companies using a commercial mail product are companies without technical talent. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [ASAP] STP port-state
On Thu, 20 Dec 2007, Hiromasa Sekiguchi wrote: Hi, In what situation does Port-State output type-inconsis? We use cat6k. (enable) sh spantree VLAN 1 Spanning tree mode PVST+ Spanning tree type ieee Spanning tree enabled Designated Root ** Designated Root Priority32769 Designated Root Cost0 Designated Root Port1/0 Root Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec Bridge ID MAC ADDR ** Bridge ID Priority 32769 (bridge priority: 32768, sys ID ext: 1) Bridge Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec Port Vlan Port-StateCost Prio Portfast Channel_id - - -- 2/1 1type-inconsis 4 32 disabled 0 ^ Is type-inconsis the same meaning as type-pvid-inconsistent? This URL should help you identify what is causing the STP inconsistency: http://www.cisco.com/warp/public/473/pvid_inconsistency_24063.html Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Rate-Limit Problem
On Thu, 20 Dec 2007, Paul Stewart wrote: Can anyone tell me why this doesn't work? We have this implemented in dozens of other locations and it works fine... rate-limit input 420 2100 2100 conform-action transmit exceed-action drop Your burst buckets are too small. The general formula you will find to yield the desired bandwidth is: rate limit direction A B C conform-action transmit exceed-action drop A: Your desired bps B: 1.5*A/8 C: 2*B What's happening is you don't allow enough burst, which causes severe TCP sawtoothing (where it drops several packets at the max bps, and which then causes a fresh slow start, thus making a sawtooth pattern on a graph). It seems somewhat counterintuitive to allow so much burst, but once you configure it and try it out you will probably find it does what you want, and under load testing you'll see a nice line on the graph at the desired rate. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] RSP redundancy with SSO?
On Tue, 18 Dec 2007, neal rauhauser wrote: Can anyone comment on RSP redundancy with SSO mode? I have a test 7507 with rsp-k4pv-mz.120-32.S8.bin on both RSP2s. I've done this: service single-slot-reload-enable redundancy no keepalive-enable mode sso And it doesn't seem to ever sync the slave Try adding: hw-module slot 2 image slot0:/rsp-k4pv-mz.120-32.S8.bin hw-module slot 3 image slot0:/rsp-k4pv-mz.120-32.S8.bin slave auto-sync config And then (make sure to wr mem first): hw-module standby-cpu reset Peer (slot: 3) information is not available because it is in 'DISABLED' state Sounds like the slave RSP didn't boot, I'm guessing because of one of the above. Make sure the image exists in both slot0 and slaveslot0. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Ethernet debugging...MTU?
On Sat, 27 Oct 2007, Pekka Savola wrote: On Fri, 26 Oct 2007, Andy Dills wrote: Cliff notes: Is the inability to get ping replies with datagrams of larger than 9216 bytes across a 100mbps ethernet circuit an indication that the far end is setup with an MTU consistent with jumbo frames? What to do if the far end swears it's set for 1500? (Note that 100mbit/s FE interfaces may also support jumbo frames so the fact that it's FE interfaces isn't a guarantee that it won't be using 9216 MTU.) When you ping with 9216 bytes, most likely your router fragments the ping packets to multiple datagrams (maximum at 1500 bytes; verify this). When the target responds, how are the response datagram(s) fragmented? You should be able to figure out the path MTU by checking the largest size of the fragments received. You may need to ping from a unix host in order to more easily diagnose this issue (e.g., run tcpdump to see the fragment offsets). When I start up a ping and trap the responses with tcpdump, I find that the responses are indeed fragmented at 1500 bytes. That said, I don't get any reply packets once the 9216 byte barrier is surpassed. Based on that you should be able to figure out what IP mtu the remote end is using. If it's set to 1500 but ping (with 9216) still fails the reason might have more to do with the fragmentation capabilities/policies of fragmenting devices (routers or the hosts at each end). This appears to be the case. I'll focus on this. It's just so odd that the fragmentation problem occurs at the boundary size for jumbo packets. The original reason I discovered this is, when we try to pass a couple of mbps across the link, we start getting significant packet loss...with no incrementing error counters anywhere on any layer. So I started trying different things and discovered the issue I mentioned in the email, as I have to assume once we resolve that issue the overall issue of packet loss will be fixed. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Ethernet debugging...MTU?
Ok, I have a weird situation that I'd like to get some input on. Cliff notes: Is the inability to get ping replies with datagrams of larger than 9216 bytes across a 100mbps ethernet circuit an indication that the far end is setup with an MTU consistent with jumbo frames? What to do if the far end swears it's set for 1500? Background: We're in the process of turning up a new fast-e to a company located in Equinix. We are not located in Equinix, but Level3 is, and we have Level3 fiber in our datacenter. There are two major segments of this circuit: the long haul to Ashburn and the cross connect in Equinix. The run at Equinix is too long for copper, and Level3 for some reason insisted upon a copper handoff, so Equinix supplied and installed transceivers to enable a fiber x-con that is delivered via copper to the cage. In our datacenter, the ethernet circuit is connected directly via a short copper run from Level3's space to a standard FE port on a Cisco router, that has previously been in use and is known to be working. If it matters, for now it's on a PA-2FEISL-TX (but will later be moved to a more current PA when put into production...to my knowledge, even though that PA isn't ideal, it should still work fine at lower bandwidth levels and with proper full-duplex, etc. The previously attached customer had no problems pushing 30mbps, for example). When testing the circuit, using the cisco ping utility, with datagrams of 9216 bytes or less, we have no packet loss. When datagrams larger than 9216 bytes are used, we have 100% packet loss. Given the packet size when total failure occurs, my first reaction is that the other company has somehow misconfigured their switch to use jumbo frames, as if the circuit was a gig-e. According to the company at the far end, their device doesn't even support jumbo frames, and other people are attached and working fine. They seem to be quite certain the problem isn't on their end. I have a hard time not believing them; checking the MTU is a pretty cut and dry thing, and I'm working with senior level people at the other company, who I'm assuming (perhaps an incorrect assumption) run and manage their international network. To narrow down the problem, we have had Level3 setup a laptop in their cage at Equinix, attached to the interface facing our datacenter. When testing to that, I had no packet loss with packets of any size up to the maximum of 18024 bytes. To me this eliminates the long-haul portion from consideration. We've also had Equinix double check (at the supervisor level) that the transceivers are 10/100, hardcoded for 100 full-duplex (as is everything else end to end). I also have a hard time believing that Equinix would have any difficulties installing the correct model and properly configured ethernet transceivers; Ashburn is a top notch facility with good people. (I was hoping with my fingers crossed they had accidently installed gig-e transceivers, or that Level3 had accidentally ordered gig-e transceivers...no luck). As this has been a lingering issue for some time, I'm currently pushing for technical reps from all of the companies involved to meet up at Equinix, sit in a room, and figure it out. This is a bit difficult for the other company as they don't have any real local staffing. So, I'm hoping to come up with a solution that doesn't involve them getting on an airplane, but it's starting to look like our only avenue of resolution. That said, has anybody encountered this before, or has any theories about ways I can debug this short of having the other company (who doesn't have local staff) visit their cage and attach a laptop facing us, to localize the issue to either their switch or the Equinix cross connect? Am I likely correct in my theory that something is configured for the jumbo frame MTU and thus response packets aren't being properly fragmented? Thanks for any insight. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] BGP path preference
On Tue, 28 Aug 2007, Justin Shore wrote: Does anyone have any suggestions on how to work around this? L3 will eventually fix it when they eliminate 19094 but who knows when that will be. I thought about trying to use a regex to match 19094 3356 to raise local pref even higher. I also see plenty of routes that from 3356 directly to 22773 (L3 peering w/ Cox wo/ Telcove in the middle). I could try to match routes that originate on 3356 and 19094 and raise the local pref of those prefixes. Do I need to raise it on one AND lower it on the other border router or would doing it on one suffice and be manageable? Would this regex matching be a good best (better?) practice for all circuits over matching an ACL or prefix-list of prefixes? If so would one match on origin or simply if the ASN was in the path? Don't forget that you can prepend incoming announcements as well as outgoing announcements. For instance, to account for the fact that there is essentially an extra AS in your transit path to 3356, you might just prepend a single 22773 to everything Cox announces you. You might find that to be a more useful tool in this scenario than the heavy-handed localpref and weights. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
