Hi Randal it is really a wired problem but i can suggest 2 causes
1- it might due to VSI interfaces or ARP table limitation problem 2- if you are running PVST , it might be due to PVST instances limitation at this IOS release but to make it clear lets gather some logs and statistcis 1- u mentioned u transferred the affected VLANs to another distribution switch , what is the active number of VLANs on this switch ? 2- as i got from ur description , the switch always drops the traffic of only 2 VLANs randomly . get the output of show vlan and show spanning tree during the problem time 3- u mentioned that the problem solved when u cleared ARP table , can u get show arp | in incomplete to see which entries are incomplete before and after the clearing and which VLAN it belongs to . i hope to hear from u soon . best regards --Abo Zaid On 4/22/08, randal k <[EMAIL PROTECTED]> wrote: > > Hey guys, > I've ran into a ridiculous problem that has me completely stumped. > > Network is a standard edge/core/access/distribution network comprised of > 7206,6509-sup7203bxls, 3550s&3750s, and 3550s/2950s, respectively. > Distribution is pure OSPF, with 226 routes currently in area 0, while the > cores & edges run full mesh bgp. The cores originate defaults for the > distribution layer, distribution layer carries all of the customer > gateways > and communicates those networks to OSPF. > > The distribution 3550-12G in question is running > c3550-ipservices-mz.122-25.SEB4.bin. It's configured with 22 VSIs, carries > all of Area 0 (226 routes), and has 354 mac addresses listed and just shy > of > 300 arp entries. Average traffic through the switch is approximately > 120mbps. Not very loaded. > > This switches decided to randomly stop routing traffic two two completely > separate VSIs (vlan 602, & vlan 149). These two VLANs are attached to the > same port & downstream access switch, G0/4 and a 2960. The Internet can > see > the VSI IP addresses without issue, OSPF still advertises the routes > without > issue, everything is great up to the switch. Hosts attached to the > 3550-12G > are able to see their appropriate VSI gateway IP, but cannot see anything > past it. Attached hosts are, however, able to see all of the other 21 VSI > IP > addresses on the switch -- just nothing off of the switch. No traffic is > able to pass from off-switch/Internet to affected attached hosts, period. > Resolution was to move the VSI/customer gateway to a different > distribution > switch. Although the affected/broken 3550-12G is still in the switching > path, it does Layer 2 forwarding without issue -- just that those 2 VSIs > just stopped forwarding traffic. > > So this morning, we lost two more networks, the primary and secondary IP > address on a VSI for a completely different customer (vlan 609). On a > lark, > I clear arp'd and the two networks came back, but two other different VSIs > went down (vlan 122, 167)! > > The only thing that all of the VSIs have in common is that they are all > servicing customers attached to the 3550-12G's port G0/4. As mentioned > earlier, there was a 2960 switch attached to G0/4, which has been replaced > to no avail. Host configuration on affected VSI makes no difference - > swapping in different servers, my laptop, etc, all yield the same problem. > However, as of right now, if I plug my laptop into an access switch on > g0/7 > configured for the same now-broken vlan 167, it works just fine. It's > almost > as if the VSI's dealing specifically with g0/4 were having problems. > > Fearing a broken g0/4 <-> 2960 trunk, my config has been reduced to 4 > lines, > no change in service: > ! > interface GigabitEthernet0/4 > description down_acc12.fac01.cos > switchport trunk encapsulation dot1q > switchport mode trunk > load-interval 30 > ! > > > If I move the VSI & Gateway to different distribution switch, it works > fine. > If I move the access to a different port, it works fine. I have not > reloaded > the switch yet, as there is some other stuff on there that I don't want to > incur 3-4 minutes of downtime on -- but I am fearing that the problem may > jump and cause more harm. Am I dealing with a randomly screwed up g0/4 > that's smoking VSIs (how?), a buggy IOS that does this or ???. I've been > searching the Internet the world over and would love to hear some ideas > and > anecdotes. > > Thanks for reading my wall of text, > Randal > _______________________________________________ > cisco-nsp mailing list [email protected] > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
