This all has to be a problem on the SBC. I've only been peripherally involved up until now and I just found out they have another SBC in the same VLAN on the same pair of switches and it is not having a problem and is not experiencing this mad MAC address behavior at all. It's perfectly stable. That pretty much proves to me beyond any reasonable doubt that this is a problem with the SBC and the switch itself is not contributing to the problem in any way.
On Mon, Jun 10, 2013 at 8:57 AM, John Neiberger <[email protected]>wrote: > Here's an example of what I'm talking about. The MAC learning log has been > stable since they reloaded the SBC on Saturday: > > show ethernet-switching mac-learning-log | match 3c:91 > Sat Jul 23 12:05:22 2011 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > learned on xe-0/1/1.0 > Sat Apr 13 04:24:15 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > deleted on xe-0/1/1.0 > Sat Apr 13 04:25:10 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > learned on ge-0/0/5.0 > Sat Apr 13 04:25:11 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > deleted on ge-0/0/5.0 > Sat Apr 13 04:25:18 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > learned on xe-0/1/1.0 > Sat Jun 8 04:27:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > deleted on ge-0/0/5.0 > Sat Jun 8 04:28:08 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > learned on ge-0/0/5.0 > Sat Jun 8 04:28:09 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > deleted on ge-0/0/5.0 > Sat Jun 8 04:28:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was > learned on ge-0/0/5.0 > > > However, we have a firewall filter on ge-0/0/6 and ge-0/0/8 to watch for > that MAC address and its counters are incrementing. They have incremented > several times this morning already: > > Filter: ACME-ge-0/0/6.0-i > Counters: > Name Bytes > Packets > from-ACME-3c91-ge-0/0/6.0-i 1090 > 5 > from-others-ge-0/0/6.0-i 215535790813 > 981754726 > > Filter: ACME-ge-0/0/8.0-i > Counters: > Name Bytes > Packets > from-ACME-3c91-ge-0/0/8.0-i 15042 > 69 > from-others-ge-0/0/8.0-i 205829895148 > 944189465 > > We also have a trace log running that shows this MAC jumping around. Here > is the log output from just the last few minutes: > > Jun 10 14:30:31.546683 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/7.0, pnac_status 0, 0 > Jun 10 14:30:31.546978 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:31:31.826971 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:32:32.535585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:32:32.536080 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:36:43.660324 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:36:43.660961 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:36:55.239016 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:36:55.239425 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:37:29.244518 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:38:32.854226 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:38:32.854532 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:39:36.204933 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:39:36.205232 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:40:11.707066 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:40:37.537264 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:40:37.537562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:43:18.421993 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:43:18.422492 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:43:59.154585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:44:00.000894 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:44:00.001405 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:44:47.068248 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:44:47.068539 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:45:36.634048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:47:27.367950 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:47:27.368359 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:48:15.151240 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:48:15.151530 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:48:51.587305 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:50:00.933896 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:50:00.934196 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:54:43.553048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:54:43.553342 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > Jun 10 14:54:50.862562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/8.0, pnac_status 0, 0 > Jun 10 14:54:50.862887 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, > ifname ge-0/0/5.0, pnac_status 0, 0 > > If the MAC address really is jumping around, why isn't the MAC learning > log being updated? I get worried when the tools don't agree about what is > occurring. > > Thanks, > John > > > > On Mon, Jun 10, 2013 at 8:36 AM, John Neiberger <[email protected]>wrote: > >> I was just checking again and I'm really having difficulty reconciling >> the conflicting information reported by the switch. The output of "show >> ethernet mac-learning-log" is stable and the MAC addresses show as learned >> on the expected interfaces. However, we created firewall filters to count >> frames coming in with the wrong MAC address and those counters are >> incrementing. We also have the ethernet trace lot that shows the MAC >> addresses jumping around. >> >> So, which is it? Do I trust the trace logs and firewall filter or do I >> trust the MAC learning log? I really with they all agreed. >> >> Thanks, >> John >> >> >> On Mon, Jun 10, 2013 at 8:23 AM, John Neiberger <[email protected]>wrote: >> >>> I think they do have them configured differently, but they swear they >>> don't and I don't have the access or the knowledge on that platform to >>> disagree with them. The network side is straightforward switching. You are >>> correct that we have multiple groups managing these devices. This group is >>> the only one having these problems, but even then it's only with a couple >>> of locations. The rest of theirs are fine. We've been having major >>> conference calls to discuss this problem and those have included several >>> people from Acme Packet. I can only assume that they've verified the high >>> availability config. >>> >>> I was hoping that they were running linux under the hood since this is >>> apparently the default behavior of linux. However, I just heard that >>> they're running Vxworks. I'll have to do some digging to see if it also >>> behaves similarly by default. >>> >>> Thanks! >>> John >>> >>> >>> On Mon, Jun 10, 2013 at 2:37 AM, Phil Mayers <[email protected]>wrote: >>> >>>> On 06/09/2013 04:59 PM, John Neiberger wrote: >>>> >>>> We have several of these throughout our network and we're only seeing >>>>> this >>>>> problem in a couple of cases. The rest work just fine. Most of the >>>>> SBCs are >>>>> >>>> >>>> Obvious question: are you sure those two aren't configured different >>>> (wrongly) compared to the others, or running different software? >>>> >>>> Reading between the lines it sounds like a different group runs these >>>> devices, and in my experience, kit like this with odd network interface >>>> capabilities can be misconfigured by staff that don't fully understand >>>> networking - and it can be exacerbated by vendors using odd terminology >>>> (e.g. "PHY" used in a non-standard way) and confusing people. >>>> >>>> Are you sure they haven't just setup these two devices in active-active >>>> mode? Or more likely, something that *sounds* like an active-passive mode >>>> in the UI, to a non-expert, but is really some kind of weird active-active >>>> spatchcock. >>>> >>>> Our storage team did this with a couple of NetApps and the symptoms >>>> were pretty much identical. It only really brought things crashing down >>>> when they started sending 1Gbit/sec of performance testing traffic from >>>> them... >>>> >>>> ______________________________**_________________ >>>> juniper-nsp mailing list [email protected] >>>> https://puck.nether.net/**mailman/listinfo/juniper-nsp<https://puck.nether.net/mailman/listinfo/juniper-nsp> >>>> >>> >>> >> > _______________________________________________ juniper-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/juniper-nsp
