Here's an example of what I'm talking about. The MAC learning log has been stable since they reloaded the SBC on Saturday:
show ethernet-switching mac-learning-log | match 3c:91 Sat Jul 23 12:05:22 2011 vlan_name sbc-core mac 00:08:25:fa:3c:91 was learned on xe-0/1/1.0 Sat Apr 13 04:24:15 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was deleted on xe-0/1/1.0 Sat Apr 13 04:25:10 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was learned on ge-0/0/5.0 Sat Apr 13 04:25:11 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was deleted on ge-0/0/5.0 Sat Apr 13 04:25:18 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was learned on xe-0/1/1.0 Sat Jun 8 04:27:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was deleted on ge-0/0/5.0 Sat Jun 8 04:28:08 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was learned on ge-0/0/5.0 Sat Jun 8 04:28:09 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was deleted on ge-0/0/5.0 Sat Jun 8 04:28:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was learned on ge-0/0/5.0 However, we have a firewall filter on ge-0/0/6 and ge-0/0/8 to watch for that MAC address and its counters are incrementing. They have incremented several times this morning already: Filter: ACME-ge-0/0/6.0-i Counters: Name Bytes Packets from-ACME-3c91-ge-0/0/6.0-i 1090 5 from-others-ge-0/0/6.0-i 215535790813 981754726 Filter: ACME-ge-0/0/8.0-i Counters: Name Bytes Packets from-ACME-3c91-ge-0/0/8.0-i 15042 69 from-others-ge-0/0/8.0-i 205829895148 944189465 We also have a trace log running that shows this MAC jumping around. Here is the log output from just the last few minutes: Jun 10 14:30:31.546683 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/7.0, pnac_status 0, 0 Jun 10 14:30:31.546978 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:31:31.826971 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:32:32.535585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:32:32.536080 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:36:43.660324 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:36:43.660961 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:36:55.239016 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:36:55.239425 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:37:29.244518 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:38:32.854226 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:38:32.854532 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:39:36.204933 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:39:36.205232 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:40:11.707066 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:40:37.537264 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:40:37.537562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:43:18.421993 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:43:18.422492 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:43:59.154585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:44:00.000894 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:44:00.001405 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:44:47.068248 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:44:47.068539 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:45:36.634048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:47:27.367950 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:47:27.368359 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:48:15.151240 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:48:15.151530 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:48:51.587305 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:50:00.933896 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:50:00.934196 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:54:43.553048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:54:43.553342 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 Jun 10 14:54:50.862562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/8.0, pnac_status 0, 0 Jun 10 14:54:50.862887 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91, ifname ge-0/0/5.0, pnac_status 0, 0 If the MAC address really is jumping around, why isn't the MAC learning log being updated? I get worried when the tools don't agree about what is occurring. Thanks, John On Mon, Jun 10, 2013 at 8:36 AM, John Neiberger <[email protected]>wrote: > I was just checking again and I'm really having difficulty reconciling the > conflicting information reported by the switch. The output of "show > ethernet mac-learning-log" is stable and the MAC addresses show as learned > on the expected interfaces. However, we created firewall filters to count > frames coming in with the wrong MAC address and those counters are > incrementing. We also have the ethernet trace lot that shows the MAC > addresses jumping around. > > So, which is it? Do I trust the trace logs and firewall filter or do I > trust the MAC learning log? I really with they all agreed. > > Thanks, > John > > > On Mon, Jun 10, 2013 at 8:23 AM, John Neiberger <[email protected]>wrote: > >> I think they do have them configured differently, but they swear they >> don't and I don't have the access or the knowledge on that platform to >> disagree with them. The network side is straightforward switching. You are >> correct that we have multiple groups managing these devices. This group is >> the only one having these problems, but even then it's only with a couple >> of locations. The rest of theirs are fine. We've been having major >> conference calls to discuss this problem and those have included several >> people from Acme Packet. I can only assume that they've verified the high >> availability config. >> >> I was hoping that they were running linux under the hood since this is >> apparently the default behavior of linux. However, I just heard that >> they're running Vxworks. I'll have to do some digging to see if it also >> behaves similarly by default. >> >> Thanks! >> John >> >> >> On Mon, Jun 10, 2013 at 2:37 AM, Phil Mayers <[email protected]>wrote: >> >>> On 06/09/2013 04:59 PM, John Neiberger wrote: >>> >>> We have several of these throughout our network and we're only seeing >>>> this >>>> problem in a couple of cases. The rest work just fine. Most of the SBCs >>>> are >>>> >>> >>> Obvious question: are you sure those two aren't configured different >>> (wrongly) compared to the others, or running different software? >>> >>> Reading between the lines it sounds like a different group runs these >>> devices, and in my experience, kit like this with odd network interface >>> capabilities can be misconfigured by staff that don't fully understand >>> networking - and it can be exacerbated by vendors using odd terminology >>> (e.g. "PHY" used in a non-standard way) and confusing people. >>> >>> Are you sure they haven't just setup these two devices in active-active >>> mode? Or more likely, something that *sounds* like an active-passive mode >>> in the UI, to a non-expert, but is really some kind of weird active-active >>> spatchcock. >>> >>> Our storage team did this with a couple of NetApps and the symptoms were >>> pretty much identical. It only really brought things crashing down when >>> they started sending 1Gbit/sec of performance testing traffic from them... >>> >>> ______________________________**_________________ >>> juniper-nsp mailing list [email protected] >>> https://puck.nether.net/**mailman/listinfo/juniper-nsp<https://puck.nether.net/mailman/listinfo/juniper-nsp> >>> >> >> > _______________________________________________ juniper-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/juniper-nsp
