Thanks Steve, > Are you only syncing in one direction? > > fe80::250:56ff:febf:3ca5 is a link-local address which looks a bit strange in > my skimming of the below. > > Overall, we have two IPv6 ranges for the routing: > WAN CARP IP: 2607:ff50::12/125 > WAN IP router 1: 2607:ff50::17/125 > WAN IP router 2: 2607:ff50::16/125 > LAN block: 2607:ff50:0:4c::0/64 > > 2607:ff50:0:4c::0/64 is routed to 2607:ff50::12 by our data center. CARP > syncs over IPv4 and we've not had a problem. We're on 2.2.6.
I also have only global IPv6 addresses on both WAN and as CARP IP. But each interface always have a link-local address. Check /status_interfaces.php for instance. And the CARP announcements seem to be sent from these LL addresses. (For the PFSYNC between both FW, I have a third interface 'opt1' dedicated to that link, using IPv4 indeed.) It probably is similar for you. If you'd like, run a packet capture on your WAN, address family IPv6, protocol ANY: you should see some lines similar to this one (among possibly many other things - a 3 to 5 seconds capture will be more than enough): > FW1: 11:59:13.379073 IP6 fe80::250:56ff:febf:3ca5 > ff02::12: ip-proto-112 36 > ... Now run the same on your other box, and the source address you will see will be the same as on the first box. For the test, do NOT filter on IPv6 and then CARP : won't work, never captures anything (on IPv6). Must be a bug in the packet capture interface of pfSense. > "CARP is not permitted on their equipment" > > Is that even possible? How would they prevent that other than tying the IP > address to a MAC address? It is more than possible because as things have (slowly) progressed today, it now looks nearly certain they're the problem. On my LAN interfaces, CARP works perfectly for our LAN side CARP IPv4 and CARP IPv6. It's only on the interfaces facing them that it fails. I guess something to do with multicasts FF02::12 being dropped by switches. They're supposed to remove blocking on request and I'm now pretty sure they do something wrong in this regard. -- Meilleures salutations, Met vriendelijke groeten, Best Regards, Olivier Mascia, integral.be/om > Le 4 mai 2016 à 21:13, Steve Yates <[email protected]> a écrit : > > "IPv6 does not seem to get proper advertisements from peer and both think > they're MASTER" > > Are you only syncing in one direction? > > fe80::250:56ff:febf:3ca5 is a link-local address which looks a bit strange in > my skimming of the below. > > Overall, we have two IPv6 ranges for the routing: > WAN CARP IP: 2607:ff50::12/125 > WAN IP router 1: 2607:ff50::17/125 > WAN IP router 2: 2607:ff50::16/125 > LAN block: 2607:ff50:0:4c::0/64 > > 2607:ff50:0:4c::0/64 is routed to 2607:ff50::12 by our data center. CARP > syncs over IPv4 and we've not had a problem. We're on 2.2.6. > > "CARP is not permitted on their equipment" > > Is that even possible? How would they prevent that other than tying the IP > address to a MAC address? > > -- > > Steve Yates > ITS, Inc. > > -----Original Message----- > From: List [mailto:[email protected]] On Behalf Of Olivier Mascia > Sent: Wednesday, May 4, 2016 5:12 AM > To: pfSense Support and Discussion Mailing List <[email protected]> > Subject: Re: [pfSense] 2.3-REL, HA, WAN CARP IPv6 MAC seen as active on both > NICs > > >> Le 3 mai 2016 à 11:17, Olivier Mascia <[email protected]> a écrit : >> >>> Le 3 mai 2016 à 09:49, Chris Buechler <[email protected]> a écrit : >>> >>>> Or would it be that my BACKUP (according to /status_carp.php) do also >>>> advertise (which it shouldn't as BACKUP)? >>> >>> That's the problem. I'm seeing that in some cases and not others with >>> IPv6 CARP in 2.3, with no apparent reason as to why. It seems like it >>> continues to work fine in that circumstance for me, but that could >>> definitely affect switch CAM tables and cause issues like packet loss >>> in some environments. I need to look at it closer tomorrow. >> >> It's a relief to read your comment. :) >> >> As I clearly have a system where this happen, what would you need from me or >> my system to maybe help you pinpoint what's the cause? >> Could this possibly be a NIC drivers issue? >> Those are vmware VMs using VMXNET3 (underlying physical NICs on the cluster >> hosts are 10 Gbe). >> Would it be worth trying to downgrade to E1000 and see if it helps? Or a >> probable pure loss of time? >> >> Also, from your comment, am I right assuming this is not known to happen >> with <2.3 releases? >> So that I could consider rebuilding those VMs using 2.2.6 for instance? >> And upgrade to 2.3.x later? >> >> Thanks! > > I'm lost trying to get CARP / IPv6 working, including on 2.2.6 (I setup two > new VM using 2.2.6 to compare results with those I had with 2.3). > CARP works for IPv4 and IPv6 on my LAN side. > On WAN side, only IPv4 is OK. IPv6 does not seem to get proper advertisements > from peer and both think they're MASTER. > > The ports on which my WAN interfaces are plugged in are managed by the > hosting provider and I tend to think they light have something setup wrong on > their side. By default, CARP is not permitted on their equipment and I have > to trigger (once) a GUI command to "activate CARP" on each of my interfaces > facing their equipment. To my understanding it probably allows the required > multicast to flow between both ports. I fear their setup might not work for > the ff02::12 traffic. > > Capturing on IPv4, I see : > > FW1: 11:54:38.719091 IP 51.254.87.130 > 224.0.0.18: VRRPv2, Advertisement, > vrid 104, prio 0, authtype none, intvl 1s, length 36 ... > and > FW2: 11:54:38.723415 IP 51.254.87.130 > 224.0.0.18: VRRPv2, Advertisement, > vrid 104, prio 0, authtype none, intvl 1s, length 36 ... > > That looks good and understandable to me. > State MASTER or BACKUP switch properly from one box or the other, when I > shutdown one of the others, and restore properly to FW1 MASTER and FW2 BACKUP > when both are online. Therefore, the IPv4 CARP VIP works properly which can > be easily tested. > > Capturing on IPv6, I see : > > FW1: 11:59:13.379073 IP6 fe80::250:56ff:febf:3ca5 > ff02::12: ip-proto-112 36 > ... > and > FW2: 11:59:13.202384 IP6 fe80::250:56ff:febf:37a3 > ff02::12: ip-proto-112 36 > ... > > And both FW switch to MASTER. > > This same behavior with 2.3 and 2.2.6. > > I'll talk again to my supplier who have the control of those ports, insisting > on checking IPv6 multicast. But I feel sad not really knowing if I'm hit by a > bug their side or my side on pfSense level. > > If someone has CARP on IPv6 working, would you be so kind to check what you > can capture about it (IPv6)? Does it differ from the scheme I'm seeing? > > Thanks!! > -- > Meilleures salutations, Met vriendelijke groeten, Best Regards, Olivier > Mascia, integral.be/om > _______________________________________________ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
