Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

After incorporating routing into carp1 device configuration fw2 booted 
with ARP entry for carp1.

I need to test it a little more but that could solve the case.
I don't know how it was working in 5.8. 


Yes, that was it. Setting the routing in physical device configuration 
(vio1) to non-existant IP address
caused the carp device on top to go into not properly configured device 
- the MASTER/SLAVE states were

running fine but the arp entry for CARP IP was not in the arp table.

Thank you guys for you time and effort helping me.

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

How does your /etc/hostname.carp1 look like?

passwords masked (it's the same unique password on both nodes):
fw1:
inet 10.24.5.1 255.255.255.0 10.24.5.255 vhid 55 carpdev vio1 pass 
fw2:
inet 10.24.5.1 255.255.255.0 10.24.5.255 vhid 55 carpdev vio1 pass 
 advskew 128 
I was wondering if the configuration of vio1 could be the case at boot 
because I compared it
with other interfaces and it's different as I am adding the routing for 
VPN to CARP IP.


/etc/hostname.vio1:
inet 10.24.5.3 255.255.255.0 10.24.5.255
!route add -net 10.8.0.0/16 10.24.5.1

I would be better doing this in /etc/hostname.carp1 as 10.24.5.1 is not 
available yet.


And YES!
After incorporating routing into carp1 device configuration fw2 booted 
with ARP entry for carp1.

I need to test it a little more but that could solve the case.
I don't know how it was working in 5.8.
Aghh... such a mistake :(

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

On 06.12.2016 14:10, Stefan Sperling wrote:


Does 'ifconfig vio1 down' followed by 'ifconfig vio1 up' restore ARP?

This is likely the CARP/ARP regression recently fixed in -current which
is fixed by the following patch.
See http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/route.c
revisions 1.337 and 1.341.


Thank you for your answer.
ifconfig vio1 down followed by ifconfig vio1 up does not help but
I believe
# sh /etc/netstart carp1
fixes the arp table.
Do you think this could be the case of regression you mentioned?

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

W dniu 06.12.2016 o 14:40, Martin Pieuchot pisze:

On 06/12/16(Tue) 13:48, Rafał Błaszczyk wrote:

At first I would like to say hello and greet everyone as this is my first
post here.

I am having strange issues with one of the CARP interfaces.

I have two OpenBSD boxes (fw1, fw2) running as HA firewalls with CARP
interfaces in each VLAN.

Both boxes are running on two Linux KVM (Proxmox 4.2) hosts.

One of CARP interfaces stopped responding on ARP requests on CARP IP - it's
carp1

running on physical dev vio1 which is also running pfsync on top.

It's weird because the rest of carp interfaces behave correctly.

# ifconfig carp1
carp1: flags=8843 mtu 1500
 lladdr 00:00:5e:00:01:37
 index 18 priority 15 llprio 3
 carp: MASTER carpdev vio1 vhid 55 advbase 1 advskew 0
 groups: carp
 status: master
 inet 10.24.5.1 netmask 0xff00 broadcast 10.24.5.255

I've checked arp table on two boxes and there is no entry for carp1.

That's the problem.  We'll have to figure out where does it come from.

Could you share your routing table?  Doing "# netstat -rnf inet"
You can find it below from fw1, I masked my public gw with G.G.G.G and 
public IP with P.P.P.P,

masked first bytes of other MAC addresses with XXX

Routing tables

Internet:
DestinationGatewayFlags   Refs  Use   Mtu Prio Iface
defaultG.G.G.GUGS  364  1698856 - 8 pppoe0
224/4  127.0.0.1  URS197662 32768 8 lo0
10.8/1610.24.5.1  UGS00 - 8 vio1
10.9.0/24  10.9.0.2   UGS040756 - 8 tun0
10.9.0.1   10.9.0.1   UHl00 - 1 tun0
10.9.0.2   10.9.0.1   UH 11 - 8 tun0
10.24.5/24 10.24.5.2  UC 524293 - 4 vio1
10.24.5/24 10.24.5.1  UC 00 - 4 carp1
10.24.5.2  52:54:00:84:51:e0  UHLl   0 8265 - 1 vio1
10.24.5.13 XXX:31:27  UHLc   024306 - 4 vio1
10.24.5.14 XXX:02:30  UHLc   124302 - 4 vio1
10.24.5.53 XXX:df:7c  UHLc   024302 - 4 vio1
10.24.5.54 XXX:70:c1  UHLc   024303 - 4 vio1
10.24.5.201XXX:9d:c1  UHLc   128783 - 4 vio1
10.24.5.25510.24.5.2  UHb0 2184 - 1 vio1
10.24.5.25510.24.5.1  UHb00 - 1 carp1
10.24.10/2410.24.10.2 UC 8  197 - 4 vio2
10.24.10/2410.24.10.1 UC 00 - 4 carp2
10.24.10.1 00:00:5e:00:01:02  UHLl   0 4093 - 1 carp2
10.24.10.2 52:54:00:a7:c6:bd  UHLl   014664 - 1 vio2
10.24.10.11XXX:f4:82  UHLc   0  210 - 4 vio2
10.24.10.15XXX:36:37  UHLc   0 3979 - 4 vio2
10.24.10.16XXX:37:37  UHLc   0 6644 - 4 vio2
10.24.10.23XXX:61:33  UHLc   0  413 - 4 vio2
10.24.10.24XXX:30:38  UHLc   0 3252 - 4 vio2
10.24.10.37link#3 UHRLc  0  245 - 4 vio2
10.24.10.38XXX:61:34  UHLc   0 4475 - 4 vio2
10.24.10.51XXX:b7:fb  UHLc   0   698374 - 4 vio2
10.24.10.255   10.24.10.2 UHb0  544 - 1 vio2
10.24.10.255   10.24.10.1 UHb00 - 1 carp2
10.24.20/2410.24.20.2 UC 3 8327 - 4 vio3
10.24.20/2410.24.20.1 UC 00 - 4 carp3
10.24.20.1 00:00:5e:00:01:03  UHLl   0 1374 - 1 carp3
10.24.20.2 52:54:00:e0:03:95  UHLl   037679 - 1 vio3
10.24.20.11XXX:a3:f9  UHLc   019191 - 4 vio3
10.24.20.212   XXX:ee:99  UHLc   0 8362 - 4 vio3
10.24.20.214   XXX:b4:d0  UHLc   015250 - 4 vio3
10.24.20.255   10.24.20.2 UHb00 - 1 vio3
10.24.20.255   10.24.20.1 UHb00 - 1 carp3
10.24.21/2410.24.21.2 UC 2  174 - 4 vio4
10.24.21/2410.24.21.1 UC 00 - 4 carp4
10.24.21.1 00:00:5e:00:01:04  UHLl   0  368 - 1 carp4
10.24.21.2 52:54:00:62:92:1e  UHLl   0   74 - 1 vio4
10.24.21.12XXX:88:e2  UHLc   1 4267 - 4 vio4
10.24.21.16XXX:12:88  UHLc   0  536 - 4 vio4
10.24.21.255   10.24.21.2 UHb00 - 1 vio4
10.24.21.255   10.24.21.1 UHb00 - 1 carp4
10.24.22/2410.24.22.2 UC 4 4560 - 4 vio5
10.24.22/2410.24.22.1 UC 00 - 4 carp5
10.24.22.1 00:00:5e:00:01:05  UHLl   0 2755 

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

W dniu 06.12.2016 o 14:40, Martin Pieuchot pisze:

On 06/12/16(Tue) 13:48, Rafał Błaszczyk wrote:

At first I would like to say hello and greet everyone as this is my first
post here.

I am having strange issues with one of the CARP interfaces.

I have two OpenBSD boxes (fw1, fw2) running as HA firewalls with CARP
interfaces in each VLAN.

Both boxes are running on two Linux KVM (Proxmox 4.2) hosts.

One of CARP interfaces stopped responding on ARP requests on CARP IP - it's
carp1

running on physical dev vio1 which is also running pfsync on top.

It's weird because the rest of carp interfaces behave correctly.

# ifconfig carp1
carp1: flags=8843 mtu 1500
 lladdr 00:00:5e:00:01:37
 index 18 priority 15 llprio 3
 carp: MASTER carpdev vio1 vhid 55 advbase 1 advskew 0
 groups: carp
 status: master
 inet 10.24.5.1 netmask 0xff00 broadcast 10.24.5.255

I've checked arp table on two boxes and there is no entry for carp1.

That's the problem.  We'll have to figure out where does it come from.

Could you share your routing table?  Doing "# netstat -rnf inet"

How does your /etc/hostname.carp1 look like?

Do you see an error when running "# sh /etc/netstart carp1" ?

If you grep for 'arp' in /var/log/messages do you get anything?

I believe it doesn't help


# ifconfig vio1 down; sleep 2; ifconfig vio1 up
fw1 ~
# arp -an | grep carp1
10.24.52.1   00:00:5e:00:01:0a carp10 permanent  l
10.24.53.1   00:00:5e:00:01:0b carp11 permanent  l
10.24.54.1   00:00:5e:00:01:0c carp12 permanent  l
10.24.55.1   00:00:5e:00:01:0d carp13 permanent  l

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Martin Pieuchot]

On 06/12/16(Tue) 13:48, Rafał Błaszczyk wrote:
> At first I would like to say hello and greet everyone as this is my first
> post here.
> 
> I am having strange issues with one of the CARP interfaces.
> 
> I have two OpenBSD boxes (fw1, fw2) running as HA firewalls with CARP
> interfaces in each VLAN.
> 
> Both boxes are running on two Linux KVM (Proxmox 4.2) hosts.
> 
> One of CARP interfaces stopped responding on ARP requests on CARP IP - it's
> carp1
> 
> running on physical dev vio1 which is also running pfsync on top.
> 
> It's weird because the rest of carp interfaces behave correctly.
> 
> # ifconfig carp1
> carp1: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:37
> index 18 priority 15 llprio 3
> carp: MASTER carpdev vio1 vhid 55 advbase 1 advskew 0
> groups: carp
> status: master
> inet 10.24.5.1 netmask 0xff00 broadcast 10.24.5.255
> 
> I've checked arp table on two boxes and there is no entry for carp1.

That's the problem.  We'll have to figure out where does it come from.

Could you share your routing table?  Doing "# netstat -rnf inet"

How does your /etc/hostname.carp1 look like?

Do you see an error when running "# sh /etc/netstart carp1" ?

If you grep for 'arp' in /var/log/messages do you get anything?

Re: One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Stefan Sperling]

On Tue, Dec 06, 2016 at 01:48:27PM +0100, Rafał Błaszczyk wrote:
> One of CARP interfaces stopped responding on ARP requests on CARP IP - it's
> carp1
> 
> running on physical dev vio1 which is also running pfsync on top.

> What I've already checked:
> 
> - ifconfig down and up on carp1 does not help

Does 'ifconfig vio1 down' followed by 'ifconfig vio1 up' restore ARP?

This is likely the CARP/ARP regression recently fixed in -current which
is fixed by the following patch.
See http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/route.c
revisions 1.337 and 1.341.

Index: route.c
===
RCS file: /cvs/src/sys/net/route.c,v
retrieving revision 1.313
diff -u -p -r1.313 route.c
--- route.c 22 Jul 2016 11:03:30 -  1.313
+++ route.c 6 Dec 2016 13:04:00 -
@@ -1256,7 +1256,7 @@ rt_ifa_add(struct ifaddr *ifa, int flags
prio = RTP_LOCAL;
 
if (flags & RTF_CONNECTED)
-   prio = RTP_CONNECTED;
+   prio = ifp->if_priority + RTP_CONNECTED;
 
error = rtrequest(RTM_ADD, , prio, , rtableid);
if (error == 0) {
@@ -1316,7 +1316,7 @@ rt_ifa_del(struct ifaddr *ifa, int flags
prio = RTP_LOCAL;
 
if (flags & RTF_CONNECTED)
-   prio = RTP_CONNECTED;
+   prio = ifp->if_priority + RTP_CONNECTED;
 
error = rtrequest_delete(, prio, ifp, , rtableid);
if (error == 0) {

One of the CARP interfaces stopped sending ARP replies on OpenBSD 6.0

[Rafał Błaszczyk]

At first I would like to say hello and greet everyone as this is my 
first post here.


I am having strange issues with one of the CARP interfaces.

I have two OpenBSD boxes (fw1, fw2) running as HA firewalls with CARP 
interfaces in each VLAN.


Both boxes are running on two Linux KVM (Proxmox 4.2) hosts.

One of CARP interfaces stopped responding on ARP requests on CARP IP - 
it's carp1


running on physical dev vio1 which is also running pfsync on top.

It's weird because the rest of carp interfaces behave correctly.

# ifconfig carp1
carp1: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:37
index 18 priority 15 llprio 3
carp: MASTER carpdev vio1 vhid 55 advbase 1 advskew 0
groups: carp
status: master
inet 10.24.5.1 netmask 0xff00 broadcast 10.24.5.255

I've checked arp table on two boxes and there is no entry for carp1.

# arp -an | grep carp
10.24.10.1   00:00:5e:00:01:02  carp2 permanent  l
10.24.20.1   00:00:5e:00:01:03  carp3 permanent  l
10.24.21.1   00:00:5e:00:01:04  carp4 permanent  l
10.24.22.1   00:00:5e:00:01:05  carp5 permanent  l
10.24.23.1   00:00:5e:00:01:06  carp6 permanent  l
10.24.24.1   00:00:5e:00:01:07  carp7 permanent  l
10.24.30.1   00:00:5e:00:01:08  carp8 permanent  l
10.24.51.1   00:00:5e:00:01:09  carp9 permanent  l
10.24.52.1   00:00:5e:00:01:0a carp10 permanent  l
10.24.53.1   00:00:5e:00:01:0b carp11 permanent  l
10.24.54.1   00:00:5e:00:01:0c carp12 permanent  l
10.24.55.1   00:00:5e:00:01:0d carp13 permanent  l
192.168.188.30   00:00:5e:00:01:1e  carp0 permanent  l

I don't know what could be the case of that as carp interface's states 
seems to be


working right (both carp1 and the rest).

What changed:

- upgrade to OpenBSD 6.0-release from 5.8-release (through 5.9)

- new pppoe interface

- ifstated checking on carp1.link, calling pppoe up and reloading 
pf.conf if master


What I've already checked:

- ifconfig down and up on carp1 does not help

- I don't see nothing suspicious in the logs

- tcpdump is showing CARP advertise 36 from master host

- PF is allowing CARP

- VHIDs are different on different carp interfaces, I got no other carp 
traffic running in this network


- multicasts are running well on physical interfaces (tested with omping 
on KVM hosts), IGMP snooping on and


IGMP queries are sent from the switch

BTW, this is /etc/sysctl.conf:
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=6
net.inet.ip.forwarding=1
net.inet.ip.redirect=0
kern.bufcachepercent=50

I am out of ideas for now and thinking about rolling back to 5.8.

Waiting for your suggestions what else could be a problem.