Hi all,
Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was using
carp in IP balance mode without problems from several months. These firewalls
are installed in a RHEL 8.2 (fully patched) KVM host.
After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I have
tested reconfiguring balance mode for ip-stealth and ip-unicast also and the
result is always the same: network packets are not processed by firewalls. But
if I configure CARP using “the simple configuration” and one node is master and
the other is backup all it is working without problems.
All CARP interfaces are configured as this one:
carpdev vio0 balancing ip pass 7254e4bc3024e35490e4b9942f919e9b
inet 172.22.55.30 0xffffffe0 172.22.55.31
carpnodes 10:0,11:100
description "Production Network"
sysctl.conf file:
net.inet.carp.preempt=1
net.inet.carp.log=2
net.inet.ip.forwarding=1
net.inet.tcp.mssdflt=1440
net.inet.ip.redirect=0
net.inet.ip.mtudisc=0
net.inet.tcp.rfc3390=1
net.inet.ip.arptimeout=60
kern.bufcachepercent=70
net.inet.icmp.tstamprepl=0
net.inet.udp.sendspace=262144
net.inet.udp.recvspace=262144
OpenBSD kvm guest config:
<domain type='kvm' id='12'>
<name>obsdfw01</name>
<description>OpenBSD Security Gateway Cluster</description>
<memory unit='KiB'>786432</memory>
<currentMemory unit='KiB'>786432</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-rhel7.6.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Broadwell</model>
<feature policy='require' name='vme'/>
<feature policy='require' name='f16c'/>
<feature policy='require' name='rdrand'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='xsaveopt'/>
<feature policy='require' name='abm'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/data/vmvol0/vmachines/obsdfw01vol.img'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x0b' slot='0x00'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='none'>
<alias name='usb'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00'
function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'
multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'
multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<alias name='pci.13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x4'/>
</controller>
<controller type='sata' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='00:50:56:6f:64:aa'/>
<source bridge='prodif'/>
<target dev='obsdprod0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:ab:44:05'/>
<source bridge='pubif'/>
<target dev='obsdpub0'/>
<model type='virtio'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:3c:e5:61'/>
<source bridge='mgmtif'/>
<target dev='obsdmgmt0'/>
<model type='virtio'/>
<alias name='net2'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:4c:d6:34'/>
<source bridge='dmzif'/>
<target dev='obsddmz0'/>
<model type='virtio'/>
<alias name='net3'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:73:a4:ff'/>
<source bridge='vpnif'/>
<target dev='obsdvpn0'/>
<model type='virtio'/>
<alias name='net4'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:29:0d:b5'/>
<source bridge='encif'/>
<target dev='obsdenc0'/>
<model type='virtio'/>
<alias name='net5'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:d1:ba:cc'/>
<source bridge='idpmif'/>
<target dev='obsdidp0'/>
<model type='virtio'/>
<alias name='net6'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:49:21:d0'/>
<source bridge='syncif'/>
<target dev='obsdsync0'/>
<model type='virtio'/>
<alias name='net7'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00'
function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:50:56:a6:72:ff'/>
<source bridge='winif'/>
<target dev='obsdwin0'/>
<model type='virtio'/>
<alias name='net8'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/4'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/4'>
<source path='/dev/pts/4'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5903' autoport='yes' listen='127.0.0.1'
keymap='es'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'
primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x0c' slot='0x00'
function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x0d' slot='0x00'
function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c82,c777</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c82,c777</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
</domain>
Dmesg output:
OpenBSD 6.8 (GENERIC) #97: Sun Oct 4 18:00:46 MDT 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 788389888 (751MB)
avail mem = 749596672 (714MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5af0 (9 entries)
bios0: vendor SeaBIOS version "1.11.1-4.module+el8.1.0+4066+0f1aadab" date
04/01/2014
bios0: Red Hat KVM
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC MCFG
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Core Processor (Broadwell), 1900.29 MHz, 06-3d-02
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,ARAT,XSAVEOPT,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line
16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
cpu0: using Broadwell MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Red Hat QXL Video" rev 0x04
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 2 function 0 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci1 at ppb0 bus 1
virtio0 at pci1 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio0 at virtio0: address 00:50:56:6f:64:aa
virtio0: msix shared
ppb1 at pci0 dev 2 function 1 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci2 at ppb1 bus 2
virtio1 at pci2 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio1 at virtio1: address 00:50:56:ab:44:05
virtio1: msix shared
ppb2 at pci0 dev 2 function 2 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci3 at ppb2 bus 3
virtio2 at pci3 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio2 at virtio2: address 00:50:56:3c:e5:61
virtio2: msix shared
ppb3 at pci0 dev 2 function 3 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci4 at ppb3 bus 4
virtio3 at pci4 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio3 at virtio3: address 00:50:56:4c:d6:34
virtio3: msix shared
ppb4 at pci0 dev 2 function 4 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci5 at ppb4 bus 5
virtio4 at pci5 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio4 at virtio4: address 00:50:56:73:a4:ff
virtio4: msix shared
ppb5 at pci0 dev 2 function 5 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci6 at ppb5 bus 6
virtio5 at pci6 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio5 at virtio5: address 00:50:56:29:0d:b5
virtio5: msix shared
ppb6 at pci0 dev 2 function 6 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci7 at ppb6 bus 7
virtio6 at pci7 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio6 at virtio6: address 00:50:56:d1:ba:cc
virtio6: msix shared
ppb7 at pci0 dev 2 function 7 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 22
pci8 at ppb7 bus 8
virtio7 at pci8 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio7 at virtio7: address 00:50:56:49:21:d0
virtio7: msix shared
ppb8 at pci0 dev 3 function 0 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci9 at ppb8 bus 9
virtio8 at pci9 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio8 at virtio8: address 00:50:56:a6:72:ff
virtio8: msix shared
ppb9 at pci0 dev 3 function 1 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci10 at ppb9 bus 10
virtio9 at pci10 dev 0 function 0 "Qumranet Virtio 1.x Console" rev 0x01
virtio9: no matching child driver; not configured
ppb10 at pci0 dev 3 function 2 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci11 at ppb10 bus 11
virtio10 at pci11 dev 0 function 0 "Qumranet Virtio 1.x Storage" rev 0x01
vioblk0 at virtio10
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 16384MB, 512 bytes/sector, 33554432 sectors
virtio10: msix shared
ppb11 at pci0 dev 3 function 3 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci12 at ppb11 bus 12
virtio11 at pci12 dev 0 function 0 vendor "Qumranet", unknown product 0x1045
rev 0x01
viomb0 at virtio11
virtio11: apic 0 int 23
ppb12 at pci0 dev 3 function 4 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci13 at ppb12 bus 13
virtio12 at pci13 dev 0 function 0 "Qumranet Virtio 1.x RNG" rev 0x01
viornd0 at virtio12
virtio12: apic 0 int 23
virtio7: msix shared
ppb8 at pci0 dev 3 function 0 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci9 at ppb8 bus 9
virtio8 at pci9 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio8 at virtio8: address 00:50:56:a6:72:ff
virtio8: msix shared
ppb9 at pci0 dev 3 function 1 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci10 at ppb9 bus 10
virtio9 at pci10 dev 0 function 0 "Qumranet Virtio 1.x Console" rev 0x01
virtio9: no matching child driver; not configured
ppb10 at pci0 dev 3 function 2 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci11 at ppb10 bus 11
virtio10 at pci11 dev 0 function 0 "Qumranet Virtio 1.x Storage" rev 0x01
vioblk0 at virtio10
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 16384MB, 512 bytes/sector, 33554432 sectors
virtio10: msix shared
ppb11 at pci0 dev 3 function 3 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci12 at ppb11 bus 12
virtio11 at pci12 dev 0 function 0 vendor "Qumranet", unknown product 0x1045
rev 0x01
viomb0 at virtio11
virtio11: apic 0 int 23
ppb12 at pci0 dev 3 function 4 vendor "Red Hat", unknown product 0x000c rev
0x00: apic 0 int 23
pci13 at ppb12 bus 13
virtio12 at pci13 dev 0 function 0 "Qumranet Virtio 1.x RNG" rev 0x01
viornd0 at virtio12
virtio12: apic 0 int 23
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
scsibus2 at ahci0: 32 targets
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (dcd0d9bbce80825c.a) swap on sd0b dump on sd0b
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp4: state transition: BACKUP -> MASTER
carp5: state transition: BACKUP -> MASTER
carp6: state transition: BACKUP -> MASTER
carp7: state transition: BACKUP -> MASTER
pfsync: failed to receive bulk update
Regards,
C. L. Martinez
