I'm using 3 different rdomains, with one BGPd instance in each of them
with different configurations, when using rdomain 255 some routing info
is leaked into rtable 0 to the point that doesn't allow to route, this
happens over a few minutes/hours of functioning (tested on two VM's with
6.2-stable and 6.3-release both amd64), network driver is vmx(4) on all
interfaces of both systems.
As I understand "netstat -nr -f inet" should have the same output as
"netstat -T0 -nr -f inet", this happens even after reboots.
* There is no BGPd running on rdomain 0, just on defined rdomains.
* Placing rtable $rdomain-number inside bgpd$RDOMAIN.conf doesn't change
the situation.
* Changing rdomain from 255 to 254 on interfaces vmx3 and carp25[45]
seems to solve the issue,
* No issue with routes belonging to different bgpd processess running on
other rdomains (179 & 253) were found.
*UPDATE* This error is only seen when logged in using SSH (user root,
key auth), if I test the same using the direct-attached console (VMWare
VMRC) the routes are shown correctly and there is no loss of connection
(I know it sound nuts).
Tests on the non working
# route -n get 8.8.8.8
route: writing to routing socket: No such process
# netstat -T0 -nr -f inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 17.2.18.33 UGS 4 13 - 8 vmx3
224/4 127.0.0.1 URS 0 0 32768 8 lo0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0
17.2.18.32/27 179.27.168.41 UCn 1 493 - 4 vmx3
17.2.18.33 0a:aa:dc:ff:10:02 UHLch 1 259 - 3 vmx3
17.2.18.41 0b:bb:57:a7:2a:e0 UHLl 0 18 - 1 vmx3
17.2.18.63 17.27.18.41 UHb 0 0 - 1 vmx3
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
10.25/16 10.24.2.1 UG 0 0 - 48 vmx0
10.25.6.20 ab:0c:5e:00:01:ff UHLl 0 0 - 1 carp255
10.25.6.20/32 10.25.6.20 UCn 0 0 - 19 carp255
10.25.6.21 ab:0c:5e:00:01:fe UHLl 0 36 - 1 carp254
10.25.6.21/32 10.25.6.21 Cn 0 0 - 19 carp254
[...]
# alias | grep bgp
bgpctl179='bgpctl -s /var/run/bgpd.sock.179'
bgpctl253='bgpctl -s /var/run/bgpd.sock.253'
bgpctl255='bgpctl -s /var/run/bgpd.sock.255'
# ls -lh /etc/rc.d/bgpd*
-r-xr-xr-x 1 root wheel 228B Mar 24 17:12 /etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd179 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd253 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd255 ->
/etc/rc.d/bgpd
# cat /etc/rc.conf.local
bgpd179_flags=-f /etc/bgpd179.conf -v
bgpd179_rtable=179
bgpd253_flags=-f /etc/bgpd253.conf -v
bgpd253_rtable=253
bgpd255_flags=-f /etc/bgpd255.conf -v
bgpd255_rtable=255
pkg_scripts=bgpd253 bgpd179 bgpd255
# bgpctl255 show ip bgp | head
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete
flags destination gateway lpref med aspath origin
*> 10.25.0.0/16 10.24.2.1 100 0 65510 i
*> 10.25.0.0/20 10.24.2.1 100 0 65510 65500 i
*> 10.25.8.0/24 10.24.2.1 100 0 65510 65500 i
*> 10.25.16.0/22 10.24.2.1 100 0 65510 65500 i
[...]
# ps aux -o rtable | grep -E '(_bgpd|USER)'
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND RTABLE
_bgpd 16269 0.0 0.5 944 1956 ?? Ip 12:53PM 0:00.00 bgpd:route deci 253
_bgpd 34173 0.0 0.5 940 1760 ?? Sp 12:53PM 0:00.34 bgpd:session en 253
_bgpd 48580 0.0 0.5 928 1940 ?? Ip 12:53PM 0:00.00 bgpd:route deci 179
_bgpd 49612 0.0 0.5 936 1768 ?? Sp 12:53PM 0:00.34 bgpd:session en 179
_bgpd 69090 0.0 0.6 1088 2248 ?? Ip 12:53PM 0:00.01 bgpd:route deci 255
_bgpd 96380 0.0 0.5 1008 1876 ?? Sp 12:53PM 0:00.33 bgpd:session en 255
# cat /etc/hostname.vmx0
rdomain 255
inet 10.24.2.3 255.255.255.224
description "Server -> Router"
up
# cat /etc/hostname.carp255
rdomain 255
vhid 255 carpdev vmx0 carppeer 10.24.2.4 advskew 0 pass mypass state master
inet 10.25.6.20/32
up
# cat /etc/hostname.carp254
rdomain 255
vhid 254 carpdev vmx0 carppeer 10.24.2.4 advskew 100 pass myotherpass
state backup
inet 10.25.6.21/32
up
# grep -v ^# /etc/bgpd255.conf
peer="10.24.2.1"
AS 65512
router-id 10.24.2.3
log updates
network 10.25.6.20/32
network 10.25.6.21/32
rtable 255
group "AS65510 Router" {
remote-as 65510
descr "Server -> Router"
neighbor $peer {
announce IPv4 unicast
}
}
[...]
dmesg:
OpenBSD 6.3 (GENERIC) #100: Sat Mar 24 14:17:45 MDT 2018
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 385810432 (367MB)
avail mem = 367239168 (350MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (239 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 04/05/2016
bios0: VMware, Inc. VMware Virtual Platform
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3)
S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3)
S1F0(S3) PE50(S3) S1F0(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 2596.75 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,ARAT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
acpitimer0: recalibrated TSC frequency 2596989038 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 65MHz
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 24 pins
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
acpihpet0 at acpi0: 14318179 Hz
acpihpet0: recalibrated TSC frequency 2597017053 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"VMW0003" at acpi0 not configured
"PNP0A05" at acpi0 not configured
acpiac0 at acpi0: AC unit online
pvbus0 at mainbus0: VMware
vmt0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus
disabled
"VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
pci2 at ppb1 bus 2
ahci0 at pci2 dev 1 function 0 "VMware AHCI" rev 0x00: apic 1 int 19,
AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware SATA CD00, 1.00> ATAPI
5/cdrom removable
ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
pci3 at ppb2 bus 3
vmwpvs0 at pci3 dev 0 function 0 "VMware PVSCSI" rev 0x02: apic 1 int 18
scsibus2 at vmwpvs0: 16 targets
sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 2.0> SCSI4 0/direct
fixed
sd0: 30720MB, 512 bytes/sector, 62914560 sectors
ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
pci4 at ppb3 bus 4
vmx0 at pci4 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 18,
address 00:50:56:a7:f1:09
ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
pci5 at ppb4 bus 5
ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
pci6 at ppb5 bus 6
ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
pci7 at ppb6 bus 7
ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
pci8 at ppb7 bus 8
ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
pci9 at ppb8 bus 9
ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
pci10 at ppb9 bus 10
ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
pci11 at ppb10 bus 11
vmx1 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 19,
address 00:50:56:a7:a0:7c
ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
pci12 at ppb11 bus 12
ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
pci13 at ppb12 bus 13
ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
pci14 at ppb13 bus 14
ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
pci15 at ppb14 bus 15
ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
pci16 at ppb15 bus 16
ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
pci17 at ppb16 bus 17
ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
pci18 at ppb17 bus 18
ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
pci19 at ppb18 bus 19
vmx2 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 16,
address 00:50:56:a7:79:91
ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
pci20 at ppb19 bus 20
ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
pci21 at ppb20 bus 21
ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
pci22 at ppb21 bus 22
ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
pci23 at ppb22 bus 23
ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
pci24 at ppb23 bus 24
ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
pci25 at ppb24 bus 25
ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
pci26 at ppb25 bus 26
ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
pci27 at ppb26 bus 27
vmx3 at pci27 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 17,
address 00:50:56:a7:ce:a1
ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
pci28 at ppb27 bus 28
ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
pci29 at ppb28 bus 29
ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
pci30 at ppb29 bus 30
ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
pci31 at ppb30 bus 31
ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
pci32 at ppb31 bus 32
ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
pci33 at ppb32 bus 33
ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
pci34 at ppb33 bus 34
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (9e2c186da5ea012d.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
Thanks
