Lets get some standard stuff out of the way first.
# uname -a
OpenBSD pbxfw 4.9 GENERIC#671 i386
# dmesg
OpenBSD 4.9 (GENERIC) #671: Wed Mar 2 07:09:00 MST 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.00GHz ("GenuineIntel" 686-class) 3 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,xTPR
real mem = 2137120768 (2038MB)
avail mem = 2092023808 (1995MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 02/09/05, BIOS32 rev. 0 @ 0xffe90,
SMBIOS rev. 2.3 @ 0xf0450 (74 entries)
bios0: vendor Dell Inc. version "A04" date 02/09/2005
bios0: Dell Inc. OptiPlex GX280
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC BOOT ASF! MCFG HPET
acpi0: wakeup devices VBTN(S4) PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5)
PCI4(S5) MOU_(S3) USB0(S3) USB1(S3) USB2(S3) USB3(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199MHz
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 8
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 4 (PCI1)
acpiprt1 at acpi0: bus 2 (PCI2)
acpiprt2 at acpi0: bus 3 (PCI3)
acpiprt3 at acpi0: bus 1 (PCI4)
acpiprt4 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C3
acpibtn0 at acpi0: VBTN
bios0: ROM list: 0xc0000/0xa800! 0xca800/0x1800!
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82915G Host" rev 0x04
ppb0 at pci0 dev 1 function 0 "Intel 82915G PCIE" rev 0x04: apic 8 int
16 (irq 11)
pci1 at ppb0 bus 1
vga1 at pci0 dev 2 function 0 "Intel 82915G Video" rev 0x04
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp0 at vga1
agp0 at intagp0: aperture at 0xc0000000, size 0x10000000
inteldrm0 at vga1: apic 8 int 16 (irq 11)
drm0 at inteldrm0
"Intel 82915G Video" rev 0x04 at pci0 dev 2 function 1 not configured
ppb1 at pci0 dev 28 function 0 "Intel 82801FB PCIE" rev 0x03: apic 8 int
16 (irq 11)
pci2 at ppb1 bus 2
bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
(0x4001): apic 8 int 16 (irq 11), address 00:11:43:7c:f3:91
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb2 at pci0 dev 28 function 1 "Intel 82801FB PCIE" rev 0x03
pci3 at ppb2 bus 3
uhci0 at pci0 dev 29 function 0 "Intel 82801FB USB" rev 0x03: apic 8 int
21 (irq 9)
uhci1 at pci0 dev 29 function 1 "Intel 82801FB USB" rev 0x03: apic 8 int
22 (irq 5)
uhci2 at pci0 dev 29 function 2 "Intel 82801FB USB" rev 0x03: apic 8 int
18 (irq 4)
uhci3 at pci0 dev 29 function 3 "Intel 82801FB USB" rev 0x03: apic 8 int
23 (irq 3)
ehci0 at pci0 dev 29 function 7 "Intel 82801FB USB" rev 0x03: apic 8 int
21 (irq 9)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd3
pci4 at ppb3 bus 4
re0 at pci4 dev 0 function 0 "D-Link DGE-528T" rev 0x10: RTL8169/8110SB
(0x1000), apic 8 int 16 (irq 11), address f0:7d:68:b8:62:95
rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 3
ichpcib0 at pci0 dev 31 function 0 "Intel 82801FB LPC" rev 0x03: PM disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801FB IDE" rev 0x03: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <SAMSUNG, CD-R/RW SW-252S, R902> ATAPI
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 31 function 2 "Intel 82801FB SATA" rev 0x03: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <WDC WD5000AAKS-00UU3A0>
wd0: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
ichiic0 at pci0 dev 31 function 3 "Intel 82801FB SMBus" rev 0x03: SMI
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-6400CL5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
# cat /etc/pf.conf
# $OpenBSD: pf.conf,v 1.49 2009/09/17 06:39:03 jmc Exp $
#
# See pf.conf(5) for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.
ext_if = "bge0"
int_if = "re0"
set skip on lo
pass out quick log on $ext_if inet from 192.168.0.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.230.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.231.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.239.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.240.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.241.0/24 nat-to $ext_if
pass out quick log on $ext_if inet from 192.168.242.0/24 nat-to $ext_if
pass in quick log on $ext_if inet proto {tcp, udp} from any to $ext_if
port ssh
pass in quick log on $ext_if inet proto icmp from any to $ext_if
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1056 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1061 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1062 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1070 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1074 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1088 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
1112 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
5060 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
8065 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
18060 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
30000 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
30001 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
40002 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
49152:65535 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
5004:5035 rdr-to 192.168.230.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
16400:17390 rdr-to 192.168.230.102
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
17400:17500 rdr-to 192.168.230.103
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
18400:19390 rdr-to 192.168.230.104
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
20400:21390 rdr-to 192.168.231.102
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
21400:21449 rdr-to 192.168.241.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
22400:22449 rdr-to 192.168.242.101
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
24400:24449 rdr-to 192.168.240.102
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
25400:25500 rdr-to 192.168.0.8
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port
8080 rdr-to 192.168.231.2 port 80
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 85
rdr-to 192.168.240.101 port 1062
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 86
rdr-to 192.168.242.101 port 1062
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 87
rdr-to 192.168.241.101 port 1062
pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 89
rdr-to 192.168.231.101 port 1062
pass in log on $ext_if from any to any
pass out log on $ext_if from any to any
pass log # to establish keep-state
# ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 1 0.0 0.0 480 332 ?? Is 9:39PM 0:00.01 /sbin/init
_syslogd 16956 0.0 0.0 516 728 ?? S 9:39PM 0:01.14
syslogd -a /var/spool/postfix/dev/log -a /var/www/dev/log -a
/var/empty/dev/log
root 20737 0.0 0.0 472 672 ?? Is 9:39PM 0:00.00
syslogd: [priv] (syslogd)
root 736 0.0 0.0 412 396 ?? Is 9:39PM 0:00.03
pflogd: [priv] (pflogd)
_pflogd 10358 0.0 0.0 476 356 ?? S 9:39PM 0:00.50
pflogd: [running] -s 160 -i pflog0 -f /var/log/pflog (pflogd)
_ntp 11468 0.0 0.0 544 960 ?? I 9:39PM 0:00.04 ntpd:
ntp engine (ntpd)
root 18585 0.0 0.0 508 872 ?? Is 9:39PM 0:00.00 ntpd:
[priv] (ntpd)
_ntp 9153 0.0 0.0 660 884 ?? I 9:39PM 0:00.02 ntpd:
dns engine (ntpd)
root 11287 0.0 0.1 616 1244 ?? Is 9:39PM 0:00.13
/usr/sbin/sshd
root 8482 0.0 0.0 560 728 ?? Is 9:39PM 0:00.00 inetd
_dnsmasq 29798 0.0 0.0 596 936 ?? I 9:39PM 0:00.05
/usr/local/sbin/dnsmasq
root 1946 0.0 0.0 552 816 ?? Is 9:39PM 0:00.09 cron
root 10375 0.0 0.1 568 1528 ?? Ss 9:39PM 0:00.30
/usr/local/libexec/postfix/master
_postfix 18950 0.0 0.1 660 1668 ?? S 9:39PM 0:00.29 qmgr
-l -t fifo -u -c
root 2691 0.0 0.1 3440 2600 ?? Ss 7:39AM 0:00.14 sshd:
root@ttyp0 (sshd)
root 8552 0.0 0.1 3452 2724 ?? Is 8:09AM 0:01.65 sshd:
root@ttyp1 (sshd)
_postfix 27294 0.0 0.1 452 1536 ?? I 2:17PM 0:00.01 pickup
-l -t fifo -u -c
root 27041 0.0 0.1 3420 2628 ?? Is 2:31PM 0:00.06 sshd:
root@ttyp2 (sshd)
root 21966 0.0 0.0 556 476 p0 Ss 7:39AM 0:00.02 -ksh (ksh)
root 2216 0.0 0.0 288 212 p0 R+ 3:14PM 0:00.00 ps -aux
root 7010 0.0 0.0 584 468 p1 Is+ 8:10AM 0:00.01 -ksh (ksh)
root 31137 0.0 0.0 472 484 p2 Is+ 2:31PM 0:00.01 -ksh (ksh)
root 16961 0.0 0.0 476 756 C0 Is+ 9:39PM 0:00.00
/usr/libexec/getty std.9600 ttyC0
root 7681 0.0 0.0 400 756 C1 Is+ 9:39PM 0:00.00
/usr/libexec/getty std.9600 ttyC1
root 12426 0.0 0.0 324 756 C2 Is+ 9:39PM 0:00.00
/usr/libexec/getty std.9600 ttyC2
root 32624 0.0 0.0 364 760 C3 Is+ 9:39PM 0:00.00
/usr/libexec/getty std.9600 ttyC3
root 4144 0.0 0.0 296 760 C5 Is+ 9:39PM 0:00.00
/usr/libexec/getty std.9600 ttyC5
# pkg_info
dnsmasq-2.55 caching DNS forwarder and DHCP server
gd-2.0.35p0 library for dynamic creation of images
gettext-0.18.1p0 GNU gettext
jpeg-8b IJG's JPEG compression utilities
libdnet-1.12p1 portable low-level networking library
libiconv-1.13p2 character set conversion library
lrzsz-0.12.20p0 receive/send files via X/Y/ZMODEM protocol
lua-5.1.4p1 powerful, light-weight programming language
lzo2-2.04 portable speedy lossless data compression library
nano-2.2.6 Pico editor clone with enhancements
nmap-5.21p3 scan ports and fingerprint stack of network hosts
oidentd-2.0.7p1 ident daemon with custom responses and NAT support
pcre-8.02p1 perl-compatible regular expression library
pfstat-2.3p1 packet filter statistics visualization
png-1.2.44 library for manipulating PNG images
postfix-2.8.20110113 fast, secure sendmail replacement
trafshow-3.1 full screen visualization of network traffic
So, down to the nitty gritty.
Jun 15 09:41:21 pbxfw /bsd: pf: state key linking mismatch! dir=OUT,
if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060,
proto=17, found af=2, a0: 192.168.230.101:5060, a1:
187.170.255.239:5060, proto=17
Jun 17 12:02:55 pbxfw /bsd: pf: state key linking mismatch! dir=OUT,
if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060,
proto=17, found af=2, a0: 192.168.230.101:5060, a1:
187.170.255.239:5060, proto=17
Is the only error output ive found on the problem.
So the problem, has to do with the ip 187.170.255.239,
239.255.170.187.in-addr.arpa domain name pointer
dsl-187-170-255-239-dyn.prod-infinitum.com.mx.
Our system has no relation at all with this ip.
But somehow our NAT translation at random intervals, decides to
redirects traffic to that ip instead of the intended destination.
Sofar we have primarily noted the problem towards 130.244.190.46 and
130.244.190.42, that are our providers sip gateways.
Since the only thing beeing used on the connection is a PBx solution.
A google on that perticular IP, gives a simular dmesg error output in
this post:
http://www.mail-archive.com/misc@openbsd.org/msg95116.html
But in his case, the system hangs, our system keeps on going.
And instead interferes with the connection of phonecalls.
since the problem was discovered ive set up pf to log the first packet
of every new state,
and then that is tcpdump thru tcpdump -n -e -ttt -s 1600 -vvv -XX to a
ascii log using the
http://www.openbsd.org/faq/pf/logging.html syslog method.
Jun 22 15:40:06.212694 rule 26/(match) [uid 0, pid 20284] pass in on
bge0: 130.244.190.46.5060 > 212.247.80.66.5060: udp 442 (DF) [tos 0xb8]
(ttl 56, id 0, len 470)
0000: 45b8 01d6 0000 4000 3811 da02 82f4 be2e
E\M-8.\M-V..@.8.\M-Z..\M-t\M->.
0010: d4f7 5042 13c4 13c4 01c2 f6b9 4259 4520
\M-T\M-wPB.\M-D.\M-D.\M-B\M-v\M-9BYE
0020: 7369 703a 3835 3933 4032 3132 2e32 3437 sip:8593@212.247
0030: 2e38 302e 3636 2053 4950 2f32 .80.66 SIP/2
Jun 22 15:40:06.307515 rule 60/(match) [uid 0, pid 20284] pass in on
re0: 192.168.230.101.5060 > 187.170.255.239.5060: udp 550 (ttl 64, id
33961, len 578)
0000: 4500 0242 84a9 0000 4011 9159 c0a8 e665
E..B.\M-)..@..Y\M-@\M-(\M-fe
0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f
\M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/
0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via:
0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP
Jun 22 15:40:06.307526 rule 0/(match) [uid 0, pid 20284] pass out on
bge0: 192.168.230.101.5060 > 187.170.255.239.5060: udp 550 (ttl 63, id
33961, len 578, bad cksum 9159! differs by 100)
0000: 4500 0242 84a9 0000 3f11 9159 c0a8 e665
E..B.\M-)..?..Y\M-@\M-(\M-fe
0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f
\M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/
0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via:
0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP
and on a side note, if anyone has a suggestion how to actually get the
complete package logged, and not just the first snap, it would be nice,
openbsd tcpdump seems to not support -s 0 as snaplen, to get the whole
thing.
anyway, that log snippet, is 130.244.190.46 asking us to setup a sip
connection with them on 5060,
but our respond to that ip, goes to 187.170.255.239. and the connection
fails.
another side note would be about the rampant amount of bad ckdsum on udp
traffic, if anyone would care to chime in about that.
Since about 98% of all udp packets gets a bad cksum.
but my main problem and concern is this 187.170.255.239, and why they
should get my phonecalls.
Regards
Magnus
