Lets get some standard stuff out of the way first. # uname -a OpenBSD pbxfw 4.9 GENERIC#671 i386
# dmesg OpenBSD 4.9 (GENERIC) #671: Wed Mar 2 07:09:00 MST 2011 [email protected]:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel(R) Pentium(R) 4 CPU 3.00GHz ("GenuineIntel" 686-class) 3 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,xTPR real mem = 2137120768 (2038MB) avail mem = 2092023808 (1995MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 02/09/05, BIOS32 rev. 0 @ 0xffe90, SMBIOS rev. 2.3 @ 0xf0450 (74 entries) bios0: vendor Dell Inc. version "A04" date 02/09/2005 bios0: Dell Inc. OptiPlex GX280 acpi0 at bios0: rev 0 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP SSDT APIC BOOT ASF! MCFG HPET acpi0: wakeup devices VBTN(S4) PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5) PCI4(S5) MOU_(S3) USB0(S3) USB1(S3) USB2(S3) USB3(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 199MHz ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 8 acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 4 (PCI1) acpiprt1 at acpi0: bus 2 (PCI2) acpiprt2 at acpi0: bus 3 (PCI3) acpiprt3 at acpi0: bus 1 (PCI4) acpiprt4 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0: C3 acpibtn0 at acpi0: VBTN bios0: ROM list: 0xc0000/0xa800! 0xca800/0x1800! pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel 82915G Host" rev 0x04 ppb0 at pci0 dev 1 function 0 "Intel 82915G PCIE" rev 0x04: apic 8 int 16 (irq 11) pci1 at ppb0 bus 1 vga1 at pci0 dev 2 function 0 "Intel 82915G Video" rev 0x04 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) intagp0 at vga1 agp0 at intagp0: aperture at 0xc0000000, size 0x10000000 inteldrm0 at vga1: apic 8 int 16 (irq 11) drm0 at inteldrm0 "Intel 82915G Video" rev 0x04 at pci0 dev 2 function 1 not configured ppb1 at pci0 dev 28 function 0 "Intel 82801FB PCIE" rev 0x03: apic 8 int 16 (irq 11) pci2 at ppb1 bus 2 bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1 (0x4001): apic 8 int 16 (irq 11), address 00:11:43:7c:f3:91 brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0 ppb2 at pci0 dev 28 function 1 "Intel 82801FB PCIE" rev 0x03 pci3 at ppb2 bus 3 uhci0 at pci0 dev 29 function 0 "Intel 82801FB USB" rev 0x03: apic 8 int 21 (irq 9) uhci1 at pci0 dev 29 function 1 "Intel 82801FB USB" rev 0x03: apic 8 int 22 (irq 5) uhci2 at pci0 dev 29 function 2 "Intel 82801FB USB" rev 0x03: apic 8 int 18 (irq 4) uhci3 at pci0 dev 29 function 3 "Intel 82801FB USB" rev 0x03: apic 8 int 23 (irq 3) ehci0 at pci0 dev 29 function 7 "Intel 82801FB USB" rev 0x03: apic 8 int 21 (irq 9) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd3 pci4 at ppb3 bus 4 re0 at pci4 dev 0 function 0 "D-Link DGE-528T" rev 0x10: RTL8169/8110SB (0x1000), apic 8 int 16 (irq 11), address f0:7d:68:b8:62:95 rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 3 ichpcib0 at pci0 dev 31 function 0 "Intel 82801FB LPC" rev 0x03: PM disabled pciide0 at pci0 dev 31 function 1 "Intel 82801FB IDE" rev 0x03: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility atapiscsi0 at pciide0 channel 0 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <SAMSUNG, CD-R/RW SW-252S, R902> ATAPI 5/cdrom removable cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 pciide0: channel 1 ignored (disabled) pciide1 at pci0 dev 31 function 2 "Intel 82801FB SATA" rev 0x03: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt wd0 at pciide1 channel 0 drive 0: <WDC WD5000AAKS-00UU3A0> wd0: 16-sector PIO, LBA48, 476940MB, 976773168 sectors wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6 ichiic0 at pci0 dev 31 function 3 "Intel 82801FB SMBus" rev 0x03: SMI iic0 at ichiic0 spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-6400CL5 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci3: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at ichpcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 mtrr: Pentium Pro MTRR support vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root root on wd0a swap on wd0b dump on wd0b # cat /etc/pf.conf # $OpenBSD: pf.conf,v 1.49 2009/09/17 06:39:03 jmc Exp $ # # See pf.conf(5) for syntax and examples. # Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1 # in /etc/sysctl.conf if packets are to be forwarded between interfaces. ext_if = "bge0" int_if = "re0" set skip on lo pass out quick log on $ext_if inet from 192.168.0.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.230.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.231.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.239.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.240.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.241.0/24 nat-to $ext_if pass out quick log on $ext_if inet from 192.168.242.0/24 nat-to $ext_if pass in quick log on $ext_if inet proto {tcp, udp} from any to $ext_if port ssh pass in quick log on $ext_if inet proto icmp from any to $ext_if pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1056 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1061 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1062 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1070 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1074 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1088 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 1112 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 5060 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 8065 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 18060 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 30000 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 30001 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 40002 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 49152:65535 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 5004:5035 rdr-to 192.168.230.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 16400:17390 rdr-to 192.168.230.102 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 17400:17500 rdr-to 192.168.230.103 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 18400:19390 rdr-to 192.168.230.104 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 20400:21390 rdr-to 192.168.231.102 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 21400:21449 rdr-to 192.168.241.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 22400:22449 rdr-to 192.168.242.101 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 24400:24449 rdr-to 192.168.240.102 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 25400:25500 rdr-to 192.168.0.8 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 8080 rdr-to 192.168.231.2 port 80 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 85 rdr-to 192.168.240.101 port 1062 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 86 rdr-to 192.168.242.101 port 1062 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 87 rdr-to 192.168.241.101 port 1062 pass in quick log on $ext_if proto {tcp,udp} from any to $ext_if port 89 rdr-to 192.168.231.101 port 1062 pass in log on $ext_if from any to any pass out log on $ext_if from any to any pass log # to establish keep-state # ps aux USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 1 0.0 0.0 480 332 ?? Is 9:39PM 0:00.01 /sbin/init _syslogd 16956 0.0 0.0 516 728 ?? S 9:39PM 0:01.14 syslogd -a /var/spool/postfix/dev/log -a /var/www/dev/log -a /var/empty/dev/log root 20737 0.0 0.0 472 672 ?? Is 9:39PM 0:00.00 syslogd: [priv] (syslogd) root 736 0.0 0.0 412 396 ?? Is 9:39PM 0:00.03 pflogd: [priv] (pflogd) _pflogd 10358 0.0 0.0 476 356 ?? S 9:39PM 0:00.50 pflogd: [running] -s 160 -i pflog0 -f /var/log/pflog (pflogd) _ntp 11468 0.0 0.0 544 960 ?? I 9:39PM 0:00.04 ntpd: ntp engine (ntpd) root 18585 0.0 0.0 508 872 ?? Is 9:39PM 0:00.00 ntpd: [priv] (ntpd) _ntp 9153 0.0 0.0 660 884 ?? I 9:39PM 0:00.02 ntpd: dns engine (ntpd) root 11287 0.0 0.1 616 1244 ?? Is 9:39PM 0:00.13 /usr/sbin/sshd root 8482 0.0 0.0 560 728 ?? Is 9:39PM 0:00.00 inetd _dnsmasq 29798 0.0 0.0 596 936 ?? I 9:39PM 0:00.05 /usr/local/sbin/dnsmasq root 1946 0.0 0.0 552 816 ?? Is 9:39PM 0:00.09 cron root 10375 0.0 0.1 568 1528 ?? Ss 9:39PM 0:00.30 /usr/local/libexec/postfix/master _postfix 18950 0.0 0.1 660 1668 ?? S 9:39PM 0:00.29 qmgr -l -t fifo -u -c root 2691 0.0 0.1 3440 2600 ?? Ss 7:39AM 0:00.14 sshd: root@ttyp0 (sshd) root 8552 0.0 0.1 3452 2724 ?? Is 8:09AM 0:01.65 sshd: root@ttyp1 (sshd) _postfix 27294 0.0 0.1 452 1536 ?? I 2:17PM 0:00.01 pickup -l -t fifo -u -c root 27041 0.0 0.1 3420 2628 ?? Is 2:31PM 0:00.06 sshd: root@ttyp2 (sshd) root 21966 0.0 0.0 556 476 p0 Ss 7:39AM 0:00.02 -ksh (ksh) root 2216 0.0 0.0 288 212 p0 R+ 3:14PM 0:00.00 ps -aux root 7010 0.0 0.0 584 468 p1 Is+ 8:10AM 0:00.01 -ksh (ksh) root 31137 0.0 0.0 472 484 p2 Is+ 2:31PM 0:00.01 -ksh (ksh) root 16961 0.0 0.0 476 756 C0 Is+ 9:39PM 0:00.00 /usr/libexec/getty std.9600 ttyC0 root 7681 0.0 0.0 400 756 C1 Is+ 9:39PM 0:00.00 /usr/libexec/getty std.9600 ttyC1 root 12426 0.0 0.0 324 756 C2 Is+ 9:39PM 0:00.00 /usr/libexec/getty std.9600 ttyC2 root 32624 0.0 0.0 364 760 C3 Is+ 9:39PM 0:00.00 /usr/libexec/getty std.9600 ttyC3 root 4144 0.0 0.0 296 760 C5 Is+ 9:39PM 0:00.00 /usr/libexec/getty std.9600 ttyC5 # pkg_info dnsmasq-2.55 caching DNS forwarder and DHCP server gd-2.0.35p0 library for dynamic creation of images gettext-0.18.1p0 GNU gettext jpeg-8b IJG's JPEG compression utilities libdnet-1.12p1 portable low-level networking library libiconv-1.13p2 character set conversion library lrzsz-0.12.20p0 receive/send files via X/Y/ZMODEM protocol lua-5.1.4p1 powerful, light-weight programming language lzo2-2.04 portable speedy lossless data compression library nano-2.2.6 Pico editor clone with enhancements nmap-5.21p3 scan ports and fingerprint stack of network hosts oidentd-2.0.7p1 ident daemon with custom responses and NAT support pcre-8.02p1 perl-compatible regular expression library pfstat-2.3p1 packet filter statistics visualization png-1.2.44 library for manipulating PNG images postfix-2.8.20110113 fast, secure sendmail replacement trafshow-3.1 full screen visualization of network traffic So, down to the nitty gritty. Jun 15 09:41:21 pbxfw /bsd: pf: state key linking mismatch! dir=OUT, if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060, proto=17, found af=2, a0: 192.168.230.101:5060, a1: 187.170.255.239:5060, proto=17 Jun 17 12:02:55 pbxfw /bsd: pf: state key linking mismatch! dir=OUT, if=re0, stored af=2, a0: 130.244.190.46:5060, a1: 192.168.230.101:5060, proto=17, found af=2, a0: 192.168.230.101:5060, a1: 187.170.255.239:5060, proto=17 Is the only error output ive found on the problem. So the problem, has to do with the ip 187.170.255.239, 239.255.170.187.in-addr.arpa domain name pointer dsl-187-170-255-239-dyn.prod-infinitum.com.mx. Our system has no relation at all with this ip. But somehow our NAT translation at random intervals, decides to redirects traffic to that ip instead of the intended destination. Sofar we have primarily noted the problem towards 130.244.190.46 and 130.244.190.42, that are our providers sip gateways. Since the only thing beeing used on the connection is a PBx solution. A google on that perticular IP, gives a simular dmesg error output in this post: http://www.mail-archive.com/[email protected]/msg95116.html But in his case, the system hangs, our system keeps on going. And instead interferes with the connection of phonecalls. since the problem was discovered ive set up pf to log the first packet of every new state, and then that is tcpdump thru tcpdump -n -e -ttt -s 1600 -vvv -XX to a ascii log using the http://www.openbsd.org/faq/pf/logging.html syslog method. Jun 22 15:40:06.212694 rule 26/(match) [uid 0, pid 20284] pass in on bge0: 130.244.190.46.5060 > 212.247.80.66.5060: udp 442 (DF) [tos 0xb8] (ttl 56, id 0, len 470) 0000: 45b8 01d6 0000 4000 3811 da02 82f4 be2e E\M-8.\[email protected].\M-Z..\M-t\M->. 0010: d4f7 5042 13c4 13c4 01c2 f6b9 4259 4520 \M-T\M-wPB.\M-D.\M-D.\M-B\M-v\M-9BYE 0020: 7369 703a 3835 3933 4032 3132 2e32 3437 sip:[email protected] 0030: 2e38 302e 3636 2053 4950 2f32 .80.66 SIP/2 Jun 22 15:40:06.307515 rule 60/(match) [uid 0, pid 20284] pass in on re0: 192.168.230.101.5060 > 187.170.255.239.5060: udp 550 (ttl 64, id 33961, len 578) 0000: 4500 0242 84a9 0000 4011 9159 c0a8 e665 E..B.\M-)[email protected]\M-@\M-(\M-fe 0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f \M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/ 0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via: 0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP Jun 22 15:40:06.307526 rule 0/(match) [uid 0, pid 20284] pass out on bge0: 192.168.230.101.5060 > 187.170.255.239.5060: udp 550 (ttl 63, id 33961, len 578, bad cksum 9159! differs by 100) 0000: 4500 0242 84a9 0000 3f11 9159 c0a8 e665 E..B.\M-)..?..Y\M-@\M-(\M-fe 0010: bbaa ffef 13c4 13c4 022e 9dc3 5349 502f \M-;\M-*\M^?\M-o.\M-D.\M-D...\M-CSIP/ 0020: 322e 3020 3230 3020 4f4b 0d0a 5669 613a 2.0 200 OK..Via: 0030: 2053 4950 2f32 2e30 2f55 4450 SIP/2.0/UDP and on a side note, if anyone has a suggestion how to actually get the complete package logged, and not just the first snap, it would be nice, openbsd tcpdump seems to not support -s 0 as snaplen, to get the whole thing. anyway, that log snippet, is 130.244.190.46 asking us to setup a sip connection with them on 5060, but our respond to that ip, goes to 187.170.255.239. and the connection fails. another side note would be about the rampant amount of bad ckdsum on udp traffic, if anyone would care to chime in about that. Since about 98% of all udp packets get a bad cksum. but my main problem and concern is this 187.170.255.239, and why they should get my phonecalls. Regards Magnus
