Tim, from your problem description I can suggest you to check if you are not hitting
states hard limit with (note - during load when you can reproduce issue): pfctl -si pfctl -sm Default limit is: states hard limit 10000 -- Evgeniy On Thu, Jun 2, 2016 at 3:29 AM, Tim Korn <[email protected]> wrote: > Hi. I have a pair of openBSD boxes (5.8) setup as a core/firewall. I have > ten VLANs tied to a physical NIC (Intel 82599). This is a new setup and it > was just recently put in service. Traffic was fine (or at least we didn't > notice any issues) until a large job was run which roughly doubled traffic > going thru the firewall. Traffic rate is still extremely low... roughly 2k > packets per second on the interface in question and around 20Mb. I have > other identical openBSD boxes that don't use VLANs, and they pass multiple > gigs of traffic per second, so I'm having a hard time not leaning towards > it being a VLAN issue, however I don't know where to look to prove it. > > If a host in vlan100 pings a host in vlan101 I see packet loss on the first > few packets, than all subsequent packets pass. Stopping and restarting the > ping results in the same thing....first few pings lost, then responses and > never fail again until the ping is stopped and restarted. We see this > behavior with pretty much any new connection. I can replicate it > consistently with ICMP, TCP, and UDP traffic. > > PF ruleset is quite basic. Simple *pass in* rules on the VLANs and *pass > out* is allowed on all interfaces. icmp has a rule at the top saying "pass > log quick proto icmp". i really don't think theres a pf issue of any kind. > > I've run a tcpdump to confirm that packets come in on vlan100, and never > leave vlan101. Here is an example: > > Ping from host in vlan100 (you can see the seq start at 9. first 8 > never left the firewall): > [root@pakkit ~]# ping 10.95.1.50 > PING 10.95.1.50 (10.95.1.50) 56(84) bytes of data. > 64 bytes from 10.95.1.50: icmp_seq=9 ttl=63 time=0.263 ms > 64 bytes from 10.95.1.50: icmp_seq=10 ttl=63 time=0.341 ms > 64 bytes from 10.95.1.50: icmp_seq=11 ttl=63 time=0.335 ms > 64 bytes from 10.95.1.50: icmp_seq=12 ttl=63 time=0.348 ms > 64 bytes from 10.95.1.50: icmp_seq=13 ttl=63 time=0.348 ms > > > > tcpdump on vlan100 showing 13 echo requests: > [root@pci-ny2-fw1:~ (master)] tcpdump -neti vlan100 host 10.95.0.5 and > host 10.95.1.50 > tcpdump: listening on vlan100, link-type EN10MB > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1b:d8 00:0c:29:16:f7:bf 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1b:d8 00:0c:29:16:f7:bf 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1b:d8 00:0c:29:16:f7:bf 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1b:d8 00:0c:29:16:f7:bf 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 00:0c:29:16:f7:bf 00:00:5e:00:01:64 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1b:d8 00:0c:29:16:f7:bf 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > ^C > 1049 packets received by filter > 0 packets dropped by kernel > > > tcpdump on vlan101 showing only 5 echo requests: > [root@pci-ny2-fw1:/etc/ (master)] tcpdump -neti vlan101 host 10.95.0.5 > and host 10.95.1.50 > tcpdump: listening on vlan101, link-type EN10MB > 24:6e:96:04:1b:d8 24:6e:96:04:1c:84 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1c:84 00:00:5e:00:01:65 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 24:6e:96:04:1b:d8 24:6e:96:04:1c:84 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1c:84 00:00:5e:00:01:65 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 24:6e:96:04:1b:d8 24:6e:96:04:1c:84 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1c:84 00:00:5e:00:01:65 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 24:6e:96:04:1b:d8 24:6e:96:04:1c:84 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1c:84 00:00:5e:00:01:65 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > 24:6e:96:04:1b:d8 24:6e:96:04:1c:84 0800 98: 10.95.0.5 > 10.95.1.50: > icmp: echo request (DF) > 24:6e:96:04:1c:84 00:00:5e:00:01:65 0800 98: 10.95.1.50 > 10.95.0.5: > icmp: echo reply > ^C > 1975 packets received by filter > 0 packets dropped by kernel > > Any help would be greatly appreciated. This is causing massive slow downs > for all traffic flowing thru this firewall. Thank you for your time. > > -Tim > -- -- With regards, Eugene Sudyr
