Re: [6.2] pf nat-to ignoring static-port?

[Martin Hlavatý]

I tried both (pass out quick right below nat-to line and also let it
go to the end of my rulebase) and it didnt change anything.

Martin

On Tue, Jan 23, 2018 at 3:19 PM, Michael Price <mich...@ectospheno.com> wrote:
> The lack of a quick keyword on that line makes me wonder if you have a later
> rule that is matching.
>
> Michael
>
> On Mon, Jan 22, 2018 at 5:34 PM Martin Hlavatý <mar...@hlavaty.eu> wrote:
>>
>> Interesting. I did a few tests now, and here are results.
>>
>> This doesn't map ports statically on 6.2 but does on 5.9:
>> pass out from 10.11.12.13 to any nat-to 1.2.3.4 static-port
>>
>> This works fine:
>> pass out quick from 10.11.12.13 to any nat-to 1.2.3.4 static-port
>>
>> This works fine too:
>> match out from 10.11.12.13 to any nat-to 1.2.3.4 static-port
>>
>> Martin
>>
>>
>> On Mon, Jan 22, 2018 at 8:23 PM, Michael Price <mich...@ectospheno.com>
>> wrote:
>> > It appears to be working on two boxes I checked using a match out rule.
>> > I’m
>> > not using a binat-to line.
>> >
>> > Michael
>> >
>> > On Mon, Jan 22, 2018 at 10:49 AM Martin Hlavatý <mar...@hlavaty.eu>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >> in December I upgraded from 5.9 to 6.2 (including 6.0 and
>> >> 6.1) and shortly after that few customers contacted me
>> >> that they are getting nat type 3 on their xbox\playstation.
>> >> When doing some investigation, I noticed that binat-to
>> >> rules have static-port specified, but looking into states
>> >> table, they were actually not mapped statically. Failing
>> >> over to backup box still running 5.9 with identical ruleset,
>> >> ports are actually mapped statically and online gaming
>> >> on consoles works fine.
>> >>
>> >> I tried to do some investigation, but am not aware of any
>> >> change in pf syntax. So wondering if anyone would be
>> >> able to confirm this behavior?
>> >>
>> >> this is in rules:
>> >>
>> >>   pass out inet from 10.11.12.13 to any flags S/SA nat-to 5.6.7.8
>> >> static-port
>> >>   pass in inet from any to 5.6.7.8 flags S/SA rdr-to 10.11.12.13
>> >>
>> >> and example of states:
>> >>
>> >> all udp 5.6.7.8:65350 (10.11.12.13:3074) -> 52.166.52.75:1986
>> >> MULTIPLE:MULTIPLE
>> >> all tcp 5.6.7.8:63203 (10.11.12.13:38010) -> 31.13.91.33:443
>> >> ESTABLISHED:ESTABLISHED
>> >> all tcp 5.6.7.8:59711 (10.11.12.13:42530) -> 74.125.133.188:5228
>> >> ESTABLISHED:ESTABLISHED
>> >>
>> >>
>> >>
>> >> Regards,
>> >> Martin
>> >>
>> >

Re: [6.2] pf nat-to ignoring static-port?

[Martin Hlavatý]

Interesting. I did a few tests now, and here are results.

This doesn't map ports statically on 6.2 but does on 5.9:
pass out from 10.11.12.13 to any nat-to 1.2.3.4 static-port

This works fine:
pass out quick from 10.11.12.13 to any nat-to 1.2.3.4 static-port

This works fine too:
match out from 10.11.12.13 to any nat-to 1.2.3.4 static-port

Martin


On Mon, Jan 22, 2018 at 8:23 PM, Michael Price <mich...@ectospheno.com> wrote:
> It appears to be working on two boxes I checked using a match out rule. I’m
> not using a binat-to line.
>
> Michael
>
> On Mon, Jan 22, 2018 at 10:49 AM Martin Hlavatý <mar...@hlavaty.eu> wrote:
>>
>> Hello everyone,
>> in December I upgraded from 5.9 to 6.2 (including 6.0 and
>> 6.1) and shortly after that few customers contacted me
>> that they are getting nat type 3 on their xbox\playstation.
>> When doing some investigation, I noticed that binat-to
>> rules have static-port specified, but looking into states
>> table, they were actually not mapped statically. Failing
>> over to backup box still running 5.9 with identical ruleset,
>> ports are actually mapped statically and online gaming
>> on consoles works fine.
>>
>> I tried to do some investigation, but am not aware of any
>> change in pf syntax. So wondering if anyone would be
>> able to confirm this behavior?
>>
>> this is in rules:
>>
>>   pass out inet from 10.11.12.13 to any flags S/SA nat-to 5.6.7.8
>> static-port
>>   pass in inet from any to 5.6.7.8 flags S/SA rdr-to 10.11.12.13
>>
>> and example of states:
>>
>> all udp 5.6.7.8:65350 (10.11.12.13:3074) -> 52.166.52.75:1986
>> MULTIPLE:MULTIPLE
>> all tcp 5.6.7.8:63203 (10.11.12.13:38010) -> 31.13.91.33:443
>> ESTABLISHED:ESTABLISHED
>> all tcp 5.6.7.8:59711 (10.11.12.13:42530) -> 74.125.133.188:5228
>> ESTABLISHED:ESTABLISHED
>>
>>
>>
>> Regards,
>> Martin
>>
>

[6.2] pf nat-to ignoring static-port?

[Martin Hlavatý]

Hello everyone,
in December I upgraded from 5.9 to 6.2 (including 6.0 and
6.1) and shortly after that few customers contacted me
that they are getting nat type 3 on their xbox\playstation.
When doing some investigation, I noticed that binat-to
rules have static-port specified, but looking into states
table, they were actually not mapped statically. Failing
over to backup box still running 5.9 with identical ruleset,
ports are actually mapped statically and online gaming
on consoles works fine.

I tried to do some investigation, but am not aware of any
change in pf syntax. So wondering if anyone would be
able to confirm this behavior?

this is in rules:

  pass out inet from 10.11.12.13 to any flags S/SA nat-to 5.6.7.8 static-port
  pass in inet from any to 5.6.7.8 flags S/SA rdr-to 10.11.12.13

and example of states:

all udp 5.6.7.8:65350 (10.11.12.13:3074) -> 52.166.52.75:1986
MULTIPLE:MULTIPLE
all tcp 5.6.7.8:63203 (10.11.12.13:38010) -> 31.13.91.33:443
ESTABLISHED:ESTABLISHED
all tcp 5.6.7.8:59711 (10.11.12.13:42530) -> 74.125.133.188:5228
ESTABLISHED:ESTABLISHED



Regards,
Martin

Re: mbuf leak in carp with ipv6

[Martin Hlavatý]

I have same errors in log and same behavior on one of my firewall
cluster. Carp flapping is a result of master system lagging and not
sending carp packets on time. You can workaround it by setting advbase
to some larger value, e.g. 10, but it will obviously still keep
lagging and dropping packets on NIC when these lags occur.

On my graphs I noticed peak in pf match ratio, peak in packesloss on
wan iface, higher cpu load and utilization when this ocucrs. I am not
graphing mbufs, it is around 500-900 usually, kern.maxclusters was set
to 32768 and and peak in netstat -m was 32770, so I raised that to
64000. It had no effect, it did not peak above 32770 and lags still
occur.

Dont really know what to do with this, I am sort of just waiting for
5.9 release and its unlocked networking stack hoping that multicore
cpu will manage that load.

On Thu, Jan 7, 2016 at 10:52 AM, Håkon Lerring  wrote:
>> On 05 Jan 2016, at 13:06, Stefan Sperling  wrote:
>>
>> On Tue, Jan 05, 2016 at 12:29:43PM +0100, Håkon Lerring wrote:
>>> Hello misc.
>>>
>>> I was investigating a problem with a firewall that goes AWOL every week.
> It
>>> happens only if i activate an ipv6 address on a carp interface. The carp
> log
>>> has this message:
>>>
>>> Jan  5 12:10:06  /bsd: carp: packet size 48 too small
>>>
>>> I think i have narrowed down the leak to the handling of too small
>>> ipv6-packets:
>>>
>>> --- ip_carp.c.orig   2016-01-05 12:18:03.0 +0100
>>> +++ ip_carp.c2016-01-05 12:18:30.0 +0100
>>> @@ -562,6 +562,7 @@
>>>  if ((m = m_pullup(m, *offp + sizeof(*ch))) == NULL) {
>>>  carpstats.carps_badlen++;
>>>  CARP_LOG(LOG_INFO, sc, ("packet size %u too small", len));
>>> +m_freem(m);
>>>  return (IPPROTO_DONE);
>>>  }
>>>  ch = (struct carp_header *)(mtod(m, caddr_t) + *offp);
>>>
>>>
>>> I have not yet tested this patch since this is a production system. Why
> the
>>> other machine is sending incomplete packets is another question i'm
> currently
>>> investigating.
>>
>> Your patch effectively just calls m_freem(NULL);
>> And m_pullup already frees the mbuf on failure.
>>
>> Can you describe the actual problem you're seeing with more words than
> "AWOL"?
>
> Sorry for my hasty conclusion about my problem (i see that i might be wrong
> about the cause), i will explain it in more detail.
>
> I have 2 firewalls/routers with multiple carp interfaces (11). 1 on uplink
> (carp2) and the rest for different internal networks. All the carp
interfaces
> are set up as a preemptive failover in case the master goes down. The
problem
> has two symptoms but i believe they might have the same cause (as they
occur
> at the same time).
>
> The first symptom is that at what seems to be random times the carp2
interface
> (see ifconfig output below) on the backup-machine starts flapping between
> backup and master (the designated master does not do the same) and complain
> about too small packets:
>
> Jan  7 09:00:11 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan  7 09:00:12 hostname /bsd: carp2: state transition: MASTER -> BACKUP
> Jan  7 09:00:12 hostname /bsd: carp: packet size 48 too small
> Jan  7 09:00:18 hostname last message repeated 6 times
> Jan  7 09:00:19 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan  7 09:00:19 hostname /bsd: carp2: state transition: MASTER -> BACKUP
> Jan  7 09:00:19 hostname /bsd: carp: packet size 48 too small
> Jan  7 09:00:25 hostname last message repeated 6 times
> Jan  7 09:00:26 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan  7 09:00:26 hostname /bsd: carp2: state transition: MASTER -> BACKUP
>
> etc..
>
> The designated master does also log that "packet size 48 too small"
>
> MASTER carp2:
> carp2: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:02
> description: uplink
> priority: 15
> carp: MASTER carpdev bnx0 vhid 2 advbase 1 advskew 0
> groups: carp
> status: master
> inet  netmask 0xfff8 broadcast 
> inet6 fe80::200:5eff:fe00:102%carp2 prefixlen 64 scopeid 0x16
> inet6  prefixlen 64
>
> BACKUP carp2:
> carp2: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:02
> description: uplink
> priority: 15
> carp: BACKUP carpdev bnx0 vhid 2 advbase 2 advskew 128
> groups: carp
> status: backup
> inet  netmask 0xfff8 broadcast 
> inet6 fe80::200:5eff:fe00:102%carp2 prefixlen 64 scopeid 0x16
> inet6  prefixlen 64
>
> The other carp interfaces on the same machine does not have this problem.
The
> only things different about carp2 from the other interfaces is that carp2
has
> an IPv6 address and is backed by a bnx interface instead of an em interface
> (though this does not seem to be of significance as i have tried it with em
> interface 

Re: pf match counter peak causes firewall to lag

[Martin Hlavatý]

On Sat, Nov 21, 2015 at 2:43 PM, Daniel Melameth <dan...@melameth.com> wrote:
> On Sat, Nov 21, 2015 at 6:21 AM, Martin Hlavatý <mar...@hlavaty.eu> wrote:
>> I have issues with firewall lags while there is peak in match
>> rule counter in pf. Normally it has match ratio of about
>> 1500/sec, but several times a day it jumps to somewhere
>> around 6k/sec and firewall lags, some traffic gets dropped.
>> This takes a few seconds.
>>
>> Lag causes system to delay sending carp packets and
>> sometimes backup box promotes itself to master and
>> immediately back to backup. Sadly, after sending inverse ARP.
>> I workarounded this issue by setting advbase to 10.
>>
>> Another problem is obviously with normal forwarding traffic,
>> like lags in online games or iptv streams.
>>
>> There is no visible raise in cpu utilization, but cpu load goes
>> from about 0.7 to 1.5 and there are packets getting dropped
>> on wan interface.
>>
>> Box is Core i3 530 on Supermicro X8SIL with 2x1GB RAM,
>> intel 40GB SSD, two 82574 and two 82571 NICs. In afternoon
>> hours it is loaded on 40k/25k tx/rx pps on wan interface.
>>
>> Looking to systat vmstat, LAN and WAN nics are getting
>> around 7.5k interrupts, while pfsync about 2.5-3k
>> and interrupts in top take about 60-70%.
>>
>> I tried to switch NICs for i350, but it had no effect, same
>> thing with openBSD versions, 5.6 5.7 and 5.8 have same
>> behavior. I also tried to replacing other hardware like CPU
>> for Xeon X3430 or motherboard S5500BC with Xeon E5620,
>> but without effect. Happens also on backup box when it
>> runs as master (same hw config).
>>
>> System is running GENERIC.MP stable amd64 kernel.
>>
>> I read in some discussions, that raising interrupt limit and
>> rx/tx queue in em(4) driver or using broadcoms instead
>> of intels might help, but didnt try it yet.
>>
>> Is there any way to determine what is causing the peaks
>> and how to prevent them or getting system powerful
>> enough to handle them?
>>
>> pfctl -si
>> Status: Enabled for 0 days 22:12:20  Debug: err
>>
>> State Table  Total Rate
>>   current entries66901
>>   searches  500333027562588.6/s
>>   inserts 47704143  596.7/s
>>   removals47637242  595.9/s
>> Counters
>>   match   96819915 1211.2/s
>>   bad-offset 00.0/s
>>   fragment18500.0/s
>>   short 860.0/s
>>   normalize 480.0/s
>>   memory7862289.8/s
>>   bad-timestamp  00.0/s
>>   congestion   3948624   49.4/s
>>   ip-option  243410.3/s
>>   proto-cksum00.0/s
>>   state-mismatch   1644853   20.6/s
>>   state-insert 4640.0/s
>>   state-limit00.0/s
>>   src-limit  00.0/s
>>   synproxy39480.0/s
>>   translate  00.0/s
>>   no-route   00.0/s
>>
>> kern.netlivelocks=1534
>>
>> netstat -si
>> em0 1500   1533962428 266567 955232172 0 0
>> em1 1500   979515291  8697 1526507571 0 0
>> em2 1500   6970941 0 140093911 0 0
>> em3*1500   0 00 0 0
>
> Are you doing packet queuing with pf?  What's the value of
> net.inet.ip.ifq.maxlen and net.inet.ip.ifq.drops?  You might want to
> try disabling any power-saving features on that hardware.
>

Yes, I am doing queuing

net.inet.ip.ifq.maxlen=1536
I modified this from original value of 768, but it has no effect

net.inet.ip.ifq.drops=3851664

pf match counter peak causes firewall to lag

[Martin Hlavatý]

Hello,

I have issues with firewall lags while there is peak in match
rule counter in pf. Normally it has match ratio of about
1500/sec, but several times a day it jumps to somewhere
around 6k/sec and firewall lags, some traffic gets dropped.
This takes a few seconds.

Lag causes system to delay sending carp packets and
sometimes backup box promotes itself to master and
immediately back to backup. Sadly, after sending inverse ARP.
I workarounded this issue by setting advbase to 10.

Another problem is obviously with normal forwarding traffic,
like lags in online games or iptv streams.

There is no visible raise in cpu utilization, but cpu load goes
from about 0.7 to 1.5 and there are packets getting dropped
on wan interface.

Box is Core i3 530 on Supermicro X8SIL with 2x1GB RAM,
intel 40GB SSD, two 82574 and two 82571 NICs. In afternoon
hours it is loaded on 40k/25k tx/rx pps on wan interface.

Looking to systat vmstat, LAN and WAN nics are getting
around 7.5k interrupts, while pfsync about 2.5-3k
and interrupts in top take about 60-70%.

I tried to switch NICs for i350, but it had no effect, same
thing with openBSD versions, 5.6 5.7 and 5.8 have same
behavior. I also tried to replacing other hardware like CPU
for Xeon X3430 or motherboard S5500BC with Xeon E5620,
but without effect. Happens also on backup box when it
runs as master (same hw config).

System is running GENERIC.MP stable amd64 kernel.

I read in some discussions, that raising interrupt limit and
rx/tx queue in em(4) driver or using broadcoms instead
of intels might help, but didnt try it yet.

Is there any way to determine what is causing the peaks
and how to prevent them or getting system powerful
enough to handle them?

pfctl -si
Status: Enabled for 0 days 22:12:20  Debug: err

State Table  Total Rate
  current entries66901
  searches  500333027562588.6/s
  inserts 47704143  596.7/s
  removals47637242  595.9/s
Counters
  match   96819915 1211.2/s
  bad-offset 00.0/s
  fragment18500.0/s
  short 860.0/s
  normalize 480.0/s
  memory7862289.8/s
  bad-timestamp  00.0/s
  congestion   3948624   49.4/s
  ip-option  243410.3/s
  proto-cksum00.0/s
  state-mismatch   1644853   20.6/s
  state-insert 4640.0/s
  state-limit00.0/s
  src-limit  00.0/s
  synproxy39480.0/s
  translate  00.0/s
  no-route   00.0/s

kern.netlivelocks=1534

netstat -si
em0 1500   1533962428 266567 955232172 0 0
em1 1500   979515291  8697 1526507571 0 0
em2 1500   6970941 0 140093911 0 0
em3*1500   0 00 0 0

OpenBSD 5.8-stable (GENERIC.MP) #1: Sun Nov 15 17:29:19 CET 2015
   :/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2121859072 (2023MB)
avail mem = 2053718016 (1958MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x9f000 (68 entries)
bios0: vendor American Megatrends Inc. version "1.1" date 05/27/2010
bios0: Supermicro X8SIL
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI DMAR SSDT EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) P0P3(S4) P0P4(S4) P0P5(S4) P0P6(S4)
BR1E(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4) USB3(S4)
USB4(S4) USB5(S4) U
SB6(S4) GBE_(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.75 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL
,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 133MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.34 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL