Re: random packet drops with syncookies/synproxy

2019-11-14 Thread Markus Wernig
On 09.11.2019 15:24, Claudio Jeker wrote:

>> So nobody is using syncookies/synproxy at all?
> 
> I guess that is a reasonably safe assumption. syncookies are rather new
> and probably need more battle testing.

OK, then I will send a bug report.

> synproxy never helped me much in
> case of a SYN attack since it will cause pf(4) to hit the state limit no
> matter what you do and then stuff starts to break.

Yes, synproxy will not help with that, but syncookies should. But the
syncookies entry in the man page also states that a connection opened
via syncookie will then run through synproxy, so the problem I'm seeing
might be in either one.

best /



Re: random packet drops with syncookies/synproxy

2019-11-09 Thread Claudio Jeker
On Sat, Nov 09, 2019 at 01:30:32PM +0100, Markus Wernig wrote:
> Hm, also no replies to that one :-)
> 
> On 11/6/19 8:15 PM, Markus Wernig wrote:
> 
> > So just to make sure: Is anybody using syncookies and/or synproxy in
> > production in a similar setup?
> 
> So nobody is using syncookies/synproxy at all?

I guess that is a reasonably safe assumption. syncookies are rather new
and probably need more battle testing. synproxy never helped me much in
case of a SYN attack since it will cause pf(4) to hit the state limit no
matter what you do and then stuff starts to break.

-- 
:wq Claudio



Re: random packet drops with syncookies/synproxy

2019-11-09 Thread Markus Wernig
Hm, also no replies to that one :-)

On 11/6/19 8:15 PM, Markus Wernig wrote:

> So just to make sure: Is anybody using syncookies and/or synproxy in
> production in a similar setup?

So nobody is using syncookies/synproxy at all?

best /m



Re: random packet drops with syncookies/synproxy

2019-11-06 Thread Markus Wernig
Hi again

Nobody has answered, so I suppose nobody else has this problem :-)
That's good.

So just to make sure: Is anybody using syncookies and/or synproxy in
production in a similar setup?

Thx /markus


On 11/4/19 8:35 PM, Markus Wernig wrote:
> Hi all
> 
> After being hit by some synflood waves recently I enabled syncookies on
> our OBSD 6.6 i386 CARP fw pair:
> 
> set syncookies always
> 
> This stopped the state table from filling up. But after some hours pf
> started (randomly?) dropping legitimate connection attempts, both on
> external->internal (dst-natted) and on internal->internal (not natted)
> connections (TCP only, afaict).
> 
> Looking at pflog and the rule number that blocked the packet, it seems
> that the preceding "pass quick" rules matching the packets were ignored.
> 
> The packets that were dropped were the ACK ones, so the SYN-SYNACK seems
> to have taken place. The client then usually retransmitted the ACK,
> which kept being dropped for ca. 15-20 seconds, after which time it was
> suddenly accepted and the connection established. Many times also only
> the first ACK was dropped, and the first retransmit was accepted.
> 
> So I disabled syncookies and set the relevant ~5 external->internal
> rules to synproxy state.
> 
> With that, the same behaviour happened within a few minutes.
> 
> During that time pfctl -vsi showed the "synproxy" counter increasing by
> multiple thousands per second (sic), while the state table entries
> remained stable around 500 (their normal value).
> 
> So I disabled the synproxy state again, but reloading the rules with
> pfctl was not enough, I had to reboot both boxes to stop them from
> dropping legitimate connections. With both syncookies and synproxy
> disabled, the problem does not occur.
> 
> Is anybody aware of anything that could trigger this behaviour? Or have
> any hint where I could look further? I have all the log files if more
> info is needed.
> 
> thx /markus
> 
> (btw. the behaviour was the same on 6.5)
> 



random packet drops with syncookies/synproxy

2019-11-04 Thread Markus Wernig
Hi all

After being hit by some synflood waves recently I enabled syncookies on
our OBSD 6.6 i386 CARP fw pair:

set syncookies always

This stopped the state table from filling up. But after some hours pf
started (randomly?) dropping legitimate connection attempts, both on
external->internal (dst-natted) and on internal->internal (not natted)
connections (TCP only, afaict).

Looking at pflog and the rule number that blocked the packet, it seems
that the preceding "pass quick" rules matching the packets were ignored.

The packets that were dropped were the ACK ones, so the SYN-SYNACK seems
to have taken place. The client then usually retransmitted the ACK,
which kept being dropped for ca. 15-20 seconds, after which time it was
suddenly accepted and the connection established. Many times also only
the first ACK was dropped, and the first retransmit was accepted.

So I disabled syncookies and set the relevant ~5 external->internal
rules to synproxy state.

With that, the same behaviour happened within a few minutes.

During that time pfctl -vsi showed the "synproxy" counter increasing by
multiple thousands per second (sic), while the state table entries
remained stable around 500 (their normal value).

So I disabled the synproxy state again, but reloading the rules with
pfctl was not enough, I had to reboot both boxes to stop them from
dropping legitimate connections. With both syncookies and synproxy
disabled, the problem does not occur.

Is anybody aware of anything that could trigger this behaviour? Or have
any hint where I could look further? I have all the log files if more
info is needed.

thx /markus

(btw. the behaviour was the same on 6.5)