Problem with Pf
Hi Guys, I hope I am posting on the right mailing list. I am sending you this email because I have been experiencing a lot of BAD State in pf recently. I don't know if this has been discussed previously. More and and more people are now using Oses that can adapt the TCP Windows Size. In pf, I could see that pf checks for the sequence number to make sure it is in the expected range. Therefore, pf will make the following check: Sequence number + tcpwindow size = Maximum expected sequence number. This check was fin when there were on on the fly tcp window change. Now, on very low latency network (few ms), we might experience a race condition where pf will not see the packet in the right order, therefore, pf will see packets coming in with a new tcp window size, but will not see the first modified packet on time. Therefore, it will produce a Bad State in the logs. To correct this, I had to remove in pf this check. From now on, I don't have any problem anymore. I think we should work to find a correct alternative solution for this. More and More oses adapt there Window size, startng with Windows Vista, Linux (from 2.6.18 I think), Mac OSX Leopard. I am also seeing a strange behavior while running backups. The backup will run for about a Gig, then I will have bad stated and the following error: Dec 5 08:34:24 pf01a-std /bsd: pf: BAD state: TCP 193.189.125.226:9103 193.189.125.226:9103 77.72.89.171:1900 [lo=1110166540 high=1110165037 win=65535 modulator=0] [lo=3660513330 high=3660578711 win=32767 modulator=0] 4:4 A seq=1110132270 (1110132270) ack=3660513330 len=1456 ackskew=0 pkts=127312:59301 dir=in,fwd Dec 5 08:34:24 pf01a-std /bsd: pf: State failure on: 2 | You could notice that the lo=1110166540 is higher than high=1110165037 and of course the Sequence Number is outbound: seq=1110132270 Any idea what could cause such a mess ? I am using OpenBSD 4.1, custom built kernel just to comment on check in pf. Lio
Pf Issue with a large number of Packet
Hi All,
I am sorry to bother the list but I think I may have encountered a bug and I
would like to share with you guys. I have been using OpenBSD to build Firewall
for a long time in solution with VLAN + CARP. When computers in the protected
network downloads a file in HTTP, everything works for the First 15 Mo then it
stops.
When I tcpdump, On the external address, I get the folowing:
08:34:19.343833 mirrors.club-internet.fr.www so-bo01-std.55692: P
17637121:17638569(1448) ack 174 win 49232 nop,nop,timestamp 3651037459
313698521 (DF)
08:34:19.343870 so-bo01-std.55692 mirrors.club-internet.fr.www: . ack
17634225 win 1810 nop,nop,timestamp 313698522 3651037459 (DF)
08:34:19.614303 mirrors.club-internet.fr.www so-bo01-std.55692: P
20054337:20055785(1448) ack 174 win 49232 nop,nop,timestamp 3651037487
313698589 (DF)
08:34:19.614326 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.024189 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037528
313698589 (DF)
08:34:20.024210 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.844464 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037610
313698589 (DF)
08:34:20.844485 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:22.485887 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037774
313698589 (DF)
08:34:22.485907 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:24.234738 so-bo01-std.55692 mirrors.club-internet.fr.www: F 174:174(0)
ack 20009449 win 1851 nop,nop,timestamp 313699744 3651037482 (DF)
08:34:24.235872 mirrors.club-internet.fr.www so-bo01-std.55692: . ack 175
win 49232 nop,nop,timestamp 3651037949 313699744 (DF)
On the internal interfaces, I see nothing related to the host unreachable,
just a Reset after a while from the server.
- If I pfctl -d, everything works
- If I remove all the blocks statement in the pf.conf, it do not work
- If I rate limit the download to 50 ko/s, then I still have unreachable but
it able to recover, above and up to 100Mo, it would fail and the transfer
stall.
- If I create an empty rules file, then it works
Here are the two rules:
# Production Firewall vers le Second FireWall
service_granted={domain, ntp, smtp, snmp, http}
block out log on $if_interco all label Protection vers le Back
pass in on $if_interco proto {tcp, udp} from {$net_back, $net_interco} to any
port $service_granted keep state label Back Office vers l'Internet
Please advise
Regarde
Lio
Alionis
Re: Pf Issue with a large number of Packet
I Think I have another piece of information, As the ping is very small, I
think there are too many packets going on at the same time. Therefore, the
system to check the states might not receive the packets in the right order
and therefore decide that certain packets arrived to early.
I hope it helps
Regards
Leo
Alionis
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Lio
Goehrs
Sent: jeudi 7 juin 2007 09:35
To: misc@openbsd.org
Subject: Pf Issue with a large number of Packet
Hi All,
I am sorry to bother the list but I think I may have encountered a bug and I
would like to share with you guys. I have been using OpenBSD to build
Firewall
for a long time in solution with VLAN + CARP. When computers in the protected
network downloads a file in HTTP, everything works for the First 15 Mo then
it
stops.
When I tcpdump, On the external address, I get the folowing:
08:34:19.343833 mirrors.club-internet.fr.www so-bo01-std.55692: P
17637121:17638569(1448) ack 174 win 49232 nop,nop,timestamp 3651037459
313698521 (DF)
08:34:19.343870 so-bo01-std.55692 mirrors.club-internet.fr.www: . ack
17634225 win 1810 nop,nop,timestamp 313698522 3651037459 (DF)
08:34:19.614303 mirrors.club-internet.fr.www so-bo01-std.55692: P
20054337:20055785(1448) ack 174 win 49232 nop,nop,timestamp 3651037487
313698589 (DF)
08:34:19.614326 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.024189 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037528
313698589 (DF)
08:34:20.024210 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.844464 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037610
313698589 (DF)
08:34:20.844485 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:22.485887 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037774
313698589 (DF)
08:34:22.485907 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:24.234738 so-bo01-std.55692 mirrors.club-internet.fr.www: F
174:174(0)
ack 20009449 win 1851 nop,nop,timestamp 313699744 3651037482 (DF)
08:34:24.235872 mirrors.club-internet.fr.www so-bo01-std.55692: . ack 175
win 49232 nop,nop,timestamp 3651037949 313699744 (DF)
On the internal interfaces, I see nothing related to the host unreachable,
just a Reset after a while from the server.
- If I pfctl -d, everything works
- If I remove all the blocks statement in the pf.conf, it do not work
- If I rate limit the download to 50 ko/s, then I still have unreachable but
it able to recover, above and up to 100Mo, it would fail and the transfer
stall.
- If I create an empty rules file, then it works
Here are the two rules:
# Production Firewall vers le Second FireWall
service_granted={domain, ntp, smtp, snmp, http}
block out log on $if_interco all label Protection vers le Back
pass in on $if_interco proto {tcp, udp} from {$net_back, $net_interco} to any
port $service_granted keep state label Back Office vers l'Internet
Please advise
Regarde
Lio
Alionis
Re: Pf Issue with a large number of Packet
Hi All,
Well, I confirm that there is a problem, when the packets arrives to fast
(about 25 000 pks/s), then it is likely that the packet does not arrive in the
right order and then the system checking the validity of the number of the
packet breaks and blocks legitimate traffic.
Regards
Lio Goehrs
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Lio
Goehrs
Sent: jeudi 7 juin 2007 09:35
To: misc@openbsd.org
Subject: Pf Issue with a large number of Packet
Hi All,
I am sorry to bother the list but I think I may have encountered a bug and I
would like to share with you guys. I have been using OpenBSD to build
Firewall
for a long time in solution with VLAN + CARP. When computers in the protected
network downloads a file in HTTP, everything works for the First 15 Mo then
it
stops.
When I tcpdump, On the external address, I get the folowing:
08:34:19.343833 mirrors.club-internet.fr.www so-bo01-std.55692: P
17637121:17638569(1448) ack 174 win 49232 nop,nop,timestamp 3651037459
313698521 (DF)
08:34:19.343870 so-bo01-std.55692 mirrors.club-internet.fr.www: . ack
17634225 win 1810 nop,nop,timestamp 313698522 3651037459 (DF)
08:34:19.614303 mirrors.club-internet.fr.www so-bo01-std.55692: P
20054337:20055785(1448) ack 174 win 49232 nop,nop,timestamp 3651037487
313698589 (DF)
08:34:19.614326 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.024189 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037528
313698589 (DF)
08:34:20.024210 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:20.844464 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037610
313698589 (DF)
08:34:20.844485 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:22.485887 mirrors.club-internet.fr.www so-bo01-std.55692: .
20009449:20010897(1448) ack 174 win 49232 nop,nop,timestamp 3651037774
313698589 (DF)
08:34:22.485907 so-knox01a-std mirrors.club-internet.fr: icmp: host
so-bo01-std unreachable
08:34:24.234738 so-bo01-std.55692 mirrors.club-internet.fr.www: F
174:174(0)
ack 20009449 win 1851 nop,nop,timestamp 313699744 3651037482 (DF)
08:34:24.235872 mirrors.club-internet.fr.www so-bo01-std.55692: . ack 175
win 49232 nop,nop,timestamp 3651037949 313699744 (DF)
On the internal interfaces, I see nothing related to the host unreachable,
just a Reset after a while from the server.
- If I pfctl -d, everything works
- If I remove all the blocks statement in the pf.conf, it do not work
- If I rate limit the download to 50 ko/s, then I still have unreachable but
it able to recover, above and up to 100Mo, it would fail and the transfer
stall.
- If I create an empty rules file, then it works
Here are the two rules:
# Production Firewall vers le Second FireWall
service_granted={domain, ntp, smtp, snmp, http}
block out log on $if_interco all label Protection vers le Back
pass in on $if_interco proto {tcp, udp} from {$net_back, $net_interco} to any
port $service_granted keep state label Back Office vers l'Internet
Please advise
Regarde
Lio
Alionis
Re: Pf Issue with a large number of Packet
if you can post dmesg and some relevant 'pass' rules, that might help.
Sure, So far, I have started my test and I have far less problems now but I
don't think the solution is fine. As of Version 4.1, the rule keep state flags
S/SA is by default.
All my problems went away when I used the following rules:
pass out on $if_prod proto tcp from any to so_prod_ad port {http, https} no
state flags any label Internet vers la prod AD
pass in on $if_prod proto tcp from so_prod_ad port {http, https} to any no
state flags any label Reply From AD to the Internet
If go on keep state, then, when I launch a download at 25 Mo/s, then it
downloads about 35 Mo then stops and my log get full of
Jun 7 12:15:26 so-knox01a-std /bsd: pf: BAD state: TCP 193.189.125.227:51872
193.189.125.227:51872 77.72.91.10:80 [lo=936647122 high=936652914 win=5840
modulator=0] [lo=2657626173 high=2657632013 win=5792 modulator=0] 4:2 SA
seq=2660626928 (2660626928) ack=936647122 len=0 ackskew=0 pkts=3:1 dir=in,rev
Jun 7 12:15:26 so-knox01a-std /bsd: pf: State failure on: 1 | 5
Jun 7 12:15:26 so-knox01a-std /bsd: pf: BAD state: TCP 193.189.125.227:51876
193.189.125.227:51876 77.72.91.10:80 [lo=941137405 high=941143197 win=5840
modulator=0] [lo=2659274591 high=2659280431 win=5792 modulator=0] 4:2 SA
seq=2662275452 (2662275452) ack=941137405 len=0 ackskew=0 pkts=3:1 dir=in,rev
Jun 7 12:15:26 so-knox01a-std /bsd: pf: State failure on: 1 | 5
Jun 7 12:15:26 so-knox01a-std /bsd: pf: BAD state: TCP 193.189.125.227:51880
193.189.125.227:51880 77.72.91.10:80 [lo=941037484 high=941043276 win=5840
modulator=0] [lo=2663170100 high=2663175940 win=5792 modulator=0] 4:2 SA
seq=2666170841 (2666170841) ack=941037484 len=0 ackskew=0 pkts=3:1 dir=in,rev
Jun 7 12:15:26 so-knox01a-std /bsd: pf: State failure on: 1 | 5
From my understanding, State failure on: 1 means the sequence number was too
much ahead, based on the RFC. But, Today, with adaptive TCP Windows, we can
have so many packets going thru at the same time.
Leo
Re: aliases on CARP interface
Then the redundant IP should be on the carp interface Leo -Message d'origine- De : Per olof Ljungmark [mailto:[EMAIL PROTECTED] Envoyi : jeudi 13 octobre 2005 10:40 @ : Lio Goehrs Cc : misc@openbsd.org Objet : Re: aliases on CARP interface Are you going to use carp on the external (public) interface ? In order to use failover, yes. We are moving from single to dual 3.7 FW's with CARP. The external interface has a lot of binated aliases and I am unsure if they are to go into hostname.carpN or stay as they are in hostname.ext_if.
Re: Install Berkeley DB both v3 and v4 from ports problem on Openbsd 3.7
Are you using a current version of OpenBSD? Lio -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Pavel M. Ivanchev Envoyi : jeudi 13 octobre 2005 13:22 @ : misc@openbsd.org Objet : Install Berkeley DB both v3 and v4 from ports problem on Openbsd 3.7 Hi there! I am new and hope that somebody will help me! Until now I didn't have any problem with installing ports but now i am in trouble with installing Berkeley DB both v3 (db-3.1.17p3) and v4 (db-4.2.52p3). I follwed these steps: 1. cvsup of the ports tree 2. cd /usr/ports/databases/db make install and the result is: Building package for db-3.1.17p3 Unknown element: @pkgpath databases/db/v3,no_tcl === Cleaning for db-3.1.17p3 rm -f /usr/ports/packages/i386/all/db-3.1.17p3.tgz *** Error code 1 Stop in /usr/ports/databases/db/v3 (line 2016 of /usr/ports/infrastructure/mk/bsd.port.mk). *** Error code 1 Stop in /usr/ports/databases/db/v3 (line 1252 of /usr/ports/infrastructure/mk/bsd.port.mk). *** Error code 1 Stop in /usr/ports/databases/db (line 109 of /usr/ports/infrastructure/mk/bsd.port.subdir.mk).
Re: zebra/ospf zero lentgh MTU's
Wust fetch the full sources, then go to /usr/src/usr.sbin/ospfd and /usr/src/usr.sbin/ospfctl and make make install :) Leo -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de stan Envoyi : jeudi 13 octobre 2005 16:21 @ : OpenBSD general usage list Cc : Claudio Jeker; Stewart Flood Objet : Re: zebra/ospf zero lentgh MTU's On Thu, Oct 13, 2005 at 03:51:05PM +0200, Claudio Jeker wrote: On Thu, Oct 13, 2005 at 09:16:29AM -0400, stan wrote: I'm trying to set up ospf using the zebra port on a 3.7 machine. It appears that (at least one) of my problems is that the router I need to peer with is sending a MTU of 0. I'm getting this error message: recv_dd_description: invalid MTU, neighbor ID 170.85.115.1 This is fixed in 3.8 and -current and it is an /usr/sbin/ospfd specific error and not one from the zebra port. Thanks, I was confused because I looked on an older machine to see if there was an osppf daemon provided with OpenBSD, decided there was not, and installed zebra. Yhen I found the OpenBSD one :-( from ospfd. I did a bit of Googling, and found some mention of this as a problem with some peices of CISCO gear. This happens to not be a Cisco router, but I think it shares this issue with Cisco. I've poked around in the source code for the ospfd that comes with zebra, but I can't seem to find where this check is made. Your looking at the wrong source. You are running OpenOSPFD and not zerba. Thanks, again. I see that now. Any sugestiosn as to how to work around this? Try to run the correct binary. If you like to give OpenOSPFD a try you should use -current ospfd/ospfctl. Some major bugs got fixed in the last few days. Is thee a way to get the latest OpenBSD ospfd source, without having to upgrade the whole machine to current? And if so, would that be a sane thing to do? -- U.S. Encouraged by Vietnam Vote - Officials Cite 83% Turnout Despite Vietcong Terror - New York Times 9/3/1967
RE : nfs mounting
Are you using the root account to try create the file ? If so, this is your problem and you have to change a few settings on the file server. Leo De: [EMAIL PROTECTED] de la part de Chuck Robey Date: sam. 10/8/2005 11:27 @: misc@openBSD.org Objet : nfs mounting I have just ogtten usb networking up on my Zaurus, and now I'm tryingto get /usr/local, /usr/ports, and /usr/src remotely mounted from my nearby FreeBSD system. I can get the mount done, but I can't affect any files ... for example, if I tryi to touch (as root on the Zaurus) /usr/local/garbage, I get Permission denied. i don't have any cklear idea if this is a problem on the Zaurus (OpenBSD) or The FreeBSD server. When I did the mount, I used -v, and the listing I got is: april.chuckr.org:/usr3/osrc/ports on /usr/ports type nfs (rw, ctime=Sat Oct 8 10:23:49 2005, v3, tcp, hard, wsize=8192, rsize=8192, rdirsize=8192, timeo=100, retrans=10, maxgrouplist=16, readahead=1, acregmin=5, acregmax=60, acdirmin=5, acdirmax=60) Is there anything else you might want to know? This is extremely frustrating, being so terribly close to having it work, but not being there.
Re: Two Isp Fault Tollerance Help
Absolutely, you need an AS The address space can be given by one of the provider. Lio -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Olivier Mehani Envoyi : vendredi 7 octobre 2005 15:34 @ : misc@openbsd.org Objet : Re: Two Isp Fault Tollerance Help On Fri, 7 Oct 2005 14:29:08 +0200 Johan M:son Lindman [EMAIL PROTECTED] wrote: One of my clients has got an Internet connection with a no much affidable provider. He reports continual disconnection and so on. I would like to do a second connection with another provider to obtain a sort of redundancy, a fault tollerance. What I have to do to obtain the automatic connection with both of the providers and to shift to the one that is connected when the other is in trouble? ( without problems for the client). Border Gateway Protocol. Doesn't it imply that said client has its own IP addresses range and not NATing behind one single ISP-provided address ? -- Olivier Mehani [EMAIL PROTECTED] PGP fingerprint: 3720 A1F7 1367 9FA3 C654 6DFB 6845 4071 E346 2FD1
Re: CARP+Pfsync+Bind
Then, you can forget about DNSSEC for example ... Lio -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de ed Envoyi : vendredi 7 octobre 2005 19:25 Cc : misc@openbsd.org Objet : Re: CARP+Pfsync+Bind On Thu, 6 Oct 2005 19:52:31 -0400 Dave Anderson [EMAIL PROTECTED] wrote: Responses long enough so that required information is truncated should be rare, so perhaps you've been lucky and not encountered any yet. I understand fully what you are saying, but I just don't want to serve DNS via TCP. I'm as sure as I can be that no replies exceed 512 bytes. If it ever becomes a problem I'll use tcpserver to provide it, but it's been fine for a long time, and it's safe, at least in my case, to assume TCP is for zone transfers, YMMV. -- Regards, Ed http://www.usenix.org.uk
About VLAN and Carp
Hi Everyone, I am using OpenBSD and the great pf in a production environment. I want to be able to use vlan and carp at the same time. I have two firewalls. These two boxes are responsible for a number of subnet. I want to have a number of vlan defined on the openbsd to feed my Distribution Switch. Now I can do it, but only on the physical interface so I loose the redundancy. On a cisco, it would mean having a few VLAN with a router-interface for each. Each virtual interface would have VRRP enabled. When I try ifconfig vlan0 vlan 11 vlandev carp0 It gives me an error. Is there a way to do that? Regards Leo Goehrs CTO Work: +33 1 39 02 76 15 Mobile: +33 6 89 99 14 06 Fax: +33 1 39 02 01 51 Email: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] IM: 10257254 (ICQ) Alionis http://www.alionis.net 15 rue de la Paroisse http://maps.google.com/maps?q=15+rue+de+la+Paroisse%2CVersailles+78000%2CFra ncehl=en Versailles 78000 France [demime 1.01d removed an attachment of type image/jpeg which had a name of image001.jpg]
