Re: PF, bridge, states and window scaling problem
On Tue, Nov 13, 2007 at 03:53:38PM +0200, Alupului Costin wrote: > On Nov 13, 2007 4:20 AM, Girish Venkatachalam > <[EMAIL PROTECTED]> wrote: > > On 22:08:03 Nov 12, Alupului Costin wrote: > > > > > > pass in quick on vlan0 from any to anIP/32 > > > pass out quick on vlan0 from anIP/32 to any keep state queue ul_client > > > pass in quick on vlan1 from anIP/32 to any > > > pass out quick on vlan1 from any to anIP/32 keep state queue dl_client > > > > > > The above rules generate state-mismatches. > > > > Didn't get you. What sort of mismatch? > > When that client tries logging in to Yahoo Messenger I can see an > increase in the number of state-mismatch reported by pfctl -si. There > are states established, but after a while the packets simply do not > match the states created. Also they will not create new states and nor > will they match a catch-all rule which follows. I wonder why it's not creating new states. Could you be running out of kernel memory? Are they actual syn packets? > I will answer here to Erik Osterholm also: > > Performance really is an issue here when I give up statefull > inspection. The firewall contains roughly 2000 filter rules and the > traffic passing through is 20kpps at peak hours. So it is a huge > difference between statefull and stateless filtering. If I drop the > stafefull filtering the machine simply cannot handle all the traffic, > or in the best case scenario it develops quite some latency. I didn't mean to imply that performance wasn't an issue on your part, just mentioning it on ours. I know that keeping state is probably ideal in general, but depending upon your ruleset, it may be possible to optimize it so that keeping state isn't required for performance. For example, if you have many rules which are identical except for the host, you can use a table to keep track of the hosts and then only a few rules. This can speed things up dramatically. (Sorry if I'm telling you things that you already know--I'm not familiar with your level of expertise.) Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On Tue, Nov 13, 2007 at 07:25:23PM +0530, Girish Venkatachalam wrote: > On 18:57:34 Nov 13, Girish Venkatachalam wrote: > > I just read the post you linked. Thanks. :) > > I read the post once again and it looks as though I understood what is > mentioned there. > > The 'no-df' in scrub rule clears the Don't fragment bit in the IP > header. When a host wrongly sends fragmented packets with the DF bit > set, this scrub rule "correctly" resets the DF bit. > > Now since the host made the mistake of sending a fragmented packet with > DF bit set ( this is like saying " Please don't fragment my packet, but > I myself have fragmented". Odd...) no-df scrub rule causes trouble. > > Scrub never causes trouble with properly formed packets. > > regards, > Girish Ah, that makes sense! In fact, if I'd done a little more reading, I'd see that OpenBSD suggests the same: http://www.openbsd.org/faq/pf/scrub.html They mention that there are some problems (NFS specifically, and "some online games"). I believe that we've also seen some weird behavior with Active Directory, but I'd have to check to make sure. Thanks for the information! Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On 15:53:38 Nov 13, Alupului Costin wrote: > When that client tries logging in to Yahoo Messenger I can see an > increase in the number of state-mismatch reported by pfctl -si. There > are states established, but after a while the packets simply do not > match the states created. Also they will not create new states and nor > will they match a catch-all rule which follows. > Have you tried bumping up the state expiry timeout values? > I have tried using "flags S/SA" with the filter rules. The result was > that states were created, but not matched by the rest of the packets > in the stream. Packets would just match a catch-all rule that follows > the above mentioned rules. Still it was better because the connection > wouldn't just stall, but after all that was not statefull inspection > anymore... States are established and looked up based on unique 5 tuples or whatever. I don't expect a bug here. I think the packets that do not match the existing state entries have different keys into the state lookup table. IOW they don't form part of the same stream. > I have tried the same setup (without the queues) on a router and I > used keep state on all the rules (even the inbound ones). Works > perfectly. So I guess the problem really is the bridge. In that case I > would kindly suggest that the pf.conf manual page should mention that > statefull firewall has an unpredictable behaviour on bridges. I.E. you > can not create states on inbound rules at all although filtering > works. Another problem is that states created by outbound traffic > don't seem to take into account the window scaling when the client > uses that. > I was a big fan of the bridge setup simply because it is transparent > and I would choose the bridge over the router setup anytime, provided > that it would work properly (i mean statefull firewall). > But bridging is more complicated to manage and this problem seems to point to that. Also did you read the other post? There is some info about bridging caused state mismatches. > I always flushed the old states over and over again. The flags did not > help me. As I mentioned earlier they did establish the connection on > the SYN packet, but the rest of the packets in the flow did not match > that connection. In that case I am pretty much exhausted. I can't think of any other possibility. > Have tried without normalization, without fragment reassembly, with > no-df... Pretty much all the combinations... > > I will answer here to Erik Osterholm also: > > Performance really is an issue here when I give up statefull > inspection. The firewall contains roughly 2000 filter rules and the > traffic passing through is 20kpps at peak hours. So it is a huge > difference between statefull and stateless filtering. If I drop the > stafefull filtering the machine simply cannot handle all the traffic, > or in the best case scenario it develops quite some latency. > Indeed. Stateful firewalling improves performance by a huge magnitude due to the shortcuts that packets take instead of having to descend down the pf ruleset. regards, Girish ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On Nov 13, 2007 2:30 PM, J65nko <[EMAIL PROTECTED]> wrote: > > On Nov 12, 2007 9:08 PM, Alupului Costin <[EMAIL PROTECTED]> wrote: > > Hello all, > > > > I seem to have quite a problem with PF. I have set up a bridge to > > shape my upstream traffic. I use ALTQ with hfsc discipline; but that's > > not really important. My problem comes with the filter rules. I have > > to use keep state because of the speed benefits (really I don't have a > > choice), but PF has a problem when the clients passing traffic through > > the bridge use TCP window scaling. Here is an example of four filter > > rules that I thought should work to pass the traffic from one client > > through the bridge and create a state: > > > > pass in quick on vlan0 from any to anIP/32 > > pass out quick on vlan0 from anIP/32 to any keep state queue ul_client > > pass in quick on vlan1 from anIP/32 to any > > pass out quick on vlan1 from any to anIP/32 keep state queue dl_client > > > > The above rules generate state-mismatches. I thought that would be > > because pf doesn't see the SYN packet, although it does (one of the > > out rules) and should create the state then... I tried writing all the > > rules with keep state (even the inbound ones) but then nothing would > > work at all. My intention was to create if-bound states, but I > > switched back to floating states in the hope that pf would associate > > the state created by an outbound rule with the traffic returning on > > another interface of the bridge; still didn't work. > > > > I have read the man page for if_bridge and set the following sysctl > > variables: > > > > net.link.bridge.pfil_onlyip: 1 > > net.link.bridge.pfil_bridge: 0 > > net.link.bridge.pfil_member: 1 > > > > I have also read some posts on the web that said that pf simply > > doesn't have all the hooks necesary to do the filtering inbound and > > outbound, but reading the pfil man page I seem to disaggree with that. > > > > Has anyone encountered the same problem? And, more important: if i > > give up the bridge setup and switch to routing, would that have any > > effect? I.E: will I then be able to use keep state with the inbound > > rules? > > > > Any help at all would be hugely appreciated as I am trying for about a > > week to sort out this problem and can't seem to get any closer. The > > only solution was to kindly ask my clients using TCP window scaling > > (Vista mostly) to turn off this feature... Now I am seriously > > considering bumping my bridge to a router but I am not sure that the > > problem will be solved then. > > > > Oh, here is the setup of the bridge from rc.conf, although there > > shouldn't be any problems there (the bridge works fine without pf, or > > with pf stateless): > > > > # > > # Core: em2 -> vlan1 > > # Border: em1 -> vlan0 > > # Bridge0 vlan0 -><- vlan1 > > # > > cloned_interfaces="bridge0 vlan0 vlan1" > > ifconfig_em0="up" > > ifconfig_em1="up" > > ifconfig_em2="up" > > ifconfig_vlan0="vlan 132 vlandev em1 up" > > ifconfig_vlan1="vlan 132 vlandev em2 up" > > ifconfig_bridge0="addm vlan0 addm vlan1 up" > > # Admin iface > > ifconfig_em0="inet adminIP netmask 255.255.255.0" > > > > See "Create TCP states on the initial SYN packet" from > http://undeadly.org/cgi?action=article&sid=20060928081238 > > That paragraph explains nicely the necessity of pf to create state on > the first packet of the 3-way TCP handshake to prevent TCP window > scaling issues. I aggree with you. My problem is why doesn't pf establish the connection correctly with the first outbound rule (the SYN packet passes that rule). Furthermore: why everything stalls if I use keep state on the inbound rules also? Because that would make the most sense: using keep state with every rule... In a routing environment it all works fine, but not with the bridge. So I guess that the problem could be the bridge, although everything else works fine besides "keep state" on inbound rules... Costin > > =Adriaan= > > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On 18:57:34 Nov 13, Girish Venkatachalam wrote: > I just read the post you linked. Thanks. :) I read the post once again and it looks as though I understood what is mentioned there. The 'no-df' in scrub rule clears the Don't fragment bit in the IP header. When a host wrongly sends fragmented packets with the DF bit set, this scrub rule "correctly" resets the DF bit. Now since the host made the mistake of sending a fragmented packet with DF bit set ( this is like saying " Please don't fragment my packet, but I myself have fragmented". Odd...) no-df scrub rule causes trouble. Scrub never causes trouble with properly formed packets. regards, Girish ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On Nov 13, 2007 4:20 AM, Girish Venkatachalam <[EMAIL PROTECTED]> wrote: > On 22:08:03 Nov 12, Alupului Costin wrote: > > I seem to have quite a problem with PF. I have set up a bridge to > > shape my upstream traffic. I use ALTQ with hfsc discipline; but that's > > not really important. My problem comes with the filter rules. I have > > to use keep state because of the speed benefits (really I don't have a > > choice), > > One should always keep state. > > > but PF has a problem when the clients passing traffic through > > the bridge use TCP window scaling. Here is an example of four filter > > rules that I thought should work to pass the traffic from one client > > through the bridge and create a state: > > > > pass in quick on vlan0 from any to anIP/32 > > pass out quick on vlan0 from anIP/32 to any keep state queue ul_client > > pass in quick on vlan1 from anIP/32 to any > > pass out quick on vlan1 from any to anIP/32 keep state queue dl_client > > > > The above rules generate state-mismatches. > > Didn't get you. What sort of mismatch? When that client tries logging in to Yahoo Messenger I can see an increase in the number of state-mismatch reported by pfctl -si. There are states established, but after a while the packets simply do not match the states created. Also they will not create new states and nor will they match a catch-all rule which follows. > > > I thought that would be > > because pf doesn't see the SYN packet, although it does (one of the > > out rules) and should create the state then... I tried writing all the > > rules with keep state (even the inbound ones) but then nothing would > > work at all. My intention was to create if-bound states, but I > > switched back to floating states in the hope that pf would associate > > the state created by an outbound rule with the traffic returning on > > another interface of the bridge; still didn't work. > > > > Have you tried adding "flags S/SAFR" to the filter rules? > > Try it and let me know. I have tried using "flags S/SA" with the filter rules. The result was that states were created, but not matched by the rest of the packets in the stream. Packets would just match a catch-all rule that follows the above mentioned rules. Still it was better because the connection wouldn't just stall, but after all that was not statefull inspection anymore... > > > I have read the man page for if_bridge and set the following sysctl > > variables: > > > > net.link.bridge.pfil_onlyip: 1 > > net.link.bridge.pfil_bridge: 0 > > net.link.bridge.pfil_member: 1 > > > > I have also read some posts on the web that said that pf simply > > doesn't have all the hooks necesary to do the filtering inbound and > > outbound, but reading the pfil man page I seem to disaggree with that. > > > > What do you mean? ? I mean that according to the pfil and if_bridge man pages pf should have all the hooks necesary to filter the packet inbound and outbound on both (/all) members of the bridge, provided that I have set up the sysctl variables as mentioned above. > > > Has anyone encountered the same problem? And, more important: if i > > give up the bridge setup and switch to routing, would that have any > > effect? I.E: will I then be able to use keep state with the inbound > > rules? > > Try it. Routing changes the topology a good deal. But I doubt if that is > the issue here. No harm in testing though. I have tried the same setup (without the queues) on a router and I used keep state on all the rules (even the inbound ones). Works perfectly. So I guess the problem really is the bridge. In that case I would kindly suggest that the pf.conf manual page should mention that statefull firewall has an unpredictable behaviour on bridges. I.E. you can not create states on inbound rules at all although filtering works. Another problem is that states created by outbound traffic don't seem to take into account the window scaling when the client uses that. I was a big fan of the bridge setup simply because it is transparent and I would choose the bridge over the router setup anytime, provided that it would work properly (i mean statefull firewall). > > > > > Any help at all would be hugely appreciated as I am trying for about a > > week to sort out this problem and can't seem to get any closer. The > > only solution was to kindly ask my clients using TCP window scaling > > (Vista mostly) to turn off this feature... Now I am seriously > > considering bumping my bridge to a router but I am not sure that the > > problem will be solved then. > > Try adding the flags switch as mentioned above. That way the states get > established only from a TCP Syn packet. > > You should also try flushing the old states using pfctl(8). I always flushed the old states over and over again. The flags did not help me. As I mentioned earlier they did establish the connection on the SYN packet, but the rest of the packets in the flow did not match that connection. > > > > > Oh, here is the setup of the brid
Re: PF, bridge, states and window scaling problem
On 23:42:20 Nov 12, Erik Osterholm wrote: > My understanding (and please correct me if I'm wrong) is that > keeping state requires fragmented packet reassembly, which can break > some applications. You mean that you cannot support "broken applications" if you do reassembly? Packet reassembly happens if you use a scrub rule as well. The main problem with fragmentation leaving aside all performance and security considerations is that you cannot figure out anything useful from the IP fragments. The headers simply lack enough information for you to deduce anything. Reassembly does have an overhead..you can perhaps mention a delay involved in waiting for all fragments to arrive. But AFAIK it only helps if you reassemble. Never hurts. I am not aware of any breakage due to reassembly. ( But I could be ignorant). Now I specifically asked about scrub because scrub does a lot of other things which might "correctly" break "broken applications." I just wanted to give him enough rope. Very likely scrub causes no harm. Neither would keeping state... > Also, I've always followed the conventional wisdom > that bridges shouldn't keep state. A posting from the maintainer > supports this: > http://lists.freebsd.org/pipermail/freebsd-pf/2005-September/001481.html > > Maybe this has changed--I'm not sure, but so far I haven't seen > performance issues with pf and if_bridge without keeping state, so I > haven't been worried about it. I just read the post you linked. Thanks. :) I would imagine that bridges would make things difficult for pf. I have never worked with bridges , so I cannot comment. Sorry. regards, Girish ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On Nov 12, 2007 9:08 PM, Alupului Costin <[EMAIL PROTECTED]> wrote: > Hello all, > > I seem to have quite a problem with PF. I have set up a bridge to > shape my upstream traffic. I use ALTQ with hfsc discipline; but that's > not really important. My problem comes with the filter rules. I have > to use keep state because of the speed benefits (really I don't have a > choice), but PF has a problem when the clients passing traffic through > the bridge use TCP window scaling. Here is an example of four filter > rules that I thought should work to pass the traffic from one client > through the bridge and create a state: > > pass in quick on vlan0 from any to anIP/32 > pass out quick on vlan0 from anIP/32 to any keep state queue ul_client > pass in quick on vlan1 from anIP/32 to any > pass out quick on vlan1 from any to anIP/32 keep state queue dl_client > > The above rules generate state-mismatches. I thought that would be > because pf doesn't see the SYN packet, although it does (one of the > out rules) and should create the state then... I tried writing all the > rules with keep state (even the inbound ones) but then nothing would > work at all. My intention was to create if-bound states, but I > switched back to floating states in the hope that pf would associate > the state created by an outbound rule with the traffic returning on > another interface of the bridge; still didn't work. > > I have read the man page for if_bridge and set the following sysctl variables: > > net.link.bridge.pfil_onlyip: 1 > net.link.bridge.pfil_bridge: 0 > net.link.bridge.pfil_member: 1 > > I have also read some posts on the web that said that pf simply > doesn't have all the hooks necesary to do the filtering inbound and > outbound, but reading the pfil man page I seem to disaggree with that. > > Has anyone encountered the same problem? And, more important: if i > give up the bridge setup and switch to routing, would that have any > effect? I.E: will I then be able to use keep state with the inbound > rules? > > Any help at all would be hugely appreciated as I am trying for about a > week to sort out this problem and can't seem to get any closer. The > only solution was to kindly ask my clients using TCP window scaling > (Vista mostly) to turn off this feature... Now I am seriously > considering bumping my bridge to a router but I am not sure that the > problem will be solved then. > > Oh, here is the setup of the bridge from rc.conf, although there > shouldn't be any problems there (the bridge works fine without pf, or > with pf stateless): > > # > # Core: em2 -> vlan1 > # Border: em1 -> vlan0 > # Bridge0 vlan0 -><- vlan1 > # > cloned_interfaces="bridge0 vlan0 vlan1" > ifconfig_em0="up" > ifconfig_em1="up" > ifconfig_em2="up" > ifconfig_vlan0="vlan 132 vlandev em1 up" > ifconfig_vlan1="vlan 132 vlandev em2 up" > ifconfig_bridge0="addm vlan0 addm vlan1 up" > # Admin iface > ifconfig_em0="inet adminIP netmask 255.255.255.0" > See "Create TCP states on the initial SYN packet" from http://undeadly.org/cgi?action=article&sid=20060928081238 That paragraph explains nicely the necessity of pf to create state on the first packet of the 3-way TCP handshake to prevent TCP window scaling issues. =Adriaan= ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On Tue, Nov 13, 2007 at 07:50:53AM +0530, Girish Venkatachalam wrote: > On 22:08:03 Nov 12, Alupului Costin wrote: > > I seem to have quite a problem with PF. I have set up a bridge to > > shape my upstream traffic. I use ALTQ with hfsc discipline; but that's > > not really important. My problem comes with the filter rules. I have > > to use keep state because of the speed benefits (really I don't have a > > choice), > > One should always keep state. <...> > > Oh, here is the setup of the bridge from rc.conf, although there > > shouldn't be any problems there (the bridge works fine without pf, or > > with pf stateless): > > Stateful filtering is always recommended. Performance is not the only > reason why you should use it. > > It also adds to security. Have you tried disabling normalization/scrub? > > Best, > Girish My understanding (and please correct me if I'm wrong) is that keeping state requires fragmented packet reassembly, which can break some applications. Also, I've always followed the conventional wisdom that bridges shouldn't keep state. A posting from the maintainer supports this: http://lists.freebsd.org/pipermail/freebsd-pf/2005-September/001481.html Maybe this has changed--I'm not sure, but so far I haven't seen performance issues with pf and if_bridge without keeping state, so I haven't been worried about it. Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: PF, bridge, states and window scaling problem
On 22:08:03 Nov 12, Alupului Costin wrote: > I seem to have quite a problem with PF. I have set up a bridge to > shape my upstream traffic. I use ALTQ with hfsc discipline; but that's > not really important. My problem comes with the filter rules. I have > to use keep state because of the speed benefits (really I don't have a > choice), One should always keep state. > but PF has a problem when the clients passing traffic through > the bridge use TCP window scaling. Here is an example of four filter > rules that I thought should work to pass the traffic from one client > through the bridge and create a state: > > pass in quick on vlan0 from any to anIP/32 > pass out quick on vlan0 from anIP/32 to any keep state queue ul_client > pass in quick on vlan1 from anIP/32 to any > pass out quick on vlan1 from any to anIP/32 keep state queue dl_client > > The above rules generate state-mismatches. Didn't get you. What sort of mismatch? > I thought that would be > because pf doesn't see the SYN packet, although it does (one of the > out rules) and should create the state then... I tried writing all the > rules with keep state (even the inbound ones) but then nothing would > work at all. My intention was to create if-bound states, but I > switched back to floating states in the hope that pf would associate > the state created by an outbound rule with the traffic returning on > another interface of the bridge; still didn't work. > Have you tried adding "flags S/SAFR" to the filter rules? Try it and let me know. > I have read the man page for if_bridge and set the following sysctl variables: > > net.link.bridge.pfil_onlyip: 1 > net.link.bridge.pfil_bridge: 0 > net.link.bridge.pfil_member: 1 > > I have also read some posts on the web that said that pf simply > doesn't have all the hooks necesary to do the filtering inbound and > outbound, but reading the pfil man page I seem to disaggree with that. > What do you mean? ? > Has anyone encountered the same problem? And, more important: if i > give up the bridge setup and switch to routing, would that have any > effect? I.E: will I then be able to use keep state with the inbound > rules? Try it. Routing changes the topology a good deal. But I doubt if that is the issue here. No harm in testing though. > > Any help at all would be hugely appreciated as I am trying for about a > week to sort out this problem and can't seem to get any closer. The > only solution was to kindly ask my clients using TCP window scaling > (Vista mostly) to turn off this feature... Now I am seriously > considering bumping my bridge to a router but I am not sure that the > problem will be solved then. Try adding the flags switch as mentioned above. That way the states get established only from a TCP Syn packet. You should also try flushing the old states using pfctl(8). > > Oh, here is the setup of the bridge from rc.conf, although there > shouldn't be any problems there (the bridge works fine without pf, or > with pf stateless): Stateful filtering is always recommended. Performance is not the only reason why you should use it. It also adds to security. Have you tried disabling normalization/scrub? Best, Girish ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
