I've got the issue solved by disabling states on all rules which deal with the tproxy.
On 4 June 2013 11:28, Raimundo Santos <[email protected]> wrote: > I am guessing that the problem lies with flags S/SA. > > Changing all rules to flags any, and the packets hits the rules, but > things go worse: no web navigation... this is driving me mad! > > > > On 3 June 2013 13:09, Raimundo Santos <[email protected]> wrote: > >> Hi there! >> >> I asked, without an answer, something about nat-to and real IPs. Well, I >> really need an answer there, so if someone get a clue, I will be glad tho >> hear :) >> >> Now, to the new issue! >> >> Here in our WiFi ISP we are have contracted a tproxy service from FreeBSD >> Brasil. It is somehow working, but I can not figure out exactly how. Here >> is a diagram of the desired paths: >> >> http://devio.us/~raitech/Obsd53PfTproxy.png >> >> These are my rules by now: >> >> RFC1918 = "{ 172.16/12, 192.168/16, 10/8, 127/8 }" >> table <INT_NET> persist { internal nets, all valid IPs } >> >> ext_if_1 = "em0" >> ext_gw_1 = "187.72.X.X" >> ext_ip_1 = "187.72.X.X" >> >> ext_if_2 = "em1" >> ext_gw_2 = "187.72.X.X" >> ext_ip_2 = "187.72.X.X" >> >> ext_if_3 = "alc0" >> ext_gw_3 = "187.72.X.X" >> ext_ip_3 = "187.72.X.X" >> >> int_if_1 = "em2" >> int_gw_1 = "187.72.X.X" >> int_ip_1 = "187.72.X.X" >> >> squid_master_if = "em3" >> squid_master_gw = "187.72.X.X" >> squid_master_ip = "187.72.X.X" >> >> set limit states 6304000 >> set limit tables 5000 >> set limit src-nodes 200000 >> set limit frags 3000 >> set optimization aggressive >> set state-defaults pflow, no-sync >> >> set skip on lo >> >> block in log quick on { \ >> $ext_if_1, \ >> $ext_if_2, \ >> $ext_if_3, \ >> $squid_master_if, \ >> $int_if_1 } from $RFC1918 label "blocking RFC1918" >> >> # trying to prioritizing ACKs... >> match set prio (3,5) >> # ... and all traffic http. https over the others >> match proto tcp to port { http, https } set prio (5,6) >> match proto tcp from port { http, https } set prio (5,6) >> >> match proto tcp to port { ssh, 9876 } set prio(5,7) >> >> pass in on $int_if_1 proto tcp from { <INT_NET>, $int_gw_1 } to port http >> \ >> route-to ($squid_master_if $squid_master_gw) >> >> pass in on { $ext_if_1, $ext_if_2, $ext_if_3 } proto tcp from port http \ >> to { <INT_NET>, $int_gw_1 } \ >> route-to ($squid_master_if $squid_master_gw) >> >> pass in on $squid_master_if proto tcp from { <INT_NET>, $int_gw_1 } to \ >> port http no state route-to \ >> { \ >> ($ext_if_1 $ext_gw_1) , \ >> ($ext_if_2 $ext_gw_2) \ >> } least-states label "cahce external outbound balancing" >> >> pass in on $squid_master_if proto tcp from port http \ >> to { <INT_NET>, $int_gw_1 } route-to ($int_if_1 $int_gw_1) \ >> label "cahce internal outbound routing" >> >> An here are a pfctl -vsr output: >> >> block drop in log quick on em0 inet from 172.16.0.0/12 to any label >> "blocking RFC1918" >> [ Evaluations: 61764339 Packets: 332 Bytes: 32854 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em0 inet from 192.168.0.0/16 to any label >> "blocking RFC1918" >> [ Evaluations: 5883927 Packets: 114 Bytes: 28621 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em0 inet from 10.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 5883813 Packets: 170 Bytes: 18354 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em0 inet from 127.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 5883643 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em1 inet from 172.16.0.0/12 to any label >> "blocking RFC1918" >> [ Evaluations: 60684174 Packets: 305 Bytes: 30912 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em1 inet from 192.168.0.0/16 to any label >> "blocking RFC1918" >> [ Evaluations: 6862827 Packets: 93 Bytes: 9232 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em1 inet from 10.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 6862734 Packets: 196 Bytes: 19396 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em1 inet from 127.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 6862538 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on alc0 inet from 172.16.0.0/12 to any label >> "blocking RFC1918" >> [ Evaluations: 50726925 Packets: 304 Bytes: 30856 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on alc0 inet from 192.168.0.0/16 to any label >> "blocking RFC1918" >> [ Evaluations: 1251 Packets: 79 Bytes: 8268 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on alc0 inet from 10.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 1172 Packets: 152 Bytes: 16948 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on alc0 inet from 127.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 1020 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em3 inet from 172.16.0.0/12 to any label >> "blocking RFC1918" >> [ Evaluations: 50726392 Packets: 304 Bytes: 30856 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em3 inet from 192.168.0.0/16 to any label >> "blocking RFC1918" >> [ Evaluations: 13589809 Packets: 76 Bytes: 8132 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em3 inet from 10.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 13589733 Packets: 152 Bytes: 16948 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em3 inet from 127.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 13589581 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em2 inet from 172.16.0.0/12 to any label >> "blocking RFC1918" >> [ Evaluations: 39571927 Packets: 10414 Bytes: 478685 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em2 inet from 192.168.0.0/16 to any label >> "blocking RFC1918" >> [ Evaluations: 6364466 Packets: 1779 Bytes: 142401 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em2 inet from 10.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 6362687 Packets: 32496 Bytes: 1375238 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> block drop in log quick on em2 inet from 127.0.0.0/8 to any label >> "blocking RFC1918" >> [ Evaluations: 6330191 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match all set ( prio(3, 5) ) >> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: >> 3831 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any to any port = 80 set ( prio(5, 6) ) >> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: >> 3831 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any to any port = 443 set ( prio(5, 6) ) >> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any port = 80 to any set ( prio(5, 6) ) >> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any port = 443 to any set ( prio(5, 6) ) >> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any to any port = 22 set ( prio(5, 7) ) >> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> match proto tcp from any to any port = 9876 set ( prio(5, 7) ) >> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass all no state allow-opts >> [ Evaluations: 61717379 Packets: 61549113 Bytes: 41451833770 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 61717379 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em1 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 55197296 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on alc0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 38378103 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 48038032 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em1 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 44966361 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on alc0 inet proto tcp from any port = 80 to 187.72.X.X flags >> S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 41608198 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em2 inet proto tcp from <INT_NET> to any port = 80 flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 48044445 Packets: 1439990 Bytes: 894473590 States: >> 435 ] >> [ Inserted: uid 0 pid 19584 State Creations: 40060 ] >> pass in on em2 inet proto tcp from 187.72.X.X to any port = 80 flags S/SA >> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >> [ Evaluations: 3694317 Packets: 12437474 Bytes: 9381159120 States: >> 3396 ] >> [ Inserted: uid 0 pid 19584 State Creations: 128206] >> pass in on em3 inet proto tcp from <INT_NET> to any port = 80 no state >> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_0> >> least-states >> [ Evaluations: 38420511 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em3 inet proto tcp from 187.72.X.X to any port = 80 no state >> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_1> >> least-states >> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em3 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >> keep state (no-sync, pflow) label "cahce internal outbound routing" >> route-to 187.72.X.X@em2 >> [ Evaluations: 13731058 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> pass in on em3 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >> keep state (no-sync, pflow) label "cahce internal outbound routing" >> route-to 187.72.X.X@em2 >> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: >> 0 ] >> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >> >> This is the same behavior with or without multipath routing. What >> bahavior? Well, only rules for in on em3 that are destineted to internal >> network are working, the others barelly catches a few thousands of packets. >> Very strange... >> >> But, as said before: more strange is the fact that the cache solution is >> almost working, just some delays to load a page here, youtube gasps there, >> but overall it seems to work! >> >> Tested without multipath routing, without keep state, and the behavior >> are the same. >> >> Will apreciate any kind of help on this, thank you in advance. >> >> Raimundo Santos
