Hi, just confirming one thing: did you flush the pf states between the tests? I must admit, I mainly glanced the problem, so sorry if this is an old tip. This was the first thing that popped into my mind when reading about your solution.
-- Sincerely, Ville Valkonen On 5 June 2013 22:39, Raimundo Santos <[email protected]> wrote: > I've got the issue solved by disabling states on all rules which deal with > the tproxy. > > > On 4 June 2013 11:28, Raimundo Santos <[email protected]> wrote: > >> I am guessing that the problem lies with flags S/SA. >> >> Changing all rules to flags any, and the packets hits the rules, but >> things go worse: no web navigation... this is driving me mad! >> >> >> >> On 3 June 2013 13:09, Raimundo Santos <[email protected]> wrote: >> >>> Hi there! >>> >>> I asked, without an answer, something about nat-to and real IPs. Well, I >>> really need an answer there, so if someone get a clue, I will be glad tho >>> hear :) >>> >>> Now, to the new issue! >>> >>> Here in our WiFi ISP we are have contracted a tproxy service from FreeBSD >>> Brasil. It is somehow working, but I can not figure out exactly how. Here >>> is a diagram of the desired paths: >>> >>> http://devio.us/~raitech/Obsd53PfTproxy.png >>> >>> These are my rules by now: >>> >>> RFC1918 = "{ 172.16/12, 192.168/16, 10/8, 127/8 }" >>> table <INT_NET> persist { internal nets, all valid IPs } >>> >>> ext_if_1 = "em0" >>> ext_gw_1 = "187.72.X.X" >>> ext_ip_1 = "187.72.X.X" >>> >>> ext_if_2 = "em1" >>> ext_gw_2 = "187.72.X.X" >>> ext_ip_2 = "187.72.X.X" >>> >>> ext_if_3 = "alc0" >>> ext_gw_3 = "187.72.X.X" >>> ext_ip_3 = "187.72.X.X" >>> >>> int_if_1 = "em2" >>> int_gw_1 = "187.72.X.X" >>> int_ip_1 = "187.72.X.X" >>> >>> squid_master_if = "em3" >>> squid_master_gw = "187.72.X.X" >>> squid_master_ip = "187.72.X.X" >>> >>> set limit states 6304000 >>> set limit tables 5000 >>> set limit src-nodes 200000 >>> set limit frags 3000 >>> set optimization aggressive >>> set state-defaults pflow, no-sync >>> >>> set skip on lo >>> >>> block in log quick on { \ >>> $ext_if_1, \ >>> $ext_if_2, \ >>> $ext_if_3, \ >>> $squid_master_if, \ >>> $int_if_1 } from $RFC1918 label "blocking RFC1918" >>> >>> # trying to prioritizing ACKs... >>> match set prio (3,5) >>> # ... and all traffic http. https over the others >>> match proto tcp to port { http, https } set prio (5,6) >>> match proto tcp from port { http, https } set prio (5,6) >>> >>> match proto tcp to port { ssh, 9876 } set prio(5,7) >>> >>> pass in on $int_if_1 proto tcp from { <INT_NET>, $int_gw_1 } to port http >>> \ >>> route-to ($squid_master_if $squid_master_gw) >>> >>> pass in on { $ext_if_1, $ext_if_2, $ext_if_3 } proto tcp from port http \ >>> to { <INT_NET>, $int_gw_1 } \ >>> route-to ($squid_master_if $squid_master_gw) >>> >>> pass in on $squid_master_if proto tcp from { <INT_NET>, $int_gw_1 } to \ >>> port http no state route-to \ >>> { \ >>> ($ext_if_1 $ext_gw_1) , \ >>> ($ext_if_2 $ext_gw_2) \ >>> } least-states label "cahce external outbound balancing" >>> >>> pass in on $squid_master_if proto tcp from port http \ >>> to { <INT_NET>, $int_gw_1 } route-to ($int_if_1 $int_gw_1) \ >>> label "cahce internal outbound routing" >>> >>> An here are a pfctl -vsr output: >>> >>> block drop in log quick on em0 inet from 172.16.0.0/12 to any label >>> "blocking RFC1918" >>> [ Evaluations: 61764339 Packets: 332 Bytes: 32854 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em0 inet from 192.168.0.0/16 to any label >>> "blocking RFC1918" >>> [ Evaluations: 5883927 Packets: 114 Bytes: 28621 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em0 inet from 10.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 5883813 Packets: 170 Bytes: 18354 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em0 inet from 127.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 5883643 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em1 inet from 172.16.0.0/12 to any label >>> "blocking RFC1918" >>> [ Evaluations: 60684174 Packets: 305 Bytes: 30912 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em1 inet from 192.168.0.0/16 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6862827 Packets: 93 Bytes: 9232 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em1 inet from 10.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6862734 Packets: 196 Bytes: 19396 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em1 inet from 127.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6862538 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on alc0 inet from 172.16.0.0/12 to any label >>> "blocking RFC1918" >>> [ Evaluations: 50726925 Packets: 304 Bytes: 30856 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on alc0 inet from 192.168.0.0/16 to any label >>> "blocking RFC1918" >>> [ Evaluations: 1251 Packets: 79 Bytes: 8268 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on alc0 inet from 10.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 1172 Packets: 152 Bytes: 16948 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on alc0 inet from 127.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 1020 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em3 inet from 172.16.0.0/12 to any label >>> "blocking RFC1918" >>> [ Evaluations: 50726392 Packets: 304 Bytes: 30856 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em3 inet from 192.168.0.0/16 to any label >>> "blocking RFC1918" >>> [ Evaluations: 13589809 Packets: 76 Bytes: 8132 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em3 inet from 10.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 13589733 Packets: 152 Bytes: 16948 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em3 inet from 127.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 13589581 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em2 inet from 172.16.0.0/12 to any label >>> "blocking RFC1918" >>> [ Evaluations: 39571927 Packets: 10414 Bytes: 478685 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em2 inet from 192.168.0.0/16 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6364466 Packets: 1779 Bytes: 142401 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em2 inet from 10.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6362687 Packets: 32496 Bytes: 1375238 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> block drop in log quick on em2 inet from 127.0.0.0/8 to any label >>> "blocking RFC1918" >>> [ Evaluations: 6330191 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match all set ( prio(3, 5) ) >>> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: >>> 3831 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any to any port = 80 set ( prio(5, 6) ) >>> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: >>> 3831 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any to any port = 443 set ( prio(5, 6) ) >>> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any port = 80 to any set ( prio(5, 6) ) >>> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any port = 443 to any set ( prio(5, 6) ) >>> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any to any port = 22 set ( prio(5, 7) ) >>> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> match proto tcp from any to any port = 9876 set ( prio(5, 7) ) >>> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass all no state allow-opts >>> [ Evaluations: 61717379 Packets: 61549113 Bytes: 41451833770 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 61717379 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em1 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 55197296 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on alc0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 38378103 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 48038032 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em1 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 44966361 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on alc0 inet proto tcp from any port = 80 to 187.72.X.X flags >>> S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 41608198 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em2 inet proto tcp from <INT_NET> to any port = 80 flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 48044445 Packets: 1439990 Bytes: 894473590 States: >>> 435 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 40060 ] >>> pass in on em2 inet proto tcp from 187.72.X.X to any port = 80 flags S/SA >>> keep state (no-sync, pflow) route-to 187.72.X.X@em3 >>> [ Evaluations: 3694317 Packets: 12437474 Bytes: 9381159120 States: >>> 3396 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 128206] >>> pass in on em3 inet proto tcp from <INT_NET> to any port = 80 no state >>> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_0> >>> least-states >>> [ Evaluations: 38420511 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em3 inet proto tcp from 187.72.X.X to any port = 80 no state >>> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_1> >>> least-states >>> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em3 inet proto tcp from any port = 80 to <INT_NET> flags S/SA >>> keep state (no-sync, pflow) label "cahce internal outbound routing" >>> route-to 187.72.X.X@em2 >>> [ Evaluations: 13731058 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> pass in on em3 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA >>> keep state (no-sync, pflow) label "cahce internal outbound routing" >>> route-to 187.72.X.X@em2 >>> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: >>> 0 ] >>> [ Inserted: uid 0 pid 19584 State Creations: 0 ] >>> >>> This is the same behavior with or without multipath routing. What >>> bahavior? Well, only rules for in on em3 that are destineted to internal >>> network are working, the others barelly catches a few thousands of packets. >>> Very strange... >>> >>> But, as said before: more strange is the fact that the cache solution is >>> almost working, just some delays to load a page here, youtube gasps there, >>> but overall it seems to work! >>> >>> Tested without multipath routing, without keep state, and the behavior >>> are the same. >>> >>> Will apreciate any kind of help on this, thank you in advance. >>> >>> Raimundo Santos
