On 2020/10/19 15:35, David Gwynne wrote:
> every few years i try and use route-to in pf, and every time it
> goes badly. i tried it again last week in a slightly different
> setting, and actually tried to understand the sharp edges i hit
> this time instead of giving up. it turns out there are 2 or 3
> different things together that have cause me trouble, which is why
> the diff below is so big.
I used to route-to/reply-to quite a lot at places with poor internet
connections to split traffic between lines (mostly those have better
connections now so I don't need it as often). It worked as I expected -
but I only ever used it with the interface specified.
I mostly used it with pppoe interfaces so the peer address was unknown
at ruleset load time. (I was lucky and had static IPs my side, but the
ISP side was variable). I relied on the fact that once packets are
directed at a point-point interface there's only one place for them to
go. I didn't notice that ":peer" might be useful here (and the syntax
'route-to pppoe1:peer@pppoe1' is pretty awkward so I probably wouldn't
have come up with it), I had 0.0.0.1@pppoe1, 0.0.0.2@pppoe2 etc
(though actually I think it works with $any_random_address@pppoeX).
> the first and i would argue most fundamental problem is a semantic
> problem. if you ask a random person who has some clue about networks
> and routing what they would expect the "argument" to route-to or
> reply-to to be, they would say "a nexthop address" or "a gateway
> address". eg, say i want to force packets to a specific backend
> server without using NAT, i would write a rule like this:
>
> n_servers="192.0.2.128/27"
> pass out on $if_internal to $n_servers route-to 192.168.0.1
>
> pfctl will happily parse this, shove it into the kernel, let you read
> the rules back out again with pfctl -sr, and it all looks plausible, but
> it turns out that it's using the argument to route-to as an interface
> name. because rulesets can refer to interfaces that don't exist yet, pf
> just passes the IP address around as a string, hoping i'll plug in an
> interface with a driver name that looks like an ip address. i spent
> literally a day trying to figure out why a rule like this wasn't
> working.
I don't think I tried this, but the pf.conf(5) BNF syntax suggests it's
supposed to work. So either doc or implementation bug there.
route = ( "route-to" | "reply-to" | "dup-to" )
( routehost | "{" routehost-list "}" )
[ pooltype ]
routehost-list = routehost [ [ "," ] routehost-list ]
routehost = host | host "@" interface-name |
"(" interface-name [ address [ "/" mask-bits ] ] ")"
> the second problem is that the pf_route calls from pfsync don't
> have all the information it is supposed to have. more specifically,
> an ifp pointer isn't set which leads to a segfault. the ifp pointer
> isn't set because pfsync doesnt track which interface a packet is
> going out, it assumes the ip layer will get it right again later, or a
> rule provided something usable.
>
> the third problem is that pf_route relies on information from rules to
> work correctly. this is a problem in a pfsync environment because you
> cannot have the same ruleset on both firewalls 100% of the time, which
> means you cannot have route-to/reply-to behave consistently on a pair of
> firwalls 100% of the time.
I didn't run into this because pppoe(4) and pfsync/carp don't really
go well together, but ouch!
> all of this together makes things work pretty obviously and smoothly.
> in my opinion anyway. route-to now works more like rdr-to, it just
> feels like it changes the address used for the route lookup rather
> than changing the actual IP address in the packet. it also works
> predictably in a pfsync pair, which is great from the point of view of
> high availability.
>
> the main caveat is that it's not backward compatible. if you're already
> using route-to, you will need to tweak your rules to have them parse.
> however, i doubt anyone is using this stuff because it feels very broken
> to me.
Do you expect this to work with a bracketed "address" to defer lookup
until rule evaluation time? i.e.
pass out proto tcp to any port 22 route-to (pppoe1:peer)
I think that will be all that's needed to allow converting the pppoe
use case. I don't have a multiple pppoe setup handy but I can probably
hack together some sort of test.
I've also used route-to with squid "transparent" proxying (shown in
the pkg-readme), I don't do that any more but I can put a squid test
together easily enough.
> @@ -1842,37 +1833,18 @@ pfrule : action dir logquick interface
> decide_address_family($7.src.host, &r.af);
> decide_address_family($7.dst.host, &r.af);
>
> - if ($8.route.rt) {
> + if ($8.rt) {
> + if ($8.rt != PF_DUPTO && !r.direction) {
> + yyerror("direction must be explicit"
> + " with rules that specify routing");
> + YYERROR;
> + }
> if (!r.direction) {
> yyerror("direction must be explicit "
> "with rules that specify routing");
> YYERROR;
> }
this stood out on reading the diff - the added if block doesn't change
any outcome, should it have actually been like this?
- if (!r.direction) {
+ if ($8.rt != PF_DUPTO && !r.direction) {
yyerror("direction must be explicit"
" with rules that specify routing");
YYERROR;
}
