Re: load balancing with rdomains
On Mon, 18 Dec 2023 14:08:04 +0100 Claudio Jeker wrote: > On Mon, Dec 18, 2023 at 01:53:50PM +0100, Marko Cupać wrote: > > What OpenBSD FAQ https://www.openbsd.org/faq/faq6.html#Multipath > > says for a bit different scenario applies to some extent for this > > one as well: > > > > "It's worth noting that if an interface used by a multipath route > > goes down (i.e., loses carrier), the kernel will still try to > > forward packets using the route that points to that interface. This > > traffic will of course be blackholed and end up going nowhere. It's > > highly recommended to use ifstated(8) to check for unavailable > > interfaces and adjust the routing table accordingly." > > Uhm. This is not accurate. The kernel tracks interface state on > routes and will not select a multipath route that is not considered > UP. There is a smaller issue when there is no other multipath route. > The lookup will select the route and not fall back to a less specific > one that is still up. > > Could please someone update the FAQ? I would like to contribute to the FAQ, but I'm not sure in which way to go. According to my tests, above is not literally correct in described case (route goes down on lost carrier). However, in frequent scenario where interface is up, route is valid, but ISP's side won't route our packets (which is perceived as "link is down" by a user), a mechanism is still needed to prevent sending packets over that interface. Would "It's worth noting that if an interface used by a multipath route loses data link while physical link is active..." be more appropriate? More radical option would be to describe rdomain-based solution instead of current examples in both: https://www.openbsd.org/faq/faq6.html#Multipath https://www.openbsd.org/faq/pf/pools.html#outgoing Best regards, -- Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupać https://www.mimar.rs/
Re: load balancing with rdomains
On Mon, Dec 18, 2023 at 01:53:50PM +0100, Marko Cupać wrote: > On Sat, 16 Dec 2023 18:53:29 +0100 > Petr Ročkai wrote: > > > Hi, > > > > On Sat, Dec 16, 2023 at 06:37:54PM +0100, Marko Cupać wrote: > > > pass in on em0 from (em0:network) to probability 50% > > > rtable 1 pass in on em0 from (em0:network) to probability > > > 50% rtable 2 > > > > IIUIC these two only add up to 75% probability – you presumably want > > probability 50% on the second of the two (the first one then being a > > match for everything that the later rule doesn't take up). > > Thank you, I can confirm that your solution: > > pass in on em0 from (em0:network) to rtable 1 > pass in on em0 from (em0:network) to probability 50% rtable 2 > > ... results in what I was trying to achieve - it load balances over both > uplinks without any blocked packets as long as both uplinks are active. > > What OpenBSD FAQ https://www.openbsd.org/faq/faq6.html#Multipath says > for a bit different scenario applies to some extent for this one as > well: > > "It's worth noting that if an interface used by a multipath route goes > down (i.e., loses carrier), the kernel will still try to forward > packets using the route that points to that interface. This traffic > will of course be blackholed and end up going nowhere. It's highly > recommended to use ifstated(8) to check for unavailable interfaces and > adjust the routing table accordingly." Uhm. This is not accurate. The kernel tracks interface state on routes and will not select a multipath route that is not considered UP. There is a smaller issue when there is no other multipath route. The lookup will select the route and not fall back to a less specific one that is still up. Could please someone update the FAQ? > ...except - if I'm not mistaken - ifstated should in this case adjust > pf ruleset instead of routing table. > > If so, would using anchors be the best way? Any working examples to > share? I used some simple ifstated rules but it is hard to wrap my head > around probability percentages for all the use cases - first link up, > second down and vice versa. > > Thank you in advance, > > -- > Before enlightenment - chop wood, draw water. > After enlightenment - chop wood, draw water. > > Marko Cupać > https://www.mimar.rs/ > -- :wq Claudio
Re: load balancing with rdomains
On Sat, 16 Dec 2023 18:53:29 +0100 Petr Ročkai wrote: > Hi, > > On Sat, Dec 16, 2023 at 06:37:54PM +0100, Marko Cupać wrote: > > pass in on em0 from (em0:network) to probability 50% > > rtable 1 pass in on em0 from (em0:network) to probability > > 50% rtable 2 > > IIUIC these two only add up to 75% probability – you presumably want > probability 50% on the second of the two (the first one then being a > match for everything that the later rule doesn't take up). Thank you, I can confirm that your solution: pass in on em0 from (em0:network) to rtable 1 pass in on em0 from (em0:network) to probability 50% rtable 2 ... results in what I was trying to achieve - it load balances over both uplinks without any blocked packets as long as both uplinks are active. What OpenBSD FAQ https://www.openbsd.org/faq/faq6.html#Multipath says for a bit different scenario applies to some extent for this one as well: "It's worth noting that if an interface used by a multipath route goes down (i.e., loses carrier), the kernel will still try to forward packets using the route that points to that interface. This traffic will of course be blackholed and end up going nowhere. It's highly recommended to use ifstated(8) to check for unavailable interfaces and adjust the routing table accordingly." ...except - if I'm not mistaken - ifstated should in this case adjust pf ruleset instead of routing table. If so, would using anchors be the best way? Any working examples to share? I used some simple ifstated rules but it is hard to wrap my head around probability percentages for all the use cases - first link up, second down and vice versa. Thank you in advance, -- Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupać https://www.mimar.rs/
Re: load balancing with rdomains
On 2023-12-16, Petr Ročkai wrote: > Hi, > > On Sat, Dec 16, 2023 at 06:37:54PM +0100, Marko Cupać wrote: >> pass in on em0 from (em0:network) to probability 50% rtable 1 >> pass in on em0 from (em0:network) to probability 50% rtable 2 > > IIUIC these two only add up to 75% probability – you presumably want > probability 50% on the second of the two (the first one then being a > match for everything that the later rule doesn't take up). exactly. -- Please keep replies on the mailing list.
Re: load balancing with rdomains
Hi, On Sat, Dec 16, 2023 at 06:37:54PM +0100, Marko Cupać wrote: > pass in on em0 from (em0:network) to probability 50% rtable 1 > pass in on em0 from (em0:network) to probability 50% rtable 2 IIUIC these two only add up to 75% probability – you presumably want probability 50% on the second of the two (the first one then being a match for everything that the later rule doesn't take up). M. -- id' Ash = Ash; id' Dust = Dust; id' _ = undefined
Re: load balancing with rdomains
Normally, standard pc conf, I would suggest to browse Peter or my blog suggestion: 20230929 15:16 ( https://bsdload.com/bsd/?blogSP=30 ) My solution taken from "Building Linux and OpenBSD Firewalls" (see Internet Archive) to solve the no traffic prb caused by inserting the default "block in all" statement. I moved the following statements just after it: # dns pass in quick proto udp from any port domain to any pass out quick proto udp from any to any port domain # icmp pass in quick inet proto icmp all icmp-type 0 pass in quick inet proto icmp all icmp-type 3 pass in quick inet proto icmp all icmp-type 11 But indeed as you are dealing with routing traffic among interfaces you should go eventually experimenting including these dns and icmps of my tip. Hope you find this somewhat helpful! -- Nowarez Market Dec 16, 2023 18:41:05 Marko Cupać : > My test ruleset: > > ---start--- > block log all > > pass in on em0 from (em0:network) to > pass in on em0 from (em0:network) to probability 50% rtable 1 > pass in on em0 from (em0:network) to probability 50% rtable 2 > > pass out on em0 > pass out on em1 > pass out on em2 > ---end--- > > ... somewhat works, in a way that sessions from lan host to do > get load balanced to both rtables most of the time. However, some of > the sessions to (I tested with ssh) get denied by default > block rule initially: > > block in on em0: PR.IV.AT.E.35528 > PU.BL.I.C.22: tcp 0 (DF) [tos 0x48] > > and then, on consequent automatic ssh retry after a few seconds, get > moved to one of two rtables. > > From above I conclude that the two rules of 50% do not make a total of > 100% in pf's logic, and there are situations where a packet won't be > passed by any of the two. That unfortunately won't work for my use case.
Re: load balancing with rdomains
On Sat, 16 Dec 2023 10:25:07 - (UTC) Stuart Henderson wrote: > See "probability" in pf.conf(5). Thank you for the tip. My test ruleset: ---start--- block log all pass in on em0 from (em0:network) to pass in on em0 from (em0:network) to probability 50% rtable 1 pass in on em0 from (em0:network) to probability 50% rtable 2 pass out on em0 pass out on em1 pass out on em2 ---end--- ... somewhat works, in a way that sessions from lan host to do get load balanced to both rtables most of the time. However, some of the sessions to (I tested with ssh) get denied by default block rule initially: block in on em0: PR.IV.AT.E.35528 > PU.BL.I.C.22: tcp 0 (DF) [tos 0x48] and then, on consequent automatic ssh retry after a few seconds, get moved to one of two rtables. >From above I conclude that the two rules of 50% do not make a total of 100% in pf's logic, and there are situations where a packet won't be passed by any of the two. That unfortunately won't work for my use case. Or perhaps I'm configuring something wrong? Best regards, -- Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupać https://www.mimar.rs/
Re: load balancing with rdomains
On 2023-12-15, Marko Cupać wrote: > Hi, > > I have a router whose LAN interface is in default rdomain 0, ISP1 in > rdomain 1 and ISP2 in rdomain 2. Reason for this is a bit complicated, > involves wireguard tunneling, I will give more details if needed. > > LAN hosts can access Internet over ISP1 by means of: > > pass in on $if_lan from ($if_lan:network) to rtable 1 > > Also over ISP2 if I change above line to: > > pass in on $if_lan from ($if_lan:network) to rtable 2 > > Is it possible to load-balance over both ISPs / rdomains? See "probability" in pf.conf(5). -- Please keep replies on the mailing list.
load balancing with rdomains
Hi, I have a router whose LAN interface is in default rdomain 0, ISP1 in rdomain 1 and ISP2 in rdomain 2. Reason for this is a bit complicated, involves wireguard tunneling, I will give more details if needed. LAN hosts can access Internet over ISP1 by means of: pass in on $if_lan from ($if_lan:network) to rtable 1 Also over ISP2 if I change above line to: pass in on $if_lan from ($if_lan:network) to rtable 2 Is it possible to load-balance over both ISPs / rdomains? Thank you in advance, -- Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupać https://www.mimar.rs/
