Re: Source IPv4 address selection vs BGP IX connection
On 25 Apr 2024, at 15:56, Gregory Shapiro wrote: >> of course, gethostid(3) is now deprecated in favour of sysctl(3), and the >> hostid(8) command is gone, and there's now more than one flavour of >> Internet-capable UNIX in the world, and there's more than one Internet >> address family now. so what i did in 1990 is a guide only inasmuch as some >> way should exist to change the default local address of a socket so that it >> isn't the address of the interface used for the destination. if that happens >> i hope we coordinate with Linux and with the other BSD's. > > Linux already has a model to give a hint for source address selection via > route table "hints". When adding routes (either manually via `ip route' > or via things like bird2 BGP daemon), Linux supports setting a source IP > for when that route is used. > > Interestingly, JunOS (which I believe is based on FreeBSD) also supports > a way to specify a default IPv4 source address, preferring the primary address > on lo0 that is not 127.0.0.1. It is a common practice for BGP systems to > attach their announced IPs to the loopback interface. > > https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/default-address-selection-edit-system.html > > For the Linux and bird (BGP) documentation: > > Linux > - > http://linux-ip.net/html/tools-ip-route.html#ex-tools-ip-route-add-src > > "The src option provides a hint to the kernel for source address selection. > When you are working with multiple routing tables and different classes of > traffic, you can ease your administrative burden, by hosting several > different IPs on your linux machine and setting the source address > differently, depending on the type of traffic. > > In the example below, let's assume that our masquerading host also runs a DNS > resolver for the internal network and we have selected all of the outbound > DNS packets to be routed according to table 7 [53]. Now, any packet which > originates on this box (or is masqueraded through this table) will have its > source IP set to 205.254.211.198. > > Example D.19. Using src in a routing command with route add > > [root@masq-gw]# ip route add default via 205.254.211.254 src 205.254.211.198 > table 7 > " > > man ip-route > > "src ADDRESS > the source address to prefer when sending to the > destinations covered by the route prefix." When you first asked this question, my first thought was that this should be in the routing table. It seems to me that choosing the source address is more a function of the destination than of the process (vnet, jail, etc). In fact, this problem seemed familiar, so I went looking. It turns out that this feature has been available since 4.4BSD. route(8) has a keyword to do just this, -ifa (interface address). It only seems to work when the alias is on the same interface. It also seems to be broken in -current and 14.0, but I got it to work with 13.3 and 12.4. While experimenting, I tried to use -ifp as well, but it seems to be ignored; route add -ifp foobar ... does not fail. (12.4 got the interface wrong when the alias was on the loopback.) Anyone know why -ifa is ineffective in 14.0 and -current? It could be fallout from netlink. The documentation is weak at best; route(8) says only "the -ifp or -ifa modifiers may be used to determine the interface or interface address". "route get" does not display the ifa; I think it did at one time. I'll also note that binding the desired source address manually works; ping -S uses this. Mike > > Bird (BGP Daemon) > > "The Kernel protocol defines several attributes. These attributes are > translated to appropriate system (and OS-specific) route attributes. We > support these attributes: > .. > ip krt_prefsrc > (Linux) The preferred source address. Used in source address selection for > outgoing packets. Has to be one of the IP addresses of the router."
Re: Source IPv4 address selection vs BGP IX connection
> > of course, gethostid(3) is now deprecated in favour of sysctl(3), and the > > hostid(8) command is gone, and there's now more than one flavour of > > Internet-capable UNIX in the world, and there's more than one Internet > > address family now. so what i did in 1990 is a guide only inasmuch as some > > way should exist to change the default local address of a socket so that it > > isn't the address of the interface used for the destination. if that happens > > i hope we coordinate with Linux and with the other BSD's. > > Linux already has a model to give a hint for source address selection via > route table "hints". When adding routes (either manually via `ip route' > or via things like bird2 BGP daemon), Linux supports setting a source IP > for when that route is used. > > Interestingly, JunOS (which I believe is based on FreeBSD) also supports > a way to specify a default IPv4 source address, preferring the primary address > on lo0 that is not 127.0.0.1. The part of JunOS which is FreeBSD based runs inside a VM on Juniper routers and is a control plane only thing, what your probably seing is an artifact of the switch to Linux by Juniper. > It is a common practice for BGP systems to > attach their announced IPs to the loopback interface. These routers are almost always reject type routes and used to hold the routes 'UP' even if the IGP of an AS flaps them. This is not done to effect source IP address selection. > > https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/default-address-selection-edit-system.html > > For the Linux and bird (BGP) documentation: > > Linux > - > http://linux-ip.net/html/tools-ip-route.html#ex-tools-ip-route-add-src > > "The src option provides a hint to the kernel for source address selection. > When you are working with multiple routing tables and different classes of > traffic, you can ease your administrative burden, by hosting several > different IPs on your linux machine and setting the source address > differently, depending on the type of traffic. > > In the example below, let's assume that our masquerading host also runs a DNS > resolver for the internal network and we have selected all of the outbound > DNS packets to be routed according to table 7 [53]. Now, any packet which > originates on this box (or is masqueraded through this table) will have its > source IP set to 205.254.211.198. This is in effect what someone else suggested using multiple route tables in freeBSD and ipfw rules, iirc. > > Example D.19. Using src in a routing command with route add > > [root@masq-gw]# ip route add default via 205.254.211.254 src 205.254.211.198 > table 7 > " > > man ip-route > > "src ADDRESS > the source address to prefer when sending to the > destinations covered by the route prefix." > > > Bird (BGP Daemon) > > "The Kernel protocol defines several attributes. These attributes are > translated to appropriate system (and OS-specific) route attributes. We > support these attributes: > .. > ip krt_prefsrc > (Linux) The preferred source address. Used in source address selection for > outgoing packets. Has to be one of the IP addresses of the router." This is for the purpose of bird process only and does not effect other processes. This is mainly used to control the source addressed used for eBGP sessions, though this can fail misserably when you have multiple interfaces speaking eBGP. Though I can see some merit in what your suggesting to do, I also know that it comes with a many problems as it might solve, like ping would stop working on a local link if the assigned "src IP" is not routable by the entity your trying to ping, which for me would be the case in many places as the IX IP only appears on the IX interface and nothing any place inside or outside my AS has a route to it. I believe this is the intent of the IX IP policy on use of unrouteable IP addresses. -- Rod Grimes rgri...@freebsd.org
Re: Source IPv4 address selection vs BGP IX connection
> of course, gethostid(3) is now deprecated in favour of sysctl(3), and the > hostid(8) command is gone, and there's now more than one flavour of > Internet-capable UNIX in the world, and there's more than one Internet > address family now. so what i did in 1990 is a guide only inasmuch as some > way should exist to change the default local address of a socket so that it > isn't the address of the interface used for the destination. if that happens > i hope we coordinate with Linux and with the other BSD's. Linux already has a model to give a hint for source address selection via route table "hints". When adding routes (either manually via `ip route' or via things like bird2 BGP daemon), Linux supports setting a source IP for when that route is used. Interestingly, JunOS (which I believe is based on FreeBSD) also supports a way to specify a default IPv4 source address, preferring the primary address on lo0 that is not 127.0.0.1. It is a common practice for BGP systems to attach their announced IPs to the loopback interface. https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/default-address-selection-edit-system.html For the Linux and bird (BGP) documentation: Linux - http://linux-ip.net/html/tools-ip-route.html#ex-tools-ip-route-add-src "The src option provides a hint to the kernel for source address selection. When you are working with multiple routing tables and different classes of traffic, you can ease your administrative burden, by hosting several different IPs on your linux machine and setting the source address differently, depending on the type of traffic. In the example below, let's assume that our masquerading host also runs a DNS resolver for the internal network and we have selected all of the outbound DNS packets to be routed according to table 7 [53]. Now, any packet which originates on this box (or is masqueraded through this table) will have its source IP set to 205.254.211.198. Example D.19. Using src in a routing command with route add [root@masq-gw]# ip route add default via 205.254.211.254 src 205.254.211.198 table 7 " man ip-route "src ADDRESS the source address to prefer when sending to the destinations covered by the route prefix." Bird (BGP Daemon) "The Kernel protocol defines several attributes. These attributes are translated to appropriate system (and OS-specific) route attributes. We support these attributes: .. ip krt_prefsrc (Linux) The preferred source address. Used in source address selection for outgoing packets. Has to be one of the IP addresses of the router."
Re: Source IPv4 address selection vs BGP IX connection
agreed. and one of my mods to the ultrix (~4.3bsd) kernel for gatekeeper.dec.com back in ~1990 was to use the result of gethostid(3) if that result was nonzero and if a socket was not already bound. so named(8) and ntpd(8) and anything else that used explicit binding got what they expected, but the vast majority who just used INADDR_ANY (or more often just bzero(3)'d the sockaddr) would get what the sysadmin wanted. multihoming wasn't well understood and has gotten worse since. of course, gethostid(3) is now deprecated in favour of sysctl(3), and the hostid(8) command is gone, and there's now more than one flavour of Internet-capable UNIX in the world, and there's more than one Internet address family now. so what i did in 1990 is a guide only inasmuch as some way should exist to change the default local address of a socket so that it isn't the address of the interface used for the destination. if that happens i hope we coordinate with Linux and with the other BSD's. Gregory Shapiro wrote on 2024-04-24 11:00: I still see value in source IP selection, even outside of the IX use case. -- P Vixie
Re: Source IPv4 address selection vs BGP IX connection
On 4/23/2024 10:12 PM, Gregory Shapiro wrote: Short version: Using FreeBSD as a BGP router has network issues caused by suboptimal default IPv4 source address selection when connected to Internet Exchanges (which are required to use IPs that aren't routable on the Internet). I was hoping to find more elegant workarounds or encourage FreeBSD to add source IPv4 selection akin to the existing IPv6 source address selection (no_prefer_iface and prefer_source). I assume that there is a group of BGP enthusiasts using FreeBSD lurking on freebsd-net. What have you done to solve this problem? For DNS in such situations I start unbound locally and bind it to an internal interface or an IP on lo0 and then tell unbound to just use that IP only (outgoing-interface IIRC) that is advertised out as a work around. Its not a proper solution, but will get your resolver working at least. I run into this problem in layered networks where the next hop is often RFC 1918 addrs. I bind applications to internal NICs that have addresses that have routing to/from. ---Mike
Re: Source IPv4 address selection vs BGP IX connection
> The mistake your making, IMHO, is that an IX connected eBGP FreeBSD > router _SHOULD NOT_ be doing ANYTHING other than BGP on the IX > connected interface, and anything like DNS and outbound SMTP should be > going inward on the AS, not outward to the internet. Fair point and thank you for the advice. I am locking it down to an extent (denying all inbound ports except 22, 179 from an ipfw table list of trusted hosts/peers/upstreams/downstreams) but not as tightly as you suggest as I do use some on-Internet services. Specifically, port 25 to my own mail server (not unwashed Internet service, but sitting off of a different network) for system generated mail (cron, /etc/periodic/ script output), 53 to admittedly "unwashed" Google DNS, and 123 to FreeBSD's NTP pool (again "unwashed" to an extent). I will look at using local instances for the latter two. I still see value in source IP selection, even outside of the IX use case.
Re: Source IPv4 address selection vs BGP IX connection
On Wed, Apr 24, 2024 at 07:10:51AM +0200, Marek Zarychta wrote: > W dniu 24.04.2024 o 04:12, Gregory Shapiro pisze: > > Short version: > > > > Using FreeBSD as a BGP router has network issues caused by suboptimal > > default IPv4 source address selection when connected to Internet > > Exchanges (which are required to use IPs that aren't routable on the > > Internet). I was hoping to find more elegant workarounds or encourage > > FreeBSD to add source IPv4 selection akin to the existing IPv6 source > > address selection (no_prefer_iface and prefer_source). > In this case, probably best solution will probably be using multiple FIBs. > Running a BGP routing daemon under not default FIB after assigning its > interface to this FIB should solve the problem but it might create > eventually new problems to solve (for example in which FIB should imported > routes be stored). Thank you for sharing the ideas. This first idea seems to negate the positive impact of multihoming and connecting to the IX for peering and additional transit. If the routes aren't usable in the default routing RIB (for downstream/LAN hosts or the router itself), then there doesn't seem to be a purpose of having multiple routes. > It's also possible to set and use non-default FIB for DNS lookups and > maintenance tasks like pkg upgrade (setfib -1 pkg ). This approach is > probably more straightforward to conduct. Until you consider that not all work is done from the command line such that 'setfib' can proceed every command. What if cron wants to send a message with output from a cron job? What if a system service needs to connect to another host (e.g., ntpd)? Even to ssh into the system, sshd needs DNS for PTR lookups. I really think this isn't an issue with routing (and therefore can't be fixed elegantly by changing routing). It is an issue with source IP selection (one that has been addressed for IPv6, just not IPv4). I'll try to dig into how FreeBSD does source IP selection and see if I can add code to tune that process.
Re: Source IPv4 address selection vs BGP IX connection
> Short version: > > Using FreeBSD as a BGP router has network issues caused by suboptimal > default IPv4 source address selection when connected to Internet > Exchanges (which are required to use IPs that aren't routable on the > Internet). I was hoping to find more elegant workarounds or encourage > FreeBSD to add source IPv4 selection akin to the existing IPv6 source > address selection (no_prefer_iface and prefer_source). > > > Long version: > > Unless I'm mistaken, today, there is no way to set the default > IPv4 source address for connections like there is with IPv6 (using > no_prefer_iface and prefer_source). > > It appears the default source IP is chosen based on IP address of > the outbound interface for the packet. This presents a problem on > FreeBSD systems acting as BGP routers that have connections to Internet > exchanges (IX). One of the rules of IX IP addresses is that they are > must not be routable on the Internet. > > As a simple example, a system with two Ethernet interfaces, one to the > transit provider and one to an IX would look like this: > > vtnet0: flags=1008843 metric > 0 mtu 1500 > description: Uplink > inet 193.148.250.141 netmask 0xff00 broadcast 193.148.250.255 > vtnet1: flags=1008843 metric > 0 mtu 1500 > description: IX > inet 185.1.147.211 netmask 0xff00 broadcast 185.1.147.255 > > Then if /etc/resolv.conf contains 8.8.8.8 and BGP selects a route for > 8.8.8.0/24 over the IX, you end up with: > > # route -n get 8.8.8.8 >route to: 8.8.8.8 > destination: 8.8.8.0 >mask: 255.255.255.0 > gateway: 185.1.147.22 > fib: 0 > interface: vtnet1 > flags: > recvpipe sendpipe ssthresh rtt,msecmtuweightexpire >0 0 0 0 1500 1 0 > > And DNS on the system doesn't work as all DNS requests go out with a > source address of 185.1.147.211 (the IX endpoint) which isn't exported > as an Internet route. > > While I can set a static route for 8.8.8.8 for this particular case, it > would be messy to have to set up static routes for every possible local > connection (other DNS servers, outbound SMTP for periodic/cron mail, > etc.). The mistake your making, IMHO, is that an IX connected eBGP FreeBSD router _SHOULD NOT_ be doing ANYTHING other than BGP on the IX connected interface, and anything like DNS and outbound SMTP should be going inward on the AS, not outward to the internet. I must ask why your using 8.8.8.8 and not your own nameservers? Why would you want or even allow outbound SMTP from such a critical infustructure point go out over the unwashed internet? One of the reasons for using the non-routable IP on IX connected eBGP routers is to minimize the exposure footprint, and what you seem to be doing is defeating that minimization by wanting to expose another IP on that very box to the public internet. > > I assume that there is a group of BGP enthusiasts using FreeBSD lurking > on freebsd-net. What have you done to solve this problem? I only trust AS internal objects from my eBGP routers, they have no need to speak to the unwashed internet other than to IX peers. > I'd also love to hear other tips for running BGP on FreeBSD. Lock it down as tight as you can if your IX connected. I dont even allow inbound BGP connection setup, all eBGP sessions have to be initiated by my router. ipfw -a list 20179 20179 23854 1131316 deny log tcp from any to any 179 This is at an ISP peer, not an IX, so not a private IX IP range, but 23854 attempts to connect to my bgp. -- Rod Grimes rgri...@freebsd.org
Re: Source IPv4 address selection vs BGP IX connection
W dniu 24.04.2024 o 04:12, Gregory Shapiro pisze: Short version: Using FreeBSD as a BGP router has network issues caused by suboptimal default IPv4 source address selection when connected to Internet Exchanges (which are required to use IPs that aren't routable on the Internet). I was hoping to find more elegant workarounds or encourage FreeBSD to add source IPv4 selection akin to the existing IPv6 source address selection (no_prefer_iface and prefer_source). Long version: Unless I'm mistaken, today, there is no way to set the default IPv4 source address for connections like there is with IPv6 (using no_prefer_iface and prefer_source). It appears the default source IP is chosen based on IP address of the outbound interface for the packet. This presents a problem on FreeBSD systems acting as BGP routers that have connections to Internet exchanges (IX). One of the rules of IX IP addresses is that they are must not be routable on the Internet. As a simple example, a system with two Ethernet interfaces, one to the transit provider and one to an IX would look like this: vtnet0: flags=1008843 metric 0 mtu 1500 description: Uplink inet 193.148.250.141 netmask 0xff00 broadcast 193.148.250.255 vtnet1: flags=1008843 metric 0 mtu 1500 description: IX inet 185.1.147.211 netmask 0xff00 broadcast 185.1.147.255 Then if /etc/resolv.conf contains 8.8.8.8 and BGP selects a route for 8.8.8.0/24 over the IX, you end up with: # route -n get 8.8.8.8 route to: 8.8.8.8 destination: 8.8.8.0 mask: 255.255.255.0 gateway: 185.1.147.22 fib: 0 interface: vtnet1 flags: recvpipe sendpipe ssthresh rtt,msecmtuweightexpire 0 0 0 0 1500 1 0 And DNS on the system doesn't work as all DNS requests go out with a source address of 185.1.147.211 (the IX endpoint) which isn't exported as an Internet route. While I can set a static route for 8.8.8.8 for this particular case, it would be messy to have to set up static routes for every possible local connection (other DNS servers, outbound SMTP for periodic/cron mail, etc.). I assume that there is a group of BGP enthusiasts using FreeBSD lurking on freebsd-net. What have you done to solve this problem? I'd also love to hear other tips for running BGP on FreeBSD. In this case, probably best solution will probably be using multiple FIBs. Running a BGP routing daemon under not default FIB after assigning its interface to this FIB should solve the problem but it might create eventually new problems to solve (for example in which FIB should imported routes be stored). It's also possible to set and use non-default FIB for DNS lookups and maintenance tasks like pkg upgrade (setfib -1 pkg ). This approach is probably more straightforward to conduct. -- Marek Zarychta
