I've got a box with an LTE cellular modem in it whose purpose is to provide a backup connection to the Internet if the hardwire service goes down. It's running OSPF to connect to the rest of the network, and the only time any traffic should go over the cellular link (which is slower and bandwidth capped) is if the hardwire interconnection is down, including ideally traffic generated from the system itself.
I have that part working, by adding in a local static default route to the cellular gateway with less priority than the OSPF default route. However, for testing purposes, I'd like to be able to poke out the cellular link on an as-needed basis without having to switch the entire box over to using it. Virtual routing tables looked perfect for this purpose, as I could just spawn a single process with a different default route, we do something similar with network name spaces under Linux. However, I can't quite get it to work. What I'd really like is to be able to make a copy of the current system routing table, then change one thing about it. However, a new rdomain shows up with no routes or interfaces in the routing table. I can add the new default route pointing out the cellular link, and get traffic to go out there. But I haven't sorted out how to make all the traffic for my internal network still go through the internal link rather than get sent out the default route. While ideally all the OSPF routes would propagate to the other routing domain I tried just adding a static to the /16 for our internal address space: Internet: Destination Gateway Flags Refs Use Mtu Prio Iface default 24.x.x.x UGS 0 6 - 8 umb0 10.0/16 10.128.0.21 UGS 0 0 - 8 em0 That doesn't work; the documentation says you need to get pf to pass packets across routing domains. However, it says: rtable number Used to select an alternate routing table for the routing lookup. Only effective before the route lookup happened, i.e. when filtering inbound. Unfortunately, for traffic originating from the system itself, there isn't really an "inbound" interface? So I'm not sure what pf rule would make this work. Is it just not possible, or am I missing something? Thanks much.