=== BACKGROUND === I'm trying to set up an OpenBSD 6.1 server having two externally visible IP numbers through one physical network port, each IP mapping to a unique MAC address[1]. I have it mostly working, but my interfaces can't talk to each other.
All traffic should use the primary IP, and most services should listen on that. The secondary IP should only be used on-demand for one or two services. Thinking that separate routing tables can solve this, I have configured my network like this[2][3]: # cat hostname.em2 up # cat hostname.vether0 lladdr 00:00:00:00:00:02 # cat hostname.vether1 lladdr 00:00:00:00:00:03 rdomain 1 # cat hostname.bridge0 add em2 add vether0 add vether1 up !dhclient vether0 !dhclient vether1 # cat sysctl.conf net.inet.ip.forwarding=1 Leading to something like this[4]: (full post in monospace: http://paste.debian.net/928811 ) .----------------.---------. 10.0.0.1 .-,( ),-. | bridge0 | em2 | dhcp server .-( )-. | | (no ip) |---> gateway --->( internet ) | '---------' __________ '-( ).-' | | [_...__...°] '-.( ).-' .-----------. .-----------. | vether0 | | vether1 | | rdomain 0 | | rdomain 1 | | dhcp | | dhcp | | 10.0.0.2 | | 10.0.0.3 | '-----------'--'-----------' Everything else should be the default, this is on a clean 6.1 install. This configuration works great, vether0 and vether1 both gets an IP number from my DHCP server, all traffic goes out on vether0 by default, but I can select vether1 manually: # traceroute -nvq1 10.0.0.1 traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets 1 10.0.0.1 48 bytes to 10.0.0.2 0.994 ms # route -T1 exec traceroute -nvq1 10.0.0.1 traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets 1 10.0.0.1 48 bytes to 10.0.0.3 0.984 ms I can also reach each IP from outside the box. They are going in on em2, through the bridge, and in to vether0 or vether1 respectively. === PROBLEM === Now to my problem: I have no connection between vether0<->vether1. # traceroute -nvq1 10.0.0.3 traceroute to 10.0.0.3 (10.0.0.3), 64 hops max, 40 byte... 1 * 2 * ^C If I listen with tcpdump on the bridge, I see lots of unanswered arp who-has: # tcpdump -nti bridge0 tcpdump: listening on bridge0, link-type EN10MB arp who-has 10.0.0.3 tell 10.0.0.2 arp who-has 10.0.0.3 tell 10.0.0.2 ^C These packets even go out on em2 to my LAN, but no one ever answers. The same thing happens in reverse. I have experimented with these bridge settings: 'blocknonip' - adding or removing on members makes no difference 'discover' - should be the default, adding makes no difference 'learn' - should be the default, adding makes no difference === EXPECTATIONS === I expected that someone should answer those arp who-is requests, either vether1 directly, or the bridge0 who should know which interfaces it has. Is there something I must configure to make this work, or is my plan flawed from the start? === INFORMATION === Various information that could help answer my question (trimmed whitespace and boilerplate): # route -n show -inet Destination Gateway Flags Refs Use Mtu Prio Iface default 10.0.0.1 UGS 0 0 - 8 vether0 224/4 127.0.0.1 URS 0 0 32768 8 lo0 10.0.0/24 10.0.0.2 UCn 1 0 - 4 vether0 10.0.0.1 link#6 UHLch 1 1 - 3 vether0 10.0.0.2 00:00:00:00:00:02 UHLl 0 0 - 1 vether0 10.0.0.255 10.0.0.2 UHb 0 0 - 1 vether0 127/8 127.0.0.1 UGRS 0 0 32768 8 lo0 127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0 # route -T1 -n show -inet Destination Gateway Flags Refs Use Mtu Prio Iface default 10.0.0.1 UGS 0 32 - 8 vether1 10.0.0/24 10.0.0.3 UCn 1 4 - 4 vether1 10.0.0.1 00:00:00:00:00:01 UHLch 1 3 - 3 vether1 10.0.0.3 00:00:00:00:00:03 UHLl 0 0 - 1 vether1 10.0.0.255 10.0.0.3 UHb 0 0 - 1 vether1 # for if in bridge0 em2 vether{0,1}; do ifconfig $if; done bridge0: flags=41<UP,RUNNING> description: Bridge for external virtual NICs index 9 llprio 3 groups: bridge priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp designated: id 00:00:00:00:00:00 priority 0 em2 flags=3<LEARNING,DISCOVER> port 3 ifpriority 0 ifcost 0 vether0 flags=3<LEARNING,DISCOVER> port 6 ifpriority 0 ifcost 0 vether1 flags=3<LEARNING,DISCOVER> port 7 ifpriority 0 ifcost 0 Addresses (max cache: 100, timeout: 240): 00:00:00:00:00:01 em2 1 flags=0<> em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr xx:xx:xx:xx:xx:xx description: External interface #1 (integrated NIC) index 3 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) status: active vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:00:00:00:02 description: Main external IP index 6 priority 0 llprio 3 groups: vether egress media: Ethernet autoselect status: active inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 vether1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 1 mtu 1500 lladdr 00:00:00:00:00:03 description: Secondary external IP index 7 priority 0 llprio 3 groups: vether media: Ethernet autoselect status: active inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255 Footnotes: [1] My ISP implements carrier-grade NAT, with an opt-out. If I supply a MAC address, they will whitelist it and give it an external IP number over DHCP. I can do this for more than one card but they must have a unique MAC, meaning that I can not simply use an IP alias. While experimenting with the configuration, I am using my own internal DHCP server. The plan is to connect the OpenBSD server directly to the ISP. [2] It should not be necessary to have two vether, em2 should be able to serve the purpose of the primary interface. However, I think I'm hitting the bug described in the first paragraph here: http://undeadly.org/cgi?action=article&sid=20160725144108 [3] The MAC addresses in this report are obfuscated: 10.0.0.xx maps to 00:00:00:00:00:xx to aid connecting a card with its IP. The virtual interfaces has valid MAC addresses taken from retired NICs. [4] http://search.cpan.org/dist/App-Asciio/lib/App/Asciio.pm