On Mon, May 10, 2021 at 02:07:59PM +0200, Thomas Huber wrote:
> Hi David,
>
> thanks for your input.
>
> And how would you do it?
Just like I described.
> Also putting each vport(4) in its own rdomain(4) and veb(4) takes care of
> Layer2 connectivity without any pf(4) involvment?
Yes.
Two things to remember.
Firstly, vport(4) interfaces are handled specially by veb(4) and get
treated by the IP stack like a physical interface connected to a real
switch. This is different to when you add a non-vport interface to a
veb, because in that situation veb steals the interface and it is only
used to move layer 2 packets between the veb and the wire.
Secondly, veb does not run pf on packets traversing the virtual bridge
by default. However, vport interfaces are excepted from this policy
because of the thing above. This means pf will run on vport interfaces
all the time, but veb is the thing that moves the packets between the
vports.
Both of these together mean that if you do something like this:
ifconfig veb0 create
jot 5 | while read i; do
ifconfig lo$i create
ifconfig lo$i rdomain $i
ifconfig lo$i inet 127.0.0.1/8
ifconfig vport$i create
ifconfig vport$i rdomain $i
ifconfig vport$i inet 192.168.0.$i/24
ifconfig veb0 add vport$i
ifconfig lo$i up
ifconfig vport$i up
done
ifconfig veb0 up
Which looks like this:
ix# pfctl -sr
block return all
block drop quick proto udp from any to any port = 9
pass all flags S/SA
block return in on ! lo0 proto tcp from any to any port 6000:6010
block return out log proto tcp all user = 55
block return out log proto udp all user = 55
ix# ifconfig veb0
veb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
index 13 llprio 3
groups: veb
vport1 flags=3<LEARNING,DISCOVER>
port 15 ifpriority 0 ifcost 0
vport2 flags=3<LEARNING,DISCOVER>
port 17 ifpriority 0 ifcost 0
vport3 flags=3<LEARNING,DISCOVER>
port 19 ifpriority 0 ifcost 0
vport4 flags=3<LEARNING,DISCOVER>
port 21 ifpriority 0 ifcost 0
vport5 flags=3<LEARNING,DISCOVER>
port 23 ifpriority 0 ifcost 0
Addresses (max cache: 100, timeout: 240):
ix# ifconfig vport
vport1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 1
mtu 1500
lladdr fe:e1:ba:d2:4a:be
index 15 priority 0 llprio 3
groups: vport
inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
vport2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 2
mtu 1500
lladdr fe:e1:ba:d3:17:a0
index 17 priority 0 llprio 3
groups: vport
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
vport3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 3
mtu 1500
lladdr fe:e1:ba:d4:f1:a9
index 19 priority 0 llprio 3
groups: vport
inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
vport4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 4
mtu 1500
lladdr fe:e1:ba:d5:a8:78
index 21 priority 0 llprio 3
groups: vport
inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
vport5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> rdomain 5
mtu 1500
lladdr fe:e1:ba:d6:cd:44
index 23 priority 0 llprio 3
groups: vport
inet 192.168.0.5 netmask 0xffffff00 broadcast 192.168.0.255
ix# route -T1 exec ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=255 time=0.269 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=0.142 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.173 ms
^C
--- 192.168.0.2 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.142/0.195/0.269/0.054 ms
ix# jot 5 | while read i; do route -T$i exec arp -na; done
Host Ethernet Address Netif Expire Flags
192.168.0.1 fe:e1:ba:d2:4a:be vport1 permanent l
192.168.0.2 fe:e1:ba:d3:17:a0 vport1 13m30s
Host Ethernet Address Netif Expire Flags
192.168.0.1 fe:e1:ba:d2:4a:be vport2 13m30s
192.168.0.2 fe:e1:ba:d3:17:a0 vport2 permanent l
Host Ethernet Address Netif Expire Flags
192.168.0.3 fe:e1:ba:d4:f1:a9 vport3 permanent l
Host Ethernet Address Netif Expire Flags
192.168.0.4 fe:e1:ba:d5:a8:78 vport4 permanent l
Host Ethernet Address Netif Expire Flags
192.168.0.5 fe:e1:ba:d6:cd:44 vport5 permanent l
ix# ifconfig veb0
veb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
index 13 llprio 3
groups: veb
vport1 flags=3<LEARNING,DISCOVER>
port 15 ifpriority 0 ifcost 0
vport2 flags=3<LEARNING,DISCOVER>
port 17 ifpriority 0 ifcost 0
vport3 flags=3<LEARNING,DISCOVER>
port 19 ifpriority 0 ifcost 0
vport4 flags=3<LEARNING,DISCOVER>
port 21 ifpriority 0 ifcost 0
vport5 flags=3<LEARNING,DISCOVER>
port 23 ifpriority 0 ifcost 0
Addresses (max cache: 100, timeout: 240):
fe:e1:ba:d2:4a:be vport1 16 flags=0<>
fe:e1:ba:d3:17:a0 vport2 16 flags=0<>
ix#
dlg
>
> thanks
> Thomas
>
> On Mon, 10 May 2021 at 08:10, David Gwynne <[email protected]> wrote:
> >
> > Hi Thomas,
> >
> > I'd give this a go with vport(4) interfaces instead of vether(4), and
> join them all together at layer 2 by adding them to a single veb(4).
> >
> > Cheers,
> > dlg
> >
> > > On 10 May 2021, at 03:04, Thomas Huber <[email protected]> wrote:
> > >
> > > Hi misc,
> > >
> > > I wanted to tinker with the cluster manager sysutils/nomad but
> > > unfortunately I??ve no spare cluster for tinkering...
> > >
> > > So I had the idea of utilizing OpenBSDs outstanding
> > > possibilities for network isolation to create a
> > > virtual cluster on my VM at openbsd.amsterdam.
> > >
> > > I had different ideas to achieve it but nothing worked so far.
> > > So I'd describe my first approach because I think this is the
> > > most OpenBSD idiomatic one:
> > >
> > > I created 5 vether[0-4] devices, everyone in its own rdomain [0-4]
> > > and assigned every device its own inet address space 10.10.[0-4].1/24
> > >
> > > I also set the 10.10.[0-4].1 as default route in each rtable.
> > >
> > > Now I learned that pf(4) is needed to route between this 5 rdomains
> > > but after several attempts I've no clue how this could be defined.
> > >
> > > Actually I wanted rdomain 0 to work as hub for all rdomains >0.
> > > Maybe someone can hint me in the right direction....
> > >
> > > regards
> > > Thomas (host of the u2k20-hackathon, if someone remembers ;-)
> > >
> > > some further listings if my description above is unclear:
> > >
> > >
> > > ud$ ifconfig vether
> > > vether0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr fe:e1:ba:d7:cc:16
> > > index 23 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 10.10.0.1 netmask 0xff000000 broadcast 10.255.255.255
> > >
> > > vether1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 1
> mtu
> > > 1500
> > > lladdr fe:e1:ba:d8:73:32
> > > index 24 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 10.10.1.1 netmask 0xff000000 broadcast 10.255.255.255
> > >
> > > vether2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 2
> mtu
> > > 1500
> > > lladdr fe:e1:ba:d9:bd:e8
> > > index 26 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 10.10.2.1 netmask 0xff000000 broadcast 10.255.255.255
> > >
> > > vether3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 3
> mtu
> > > 1500
> > > lladdr fe:e1:ba:da:07:4d
> > > index 28 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 10.10.3.1 netmask 0xff000000 broadcast 10.255.255.255
> > >
> > > vether4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 4
> mtu
> > > 1500
> > > lladdr fe:e1:ba:db:31:c8
> > > index 30 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 10.10.4.1 netmask 0xff000000 broadcast 10.255.255.255
> > >
> > > ud$ netstat -R
> > > Rdomain 0
> > > Interfaces: lo0 vio0 enc0 pflog0 vether0
> > > Routing tables: 0 71
> > >
> > > Rdomain 1
> > > Interfaces: vether1 lo1
> > > Routing table: 1
> > >
> > > Rdomain 2
> > > Interfaces: vether2 lo2
> > > Routing table: 2
> > >
> > > Rdomain 3
> > > Interfaces: vether3 lo3
> > > Routing table: 3
> > >
> > > Rdomain 4
> > > Interfaces: vether4 lo4
> > > Routing table: 4
