Re: Stretch/L2VPN between two datacenters
On Fri, Jan 20, 2023 at 12:48:33AM +0100, Lars Bonnesen wrote: > I have been fighting with this for a while now, trying to make it work > reading man pages... But it does not work as I want it to work. tcpdump can > see a lot of arp requests on bridge0, egre0, vlan172 - but nothing seems to > get to wg0. This is my ifconfig filtered for public IPs: > > lo0: flags=8049 mtu 32768 > index 5 priority 0 llprio 3 > groups: lo > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > inet 127.0.0.1 netmask 0xff00 > vmx0: flags=8843 mtu 1500 > lladdr 00:50:56:b4:a5:ab > index 1 priority 0 llprio 3 > groups: egress > media: Ethernet autoselect (10GbaseT) > status: active > inet qq.ww.ee.rr netmask 0xff00 broadcast ee.rr.tt.yy > vmx1: flags=8802 mtu 1500 > lladdr 00:50:56:b4:0d:26 > index 2 priority 0 llprio 3 > media: Ethernet autoselect (10GbaseT) > status: active > vmx2: flags=8b43 mtu > 1600 > lladdr 00:50:56:b4:ef:b4 > description: corp > index 3 priority 0 llprio 3 > media: Ethernet autoselect (10GbaseT) > status: active > enc0: flags=0<> > index 4 priority 0 llprio 3 > groups: enc > status: active > pflog0: flags=141 mtu 33136 > index 6 priority 0 llprio 3 > groups: pflog > lo1: flags=8008 rdomain 1 mtu 32768 > index 8 priority 0 llprio 3 > groups: lo > wg0: flags=80c3 mtu 1420 > index 9 priority 0 llprio 3 > wgport 51820 > wgpubkey GIWFxfaaxt1VmURRvEtJkG/mZQgVLNtHuEtPa6vt/kM= > wgpeer MSS4DjJjPtp9DsTpMbNQ1ict6jEx07DICfipOpnOUR4= > wgendpoint aa.bb.cc.dd 51820 > tx: 1690108800, rx: 2934539600 > last handshake: x seconds ago > wgaip 192.168.5.1/32 > groups: wg > inet 192.168.5.2 netmask 0xff00 broadcast 192.168.5.255 > egre0: flags=8943 mtu 1500 > lladdr fe:e1:ba:d0:31:5b > index 14 priority 0 llprio 3 > encap: vnetid 172 txprio 0 rxprio packet > groups: egre > tunnel: inet 172.24.90.92 --> 172.24.90.91 ttl 64 nodf > vlan172: flags=8943 mtu 1500 > lladdr 00:50:56:b4:ef:b4 > index 24 priority 0 llprio 3 > encap: vnetid 172 parent vmx2 txprio packet rxprio outer > groups: vlan > media: Ethernet autoselect (10GbaseT) > status: active > inet 172.24.90.94 netmask 0xff00 broadcast 172.24.90.255 > bridge0: flags=41 mtu 1500 > index 25 llprio 3 > groups: bridge > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp > vlan172 flags=3 > port 24 ifpriority 0 ifcost 0 > egre0 flags=3 > port 14 ifpriority 0 ifcost 0 > vmx2 flags=3 > port 3 ifpriority 0 ifcost 0 > > On the other end the ifconfig is similar > > wg0 is working. I can ping 192.168.5.1 from 192.168.5.2 and visa versa. > > 172.24.90.0/24 (vlan172) is the network that I want to strech... and is > presented to the obsd as vmx2 connected to an access port on a switch > > Can anyone guide me in the right direction, thx? sure. the first thing i would like to point out is that you want wg to protect the egre traffic, so you should set the egre tunnel addresses to the IPs you've set up inside wg. right now you're trying to use IPs from the network you're trying to stretch as the tunnel enspoints, which, as the gre manpage says, is not going to work. the gre manpage also has points out that egre also needs the net.inet.gre.allow sysctl set 1. so, this is a good step: # sysctl net.inet.gre.allow=1 # ifconfig egre0 tunnel 192.168.5.2 192.168.5.1 # ifconfig egre0 up you can do that in /etc/sysctl.conf and /etc/hostname.egre0 too. the second thing to clean up is how vlan172 is wired up to the tunnel. while bridge(4) can do it, i think there are better options now so i would tear that down. if this box does not need to interact with traffic inside the vlan, ie, its only job is to stretch the traffic, then you can use tpmr(4) to plug them together: # ifconfig tpmr0 create # ifconfig tpmr0 add vlan172 add egre0 # ifconfig tpmr0 up if this box does need to do stuff on the 172.24.90.0/24 network, then veb(4) and vport(4) are better: # ifconfig vport0 create # ifconfig vport0 inet 172.24.90.94/24 # ifconfig vport0 up # ifconfig veb0 create # ifconfig veb0 add vport0 vlan172 egre0 # ifconfig veb0 up that should work, or should be a big step closer. > > Regards, Lars. > > On Wed, Jan 4, 2023 at 7:24 AM Lars Bonnesen > wrote: > > > Thanks for your replies. It has been Xmas and I have been delayed, but I > > have now read up upon it. I am going for the tpmr(4). We are going to > > replicate a lot of live data from Site1 to Site2, and my experiences with > > OpenVPN is that it is great, but not high
Re: Stretch/L2VPN between two datacenters
As far as I remember, I could only get gre to work over wg. Anything that is ethernet (like tmpr, etherip, etc) didn't work. My wild guess is that packet overhead is becoming to big as there is a lot of encapsulation. Also, wg interfaces do not have same features like normal interfaces so for example you can't add them to bridges and the like. On Friday, January 20, 2023 at 08:48:58 a.m. GMT+9, Lars Bonnesen wrote: I have been fighting with this for a while now, trying to make it work reading man pages... But it does not work as I want it to work. tcpdump can see a lot of arp requests on bridge0, egre0, vlan172 - but nothing seems to get to wg0. This is my ifconfig filtered for public IPs: lo0: flags=8049 mtu 32768 index 5 priority 0 llprio 3 groups: lo inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff00 vmx0: flags=8843 mtu 1500 lladdr 00:50:56:b4:a5:ab index 1 priority 0 llprio 3 groups: egress media: Ethernet autoselect (10GbaseT) status: active inet qq.ww.ee.rr netmask 0xff00 broadcast ee.rr.tt.yy vmx1: flags=8802 mtu 1500 lladdr 00:50:56:b4:0d:26 index 2 priority 0 llprio 3 media: Ethernet autoselect (10GbaseT) status: active vmx2: flags=8b43 mtu 1600 lladdr 00:50:56:b4:ef:b4 description: corp index 3 priority 0 llprio 3 media: Ethernet autoselect (10GbaseT) status: active enc0: flags=0<> index 4 priority 0 llprio 3 groups: enc status: active pflog0: flags=141 mtu 33136 index 6 priority 0 llprio 3 groups: pflog lo1: flags=8008 rdomain 1 mtu 32768 index 8 priority 0 llprio 3 groups: lo wg0: flags=80c3 mtu 1420 index 9 priority 0 llprio 3 wgport 51820 wgpubkey GIWFxfaaxt1VmURRvEtJkG/mZQgVLNtHuEtPa6vt/kM= wgpeer MSS4DjJjPtp9DsTpMbNQ1ict6jEx07DICfipOpnOUR4= wgendpoint aa.bb.cc.dd 51820 tx: 1690108800, rx: 2934539600 last handshake: x seconds ago wgaip 192.168.5.1/32 groups: wg inet 192.168.5.2 netmask 0xff00 broadcast 192.168.5.255 egre0: flags=8943 mtu 1500 lladdr fe:e1:ba:d0:31:5b index 14 priority 0 llprio 3 encap: vnetid 172 txprio 0 rxprio packet groups: egre tunnel: inet 172.24.90.92 --> 172.24.90.91 ttl 64 nodf vlan172: flags=8943 mtu 1500 lladdr 00:50:56:b4:ef:b4 index 24 priority 0 llprio 3 encap: vnetid 172 parent vmx2 txprio packet rxprio outer groups: vlan media: Ethernet autoselect (10GbaseT) status: active inet 172.24.90.94 netmask 0xff00 broadcast 172.24.90.255 bridge0: flags=41 mtu 1500 index 25 llprio 3 groups: bridge priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp vlan172 flags=3 port 24 ifpriority 0 ifcost 0 egre0 flags=3 port 14 ifpriority 0 ifcost 0 vmx2 flags=3 port 3 ifpriority 0 ifcost 0 On the other end the ifconfig is similar wg0 is working. I can ping 192.168.5.1 from 192.168.5.2 and visa versa. 172.24.90.0/24 (vlan172) is the network that I want to strech... and is presented to the obsd as vmx2 connected to an access port on a switch Can anyone guide me in the right direction, thx? Regards, Lars. On Wed, Jan 4, 2023 at 7:24 AM Lars Bonnesen wrote: > Thanks for your replies. It has been Xmas and I have been delayed, but I > have now read up upon it. I am going for the tpmr(4). We are going to > replicate a lot of live data from Site1 to Site2, and my experiences with > OpenVPN is that it is great, but not high performing. So I have established > a WireGuard connection with one OBSD on each site, and I am planning to > tunnel tpmr through this - I guess that tpmr itself is not encrypted in any > way? > > Regards, Lars. > > On Fri, Dec 16, 2022 at 4:30 PM deich...@placebonol.com < > deich...@placebonol.com> wrote: > >> I've run L2 over an IPsec tunnel using egre (gre(4)) and bridge (bridge >> (4)) to connect systems in different locations together. >> >> This was done before David Gwynne created tpmr(4). I've been to lazy to >> reimplement my current configuration. >> >> 73 >> diana >> >
Re: Stretch/L2VPN between two datacenters
I have been fighting with this for a while now, trying to make it work reading man pages... But it does not work as I want it to work. tcpdump can see a lot of arp requests on bridge0, egre0, vlan172 - but nothing seems to get to wg0. This is my ifconfig filtered for public IPs: lo0: flags=8049 mtu 32768 index 5 priority 0 llprio 3 groups: lo inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff00 vmx0: flags=8843 mtu 1500 lladdr 00:50:56:b4:a5:ab index 1 priority 0 llprio 3 groups: egress media: Ethernet autoselect (10GbaseT) status: active inet qq.ww.ee.rr netmask 0xff00 broadcast ee.rr.tt.yy vmx1: flags=8802 mtu 1500 lladdr 00:50:56:b4:0d:26 index 2 priority 0 llprio 3 media: Ethernet autoselect (10GbaseT) status: active vmx2: flags=8b43 mtu 1600 lladdr 00:50:56:b4:ef:b4 description: corp index 3 priority 0 llprio 3 media: Ethernet autoselect (10GbaseT) status: active enc0: flags=0<> index 4 priority 0 llprio 3 groups: enc status: active pflog0: flags=141 mtu 33136 index 6 priority 0 llprio 3 groups: pflog lo1: flags=8008 rdomain 1 mtu 32768 index 8 priority 0 llprio 3 groups: lo wg0: flags=80c3 mtu 1420 index 9 priority 0 llprio 3 wgport 51820 wgpubkey GIWFxfaaxt1VmURRvEtJkG/mZQgVLNtHuEtPa6vt/kM= wgpeer MSS4DjJjPtp9DsTpMbNQ1ict6jEx07DICfipOpnOUR4= wgendpoint aa.bb.cc.dd 51820 tx: 1690108800, rx: 2934539600 last handshake: x seconds ago wgaip 192.168.5.1/32 groups: wg inet 192.168.5.2 netmask 0xff00 broadcast 192.168.5.255 egre0: flags=8943 mtu 1500 lladdr fe:e1:ba:d0:31:5b index 14 priority 0 llprio 3 encap: vnetid 172 txprio 0 rxprio packet groups: egre tunnel: inet 172.24.90.92 --> 172.24.90.91 ttl 64 nodf vlan172: flags=8943 mtu 1500 lladdr 00:50:56:b4:ef:b4 index 24 priority 0 llprio 3 encap: vnetid 172 parent vmx2 txprio packet rxprio outer groups: vlan media: Ethernet autoselect (10GbaseT) status: active inet 172.24.90.94 netmask 0xff00 broadcast 172.24.90.255 bridge0: flags=41 mtu 1500 index 25 llprio 3 groups: bridge priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp vlan172 flags=3 port 24 ifpriority 0 ifcost 0 egre0 flags=3 port 14 ifpriority 0 ifcost 0 vmx2 flags=3 port 3 ifpriority 0 ifcost 0 On the other end the ifconfig is similar wg0 is working. I can ping 192.168.5.1 from 192.168.5.2 and visa versa. 172.24.90.0/24 (vlan172) is the network that I want to strech... and is presented to the obsd as vmx2 connected to an access port on a switch Can anyone guide me in the right direction, thx? Regards, Lars. On Wed, Jan 4, 2023 at 7:24 AM Lars Bonnesen wrote: > Thanks for your replies. It has been Xmas and I have been delayed, but I > have now read up upon it. I am going for the tpmr(4). We are going to > replicate a lot of live data from Site1 to Site2, and my experiences with > OpenVPN is that it is great, but not high performing. So I have established > a WireGuard connection with one OBSD on each site, and I am planning to > tunnel tpmr through this - I guess that tpmr itself is not encrypted in any > way? > > Regards, Lars. > > On Fri, Dec 16, 2022 at 4:30 PM deich...@placebonol.com < > deich...@placebonol.com> wrote: > >> I've run L2 over an IPsec tunnel using egre (gre(4)) and bridge (bridge >> (4)) to connect systems in different locations together. >> >> This was done before David Gwynne created tpmr(4). I've been to lazy to >> reimplement my current configuration. >> >> 73 >> diana >> >
Re: Stretch/L2VPN between two datacenters
On 2023-01-04, Lars Bonnesen wrote: > Thanks for your replies. It has been Xmas and I have been delayed, but I > have now read up upon it. I am going for the tpmr(4). We are going to > replicate a lot of live data from Site1 to Site2, and my experiences with > OpenVPN is that it is great, but not high performing. So I have established > a WireGuard connection with one OBSD on each site, and I am planning to > tunnel tpmr through this - I guess that tpmr itself is not encrypted in any > way? Trying to give an idea of how it fits together as from this I think you're missing one piece. tpmr(4) itself is a special type of bridge device; you add two layer2 (ethernet-like) interfaces to it and it shuffles network frames between them. So there's no *concept* of encryption in tpmr(4), it isn't at that layer of processing. wg(4) is a layer3 type of interface, not layer2, you can't attach it directly to tpmr. For your use-case you need a layer2 tunnel interface, for example egre(4), etherip(4), eoip(4) etc. [Choosing between them: If you only have one such tunnel between a pair of IP addresses then etherip is sufficient. If you have multiple (say, several ethernet segments via individual eth-tpmr-tunnel combinations) then you'll need one of the other types which also has a tunnel id to distinguish them; packet size overheads are slightly higher for those.] That tunnel interface is configured with endpoint IP addresses. Those can either be normal externally routable addresses (either unencrypted or protected by IPsec in 'transport' mode i.e. each packet just has the encr/auth overhead and not an extra IP header), *or* you can use internal addresses provided by a separate VPN tunnel (wg, openvpn, IPsec in 'tunnel' mode, ...); the latter will likely have more bytes overhead per packet. Check the manual for the tunnel interface that you use as there are sysctls that must be set. And when testing, make sure to use a variety of packet sizes to make sure they all make it through correctly and you don't have MTU blackhole-related problems. -- Please keep replies on the mailing list.
Re: Stretch/L2VPN between two datacenters
Thanks for your replies. It has been Xmas and I have been delayed, but I have now read up upon it. I am going for the tpmr(4). We are going to replicate a lot of live data from Site1 to Site2, and my experiences with OpenVPN is that it is great, but not high performing. So I have established a WireGuard connection with one OBSD on each site, and I am planning to tunnel tpmr through this - I guess that tpmr itself is not encrypted in any way? Regards, Lars. On Fri, Dec 16, 2022 at 4:30 PM deich...@placebonol.com < deich...@placebonol.com> wrote: > I've run L2 over an IPsec tunnel using egre (gre(4)) and bridge (bridge > (4)) to connect systems in different locations together. > > This was done before David Gwynne created tpmr(4). I've been to lazy to > reimplement my current configuration. > > 73 > diana >
Re: Stretch/L2VPN between two datacenters
I've run L2 over an IPsec tunnel using egre (gre(4)) and bridge (bridge (4)) to connect systems in different locations together. This was done before David Gwynne created tpmr(4). I've been to lazy to reimplement my current configuration. 73 diana
Re: Stretch/L2VPN between two datacenters
On Fri, Dec 16, 2022 at 11:39:02AM +0100, Hrvoje Popovski wrote: > On 16.12.2022. 11:33, Lars Bonnesen wrote: > > We are about to migrate VM's from one datacenter to another and the VMware > > L2VPN we are using for this is simply not stable for some reason that we > > cannot figure out why. > > > > I have used GRE-tunneling before on a software router that I actually > > cannot remember the name of now, but if OpenBSD can do the same, I would > > rather deploy one OpenBSD on each site and have that task handled by > > OpenBSD. > > > > Each site should be able to use the other site gateway over a > > L2-network.and VMs on each site should be able to see each other as they > > are on the same LAN > > > > Where to start reading? > > > man tpmr yes. i wrote tpmr for this exact situation. i wanted to connect switches in different datacentres together over tunnels (etherip in my case) while i was migrating from one site to the other. i was considering calling the driver xconnect or xcon, but went with tpmr because i was reading the ethernet bridge specification at the time and it talks about a special type of bridge called a two port mac relay.
Re: Stretch/L2VPN between two datacenters
Dnia 16 grudnia 2022 11:33:55 CET, Lars Bonnesen napisał/a: >We are about to migrate VM's from one datacenter to another and the VMware >L2VPN we are using for this is simply not stable for some reason that we >cannot figure out why. > >I have used GRE-tunneling before on a software router that I actually >cannot remember the name of now, but if OpenBSD can do the same, I would >rather deploy one OpenBSD on each site and have that task handled by >OpenBSD. > >Each site should be able to use the other site gateway over a >L2-network.and VMs on each site should be able to see each other as they >are on the same LAN > >Where to start reading? > >Regards, Lars. You could use vxlan and bridge it to physical interface. If you need to also encrypt that traffic, then vxlan over ipsec Or openvpn in layer 2 (tap mode), then bridge tap0 on both ends to physical interface Regards -- Łukasz Moskała
Re: Stretch/L2VPN between two datacenters
On 16.12.2022. 11:33, Lars Bonnesen wrote: > We are about to migrate VM's from one datacenter to another and the VMware > L2VPN we are using for this is simply not stable for some reason that we > cannot figure out why. > > I have used GRE-tunneling before on a software router that I actually > cannot remember the name of now, but if OpenBSD can do the same, I would > rather deploy one OpenBSD on each site and have that task handled by > OpenBSD. > > Each site should be able to use the other site gateway over a > L2-network.and VMs on each site should be able to see each other as they > are on the same LAN > > Where to start reading? man tpmr
Stretch/L2VPN between two datacenters
We are about to migrate VM's from one datacenter to another and the VMware L2VPN we are using for this is simply not stable for some reason that we cannot figure out why. I have used GRE-tunneling before on a software router that I actually cannot remember the name of now, but if OpenBSD can do the same, I would rather deploy one OpenBSD on each site and have that task handled by OpenBSD. Each site should be able to use the other site gateway over a L2-network.and VMs on each site should be able to see each other as they are on the same LAN Where to start reading? Regards, Lars.
