Signed-off-by: Hannes Laimer <[email protected]> --- pvesdn.adoc | 212 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 212 insertions(+)
diff --git a/pvesdn.adoc b/pvesdn.adoc index 267ce97..e68a889 100644 --- a/pvesdn.adoc +++ b/pvesdn.adoc @@ -1424,6 +1424,218 @@ can reply back. If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 and 10.0.2.0/24 in this example), will be announced dynamically. +[[pvesdn_setup_example_evpn_ipv6]] +EVPN with IPv6 Setup Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The example assumes a cluster with three nodes (node1, node2, node3) with P2P +connections between `node1<->node2` and `node2<->node3`. Additionally we'll +also need an `/48` IPv6 prefix routed to the exit node. + +We will start by creating an OpenFabric which will be the underlay for our EVPN +zone. Then we'll set up a VNet with two subnets, one with SLAAC and a `/96` one +with RA enabled and statically assigned IPs. + +.Prefixes +. to `node2` routed `/48`: `2001:db8:abcd::/48` +. `/64` subnet with SLAAC: `2001:db8:abcd:1::/64` +. `/96` subnet without SLAAC: `2001:db8:abcd:2::/96` + +.IPv6 forwarding +This has to be enabled on all nodes. +---- +sysctl -w net.ipv6.conf.all.forwarding=1 +---- + +NOTE: The underlay will be IPv4 based, the overlay IPv6. + +Setup `OpenFabric` +^^^^^^^^^^^^^^^^^^ + +. Create the fabric with `10.0.0.0/24` as its IPv4 prefix and `underlay` as its + name, we can leave the defaults for the rest +. Add the three nodes to the fabric, +.. `node1` with `10.0.0.1`, select P2P NIC between `node1<->node2` +.. `node2` with `10.0.0.2`, select both P2P NICs between `node1<->node2` and + `node2<->node3` +.. `node3` with `10.0.0.3`, select P2P NIC between `node2<->node3` +. Apply the changes + +The resulting config in `/etc/pve/sdn/fabrics.cfg` should look something like +this: + +---- +openfabric_fabric: underlay + ip_prefix 10.0.0.0/24 + +openfabric_node: underlay_node1 + interfaces name=ens19 + ip 10.0.0.1 + +openfabric_node: underlay_node2 + interfaces name=ens20 + interfaces name=ens19 + ip 10.0.0.2 + +openfabric_node: underlay_node3 + interfaces name=ens19 + ip 10.0.0.3 +---- + +Each node should now also be able to reach all of the others, this can quickly +be checked with a + +---- +ping 10.0.0.X +---- + +With `ip -4 r` you should also be able to see the respective `onlink` routes. + +---- +10.0.0.2 nhid 28 via 10.0.0.2 dev ens19 proto openfabric src 10.0.0.1 metric 20 onlink +10.0.0.3 nhid 28 via 10.0.0.2 dev ens19 proto openfabric src 10.0.0.1 metric 20 onlink +---- + +If this works we can continue. + +Create controller +^^^^^^^^^^^^^^^^^ + +Create an EVPN controller, keep `65000` as `ASN#` and select the `underlay` +fabric we have just created. For this we'll name the controller `v6ctl`. Then +apply the changes. + +The relevant config here is `/etc/pve/sdn/controllers.cfg`, and should look +like this: + +---- +evpn: v6ctl + asn 65000 + fabric underlay +---- + +Setup the zone +^^^^^^^^^^^^^^ + +Create a new EVPN zone, select the `v6ctl` as its controller, set the +`VRF_VXLAN Tag` to `6` and name it `v6zone`. As exit node select the node that +the `/48` prefix is routed to. Then apply the changes. + +The relevant config here is `/etc/pve/sdn/zones.cfg`, and after applying should +look something like this: + +---- +evpn: v6zone + controller v6ctl + vrf-vxlan 6 + exitnodes node2 + ipam pve + mac BC:24:11:F2:72:B8 +---- + + +Create the VNets +^^^^^^^^^^^^^^^^ + +.The bigger `/64` with SLAAC +Create a new VNet, name it `net64`, select `v6zone` as zone and set `64` as tag. + +.The smaller `/96` without SLAAC and static IPs +Create a new VNet, name it `net96`, select `v6zone` as zone and set `96` as tag. + +After applying `/etc/pve/sdn/vnets.cfg` should contain: + +---- +vnet: net64 + zone v6zone + tag 64 + +vnet: net96 + zone v6zone + tag 96 +---- + + +Setup the subnets +^^^^^^^^^^^^^^^^^ + +First we'll configure the `/64` subnet in `net64` with SLAAC: + +. Select the `net64` vnet +. Create subnet with `2001:db8:abcd:1::/64` and `2001:db8:abcd:1::1/64` as + gateway +. Under "IPv6 Options" check both `Enable RA` and `SLAAC (A)` + +Then configure the `/96` subnet in `net96`: + +. Select the `net96` vnet +. Create subnet with `2001:db8:abcd:2::/96` and `2001:db8:abcd:2::1/96` as + gateway +. Under "IPv6 Options" only check `Enable RA` + +NOTE: With RAs enabled a default gateway is advertised so we don't have to +specify a gateway explicitly for guests with a static IP. It'll also allow us to +advertise a DNS server, which can be handy because usually we'd need a whole DHCP +server for that. Whether the advertised DNS is used is somewhat client +dependent, as sometimes this option is ignored. + +The relevant config for these is `/etc/pve/sdn/subnets.cfg`, and after applying +these changes should contain + +---- +subnet: v6zone-2001:db8:abcd:1::-64 + vnet net64 + gateway 2001:db8:abcd:1::1 + nd-ra-enable 1 + nd-ra-flag-auto 1 + +subnet: v6zone-2001:db8:abcd:2::-96 + vnet net96 + gateway 2001:db8:abcd:2::1 + nd-ra-enable 1 +---- + +Result +^^^^^^ + +Now we can create two guests to test this, we'll use Debian CTs here, but any +guest will do. + +. configure first guest +.. Select `net64` as bridge during +.. leave the IPv4 section `static` and the IP field empty +.. for IPv6 select `SLAAC` +. configure second guest +.. select `net96` as bridge during +.. leave the IPv4 section `static` and the IP field empty +.. for IPv6 select `static` and set `2001:db8:abcd:2::200/96` as `IPv6` + +NOTE: We can leave `gateway` empty because we have RAs enabled on `net96`, +setting it is also not a problem, just not needed. + +Both guests should have a default route via the gateway's link-local address, +`ip -6 r` + +---- +default via fe80::be24:11ff:fef2:72b8 dev eth0 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium +---- + +This address will match the link-local address on the host's `net64` (or `net96`) interfaces, `ip a show net64`. + +---- +7: net64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_v6zone state UP group default qlen 1000 + link/ether bc:24:11:f2:72:b8 brd ff:ff:ff:ff:ff:ff + inet6 2001:db8:abcd:1::1/64 scope global + valid_lft forever preferred_lft forever + inet6 fe80::be24:11ff:fef2:72b8/64 scope link proto kernel_ll + valid_lft forever preferred_lft forever +---- + +Before the guests first traffic, `ip -6 neigh show vrf vrf_v6zone` will only +contain an entry for the guests link-local address. After the first `ping +2a00:1450:4001:818::2003`, also the guests GUA should have an entry. + + [[pvesdn_notes]] Notes -- 2.47.3
