I know this is some time now. But, I remembered that my previous issue was FRR not accepting RT with different ASN generated with auto RT by the leaf switches. I've just done another test with FRR 10.6.1 on Rocky Linux 10.2. Looks like they might have changed the RT import behaviour to accept any ASN for eBGP peering. GitHub issue that is somewhat related: https://github.com/frrouting/frr/issues/20712
Anyone aware of this? On Tue, 23 Jul 2024 at 13:50, Wido den Hollander <[email protected]> wrote: > > > Op 22/07/2024 om 16:22 schreef Muhammad Hanis Irfan Mohd Zaid: > > Looks like we maybe need to go for automation route, or just continue > > with VLANs at this point. Buying another switches might be hard hahaha > > > > You can probably get this done with Frr on the HVs setting the right RT > there. > > > If you don't mind sharing, how many guest networks are there on average > > for the zones that you managed? > > > > We have >100 guest networks in these zones on average, some even more. > > But we don't use EVPN for the amount of networks we can create, we use > it because it's so much better on networking level than using > traditional VLANs. > > > May I know can the same range of VLANs be used by multiple zones? Or the > > allocation is unique per zone? > > > > On Mon, 22 Jul 2024, 21:41 Wido den Hollander, <[email protected] > > <mailto:[email protected]>> wrote: > > > > > > > > Op 22/07/2024 om 14:52 schreef Muhammad Hanis Irfan Mohd Zaid: > > > Hi Wido, > > > > > > I'm not at a point to add the zones yet. I'm currently stuck with > > EVPN > > > (as before unfortunately). For my hypervisors, of course they're > > running > > > FRR. Meanwhile, my switches are running Mellanox Onyx. My issue > > is that > > > the type 2 route is not being accepted because of different route > > targets. > > > > > > On the switch, I can of course manually set the import/export RT > > just > > > for the VNI I'm using for CS management. But for the hypervisors, > is > > > there any config that can be done on FRR so that every VNI will > > follow > > > the same RTs? For sure we don't want to configure the RTs for > every > > > single VNI (say we allocated over 10k VNI per zone). > > > > > > Oh ya, the hypervisors are running EBGP between them and the > > switches. > > > Current RTs look like this for VNI 10027 (management): > > > 1. Switches (manually configured for both import/export) > > > > > > <OUR_IRR_ASN>:10027 > > > > > > 2. Hypervisors FRR (auto) > > > > > > First = 38933:10027 > > > Second = 38932:10027 > > > > > > After looking at this blog post from ipspace > > > > > < > https://www.ipspace.net/Data_Center_BGP/EVPN_Route_Target_Considerations < > https://www.ipspace.net/Data_Center_BGP/EVPN_Route_Target_Considerations>>, > it looks like it's a bit hard to integrate the switches that we had with > FRR. Because different vendors seem to generate the auto RTs differently. > Thanks for any recommendations! > > > > > > > > > > Aha! That might explain. We use Cumulus Linux with Frr and the > Auto-RT > > there works flawlessly with the Frr on Linux hypervisors. > > > > I did try to connect Juniper MX routers with our environment and we > had > > to use manual RTs and that was a pita. > > > > So I do understand what you are running in to, but I have no other > > solution, then what we did: > > > > - Pre-define a range of 100 VNIs in Frr > > - Pre-set the RT of each of these VNIs > > - These VNIs don't have to exist at that moment > > - Use Ansible/Puppet/Salt/Chef to configure Frr on your hypervisors > > > > Wido > > > > > On Mon, 15 Jul 2024 at 18:54, Wido den Hollander <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > See my reply inline > > > > > > Op 12/07/2024 om 04:33 schreef Muhammad Hanis Irfan Mohd Zaid: > > > > I'm still trying to figure out the physical network setup > > based > > > on your > > > > talks, Wido. > > > > > > > > Just to clarify that VXLAN isolation only works for Guest > > traffic > > > types, > > > > correct? Then how would Management and Public traffic type > be > > > configured > > > > with a setup running fully L3 (BGP unnumbered etc) between > > the leaf > > > > switches and KVM hosts? > > > > > > > > > > No, you can isolate all traffic. Public/Management traffic > > just goes > > > via > > > the loopback address of the hypervisor. > > > > > > > > > > > > > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit> > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit>> > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit> > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit > < > https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit > >>> > > > > > > > > From the diagram I found here, it looks like they create > > a VLAN > > > > interface on top of a bond as their VTEP. For the public > > network, > > > > another VLAN interface is created for it. > > > > > > > > > > There is no need to create a bond0 device. Via each uplink > > (p1p1 and > > > p1p2 in this example) you create a BGP session with the ToR. > > MLAG is > > > not > > > required there. > > > > > > On the hypervisor you then have p1p1 and p1p2 over which you > > create a > > > BGP (Unnumbered) session with the ToR. > > > > > > We have then created only one bridge, cloudbr1: > > > > > > root@hv-138-a13-37:~# ip addr show dev cloudbr1 > > > 6: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue > > > state UP group default qlen 1000 > > > link/ether 12:5b:28:2a:a7:f4 brd ff:ff:ff:ff:ff:ff > > > inet 10.100.2.108/20 <http://10.100.2.108/20> > > <http://10.100.2.108/20 <http://10.100.2.108/20>> brd > > > 10.100.15.255 scope global cloudbr1 > > > valid_lft forever preferred_lft forever > > > root@hv-138-a13-37:~# > > > > > > root@hv-138-a13-37:~# bridge link|grep cloudbr1 > > > 7: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > master cloudbr1 > > > state forwarding priority 32 cost 100 > > > root@hv-138-a13-37:~# > > > > > > We have used VNI 100 as our internal POD network which is > > also carried > > > by Frr: > > > > > > hv-138-a13-37.ams02.cldin.net# > > <http://hv-138-a13-37.ams02.cldin.net#> > > > <http://hv-138-a13-37.ams02.cldin.net# > > <http://hv-138-a13-37.ams02.cldin.net#>> sh evpn vni 100 > > > VNI: 100 > > > Type: L2 > > > Tenant VRF: default > > > VxLAN interface: vxlan > > > VxLAN ifIndex: 7 > > > Local VTEP IP: 10.255.255.108 > > > Mcast group: 0.0.0.0 > > > Remote VTEPs for this VNI: > > > 10.255.255.81 flood: HER > > > 10.255.255.123 flood: HER > > > 10.255.255.11 flood: HER > > > 10.255.255.99 flood: HER > > > ... > > > ... > > > 10.255.255.36 flood: HER > > > 10.255.255.35 flood: HER > > > 10.255.255.33 flood: HER > > > 10.255.255.2 flood: HER > > > 10.255.255.106 flood: HER > > > 10.255.255.100 flood: HER > > > 10.255.255.105 flood: HER > > > 10.255.255.124 flood: HER > > > 10.255.255.127 flood: HER > > > 10.255.255.107 flood: HER > > > 10.255.255.126 flood: HER > > > 10.255.255.104 flood: HER > > > Number of MACs (local and remote) known for this VNI: 109 > > > Number of ARPs (IPv4 and IPv6, local and remote) known for > > this > > > VNI: 107 > > > Advertise-gw-macip: No > > > hv-138-a13-37.ams02.cldin.net# > > <http://hv-138-a13-37.ams02.cldin.net#> > > <http://hv-138-a13-37.ams02.cldin.net# > > <http://hv-138-a13-37.ams02.cldin.net#>> > > > > > > The management server is a VM running somewhere on a Proxmox > > > environment > > > (not really relevant) and has a NIC in VNI 100. This is how > > the Agents > > > communicate with the Management server. > > > > > > The hypervisors have a /128 and /32 on their loopback which > > they use to > > > communicate to their storage (Ceph and TrueNAS NFS): > > > > > > root@hv-138-a13-37:~# ip addr show dev lo > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > UNKNOWN > > > group default qlen 1000 > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > inet 127.0.0.1/8 <http://127.0.0.1/8> > > <http://127.0.0.1/8 <http://127.0.0.1/8>> scope host lo > > > valid_lft forever preferred_lft forever > > > inet 10.255.255.108/32 <http://10.255.255.108/32> > > <http://10.255.255.108/32 <http://10.255.255.108/32>> brd > > > 10.255.255.108 scope global lo > > > valid_lft forever preferred_lft forever > > > inet6 2a05:xxxx:xxxx:2::108/128 scope global > > > valid_lft forever preferred_lft forever > > > inet6 ::1/128 scope host > > > valid_lft forever preferred_lft forever > > > root@hv-138-a13-37:~# > > > > > > Wido > > > > > > > May I know how you achieve this Wido? > > > > > > > > On Sat, 29 Jun 2024 at 12:53, Wido den Hollander > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>> wrote: > > > > > > > > > > > > > > > > Op 24/06/2024 om 09:19 schreef Muhammad Hanis Irfan > > Mohd Zaid: > > > > > Then how would the other networks (guest and > public) be > > > configured > > > > > during the zone creation process? > > > > > > > > > > > > > Just use the defaults labels, because the rest will be > > done > > > by the > > > > hypervisor. > > > > > > > > Make sure you have selected VXLAN and you are good to > go! > > > > > > > > > For management, I've created a fixed VNI 10028. The > > management > > > > server > > > > > and the hosts can ping each other via this VNI. I'm > > using > > > > cloudbr0 tho > > > > > for the internal comm bridge name. > > > > > > > > > > I'm in need to understand how should physical > > network be > > > > configured with > > > > > the first zone creation wizard. I've added the > > modifyvxlan.sh > > > > script to > > > > > folder. And so far BGP EVPN works perfectly to the > ToR > > > switches. > > > > > > > > > > > > > Once the network is running, try to start the first VM > and > > > see what the > > > > agent does. VXLAN devices and bridges will be created. > > > > > > > > > I don't think I can join the conference because of > > financial > > > > reasons 😂 > > > > > > > > > > > > > Remotely :-) > > > > > > > > > On Mon, 24 Jun 2024, 14:22 Wido den Hollander, > > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>>> wrote: > > > > > > > > > > > > > > > > > > > > Op 21/06/2024 om 05:22 schreef Muhammad Hanis > Irfan > > > Mohd Zaid: > > > > > > On the web UI, when setting up a zone, how > > would I > > > create the > > > > > physical > > > > > > networks and assign the correct traffic > > types and > > > labels? > > > > > > > > > > > > > > > > You don't need to do that. No need for separate > > networks. > > > > > > > > > > Just make sure you have a cloudbr1 for internal > > > > commmunication, this > > > > > can > > > > > be a fixed VNI where you create a network using > > > > systemd-networkd for > > > > > example. > > > > > > > > > > > You can refer to a simple diagram I drew > below. > > > > > > > > > > > > > > > > > > > > > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link>> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link>>> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link>> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link> > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > < > https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link > >>>> > > > > > > > > > > > > So let's say based on the diagram, I create > > a physical > > > > network named > > > > > > "Management", an isolation method "VLAN", > with > > > traffic type > > > > > "MANAGEMENT" > > > > > > and label of "cloudbr0". Next, I create > another > > > physical > > > > network > > > > > named > > > > > > "Public", an isolation method "VXLAN", with > > traffic > > > type > > > > "PUBLIC" > > > > > and label > > > > > > of "lo". Lastly, I create another physical > > network > > > named > > > > "Guest", an > > > > > > isolation method "VXLAN", with traffic type > > "GUEST" and > > > > label of > > > > > "lo". > > > > > > > > > > > > Will this work? Is my understanding of > physical > > > networks > > > > correct? > > > > > > > > > > > > > > > > No, again. Not needed. The modifyvxlan.sh > > script creates > > > > bridges on the > > > > > fly and your Frrouting will see these bridges > and > > > start the VXLAN > > > > > advertisement. > > > > > > > > > > Anything specific in your config you need help > > with? > > > > > > > > > > Might be good if I gave a VXLAN masterclass > > during the > > > > conference in > > > > > November ;-) > > > > > > > > > > Wido > > > > > > > > > > > > > > > > > > > > > > > On Thu, 20 Jun 2024 at 20:21, Wido den > Hollander > > > > > <[email protected]> > > > > > > wrote: > > > > > > > > > > > >> > > > > > >> > > > > > >> Op 20/06/2024 om 11:15 schreef Alex > Mattioli: > > > > > >>> Hi Muhammad, > > > > > >>> > > > > > >>> Are you planning on using VXLAN or bridges? > > > Those are > > > > mutually > > > > > >> exclusive, with VXLAN you have a single > > VLAN interface > > > > with an > > > > > IP which is > > > > > >> the VTEP (Virtual Tunnel EndPoint) for your > > VXLAN > > > > encapsulated > > > > > traffic. > > > > > >> > > > > > >> Yes, however, keep in mind that the script > > > > 'modifyvxlan.sh' creates > > > > > >> Linux bridges on the fly, one for each VNI. > > > > > >> > > > > > >> The administrator doesn't need to do > anything > > > though, this is > > > > > all done > > > > > >> by CS. > > > > > >> > > > > > >> Just make sure you use this script: > > > > > >> > > > > > > > > > > > > > > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh > > <https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh> > > > > > < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>> > > > > > > > > > < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh> < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>>> > > > > > > > > > > > > > > < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh> < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>> < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh> < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh < > https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>>>> > > > > > >> > > > > > >> Wido > > > > > >> > > > > > >>> > > > > > >>> Cheers > > > > > >>> Alex > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> -----Original Message----- > > > > > >>> From: Muhammad Hanis Irfan Mohd Zaid > > > > <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>> > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>>> > > > > > >>> Sent: Thursday, June 20, 2024 8:59 AM > > > > > >>> To: [email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>> > > > > > >>> Cc: [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>> > > > > > >>> Subject: Physical network labels when > > using VXLAN > > > > > >>> > > > > > >>> Hi. We're trying to deploy a POC > > environment with > > > VXLAN > > > > EVPN. The > > > > > >> underlay works perfectly and the overlay > when > > > creating a > > > > bridge > > > > > for the > > > > > >> management network (cloudbr0) can ping > > without any > > > issues > > > > > between the hosts > > > > > >> and management server. > > > > > >>> > > > > > >>> Now I'm trying to figure out how the > bridges > > > should be > > > > > configured for > > > > > >> the guest and public network. The hosts are > > fully > > > running L3 > > > > > towards our > > > > > >> leaf switches. I'm clueless when trying to > > > configure the > > > > > physical networks > > > > > >> of the zone in the web UI. > > > > > >>> > > > > > >>> Any suggestions? Thanks > > > > > >> > > > > > > > > > > > > > > > > > > > > >
