Hi Wido,
I'm currently running Rocky Linux 9 for the HV.
> Why are you setting anything on cloudbr0? There is no need to create cloudbr0
> with VXLAN.
cloudbr0 is just a naming choice on my end. Is it okay for me to use something
like NetworkManager to create the bridge?
So, no need to create a VXLAN interface the as the bridge slave? Then how would
the bridge communicate through the VXLAN tunnels?
Assuming we're using VNI 10027.
lo (VTEP) <---- vxlan10027 (slave of bridge) <---- cloudbr0/cloudbr1
Shouldn't it be the same for the guest VNIs later on? E.g:
lo <---- vxlan1000 <---- vxlan1000br <---- vnet1 <---- Guest VM
^
|---------------------vnet2 <---- Guest VM
> This /20 IPv4 is used for everything within CloudStack's communcation.
When talking about subnet size, the /20 is used for a couple of pods (row of
racks) and not just a single pod, correct?
How do you calculate to allocate the range for system VMs in a pod? Considering
maximum 16 hosts per cluster.
If let's say in the future we ran out of the /20 space (more than 4k host in a
zone), we can connect new /20 subnets via a router, right?
> In *agent.properties* we have only set 'private.network.device=cloudbr1'
I actually got an issue when trying to add a new KVM host before. I got the
error "resource not found". I was able to add the host if we restart the host
OS while adding the host in the web UI.
I don't really configure anything in the *agent.properties* file before adding
the host via the web UI. Did I did it wrong? Is there any deployment procedure
that you can share with me?
> You first need to make sure that the HV can ping all the other loopback
> addresses of the Leaf and Spine switches and all HVs can connect to eachother
> via their loopback addresses.
I can assure you that the spine switches, leaf switches and HV can all ping
each other via their loopback addresses. Just a note, all the L3 routing
between the switches and the HV is configured BGP unnumbered and eBGP.
Thank you and sorry if there are unrelated questions in the thread.
Best Regards,
Hanis Irfan
-----Original Message-----
From: Wido den Hollander <w...@widodh.nl.INVALID>
Sent: Tuesday, 21 May, 2024 04:06
To: Hanis Irfan <hanis.ir...@nebula-sys.com>; users@cloudstack.apache.org
Subject: Re: BGP EVPN with CloudStack
Hi Hanis,
See my reply inline.
Op 17/05/2024 om 12:38 schreef Hanis Irfan:
> I think this is more about BGP EVPN than CloudStack but would
> appreciate anyone that could help. So basically, I’ve tried the
> Advanced Networking with VLAN isolation for my POC and now want to migrate to
> VXLAN.
>
> I would say that I’ve little to no knowledge about VXLAN in particular
> and BGP EVPN. Why don’t I use multicast instead is because our spine
> leaf switches have been already configured with basic EVPN (though not
> tested yet).
>
> We have 2 spine switches (Mellanox SN2700) and 2 leaf switches
> (Mellanox
> SN2410) running BGP unnumbered for underlay between them. Underlay
> between the hypervisor that is running FRR and the leaf switches also
> configured with BGP unnumbered.
>
> All the switches and hypervisor are assigned with 4-byte private ASN
> individually. Here is the FRR config on the hypervisor, all basic config:
>
> ```
>
> ip forwarding
>
> ipv6 forwarding
>
> interface ens3f0np0
>
> no ipv6 nd suppress-ra
>
> exit
>
> interface ens3f1np1
>
> no ipv6 nd suppress-ra
>
> exit
>
> router bgp 4200100005
>
> bgp router-id 10.XXX.118.1
>
> no bgp ebgp-requires-policy
>
> neighbor uplink peer-group
>
> neighbor uplink remote-as external
>
> neighbor ens3f0np0 interface peer-group uplink
>
> neighbor ens3f1np1 interface peer-group uplink
>
> address-family ipv4 unicast
>
> network 10.XXX.118.1/32
>
> exit-address-family
>
> address-family ipv6 unicast
>
> network 2407:XXXX:0:1::1/128
>
> neighbor uplink activate
>
> neighbor uplink soft-reconfiguration inbound
>
> exit-address-family
>
> address-family l2vpn evpn
>
> neighbor uplink activate
>
> neighbor uplink attribute-unchanged next-hop
>
> advertise-all-vni
>
> advertise-svi-ip
>
> exit-address-family
>
> ```
>
> Now I want to configure cloudbr0 bridge as the management interface
> for ACS. I’ve done it like so:
>
> ```
>
> nmcli connection add type bridge con-name cloudbr0 ifname cloudbr0 \
>
> ipv4.method manual ipv4.addresses 10.XXX.113.11/24 ipv4.gateway
> 10.XXX.113.1 \
>
> ipv4.dns 1.1.1.1,8.8.8.8 \
>
> ipv6.method manual ipv6.addresses 2407:XXXX:200:c002::11/64
> ipv6.gateway
> 2407:XXXX:200:c002::1 \
>
> ipv6.dns 2606:4700:4700::1111,2001:4860:4860::8888 \
>
> bridge.stp no ethernet.mtu 9216
>
> nmcli connection add type vxlan slave-type bridge con-name vxlan10027
> ifname vxlan10027 \
>
> id 10027 destination-port 4789 local 2407:XXXX:0:1::1 vxlan.learning
> no \
>
> master cloudbr0 ethernet.mtu 9216 dev lo
>
> nmcli connection up cloudbr0
>
> nmcli connection up vxlan10027
>
> ```
>
Why are you setting anything on cloudbr0? There is no need to create
cloudbr0 with VXLAN.
We only have created cloudbr1 (using systemd-networkd) for the POD
communcation, but that's all:
*cloudbr1.network*
[Match]
Name=cloudbr1
[Network]
LinkLocalAddressing=no
[Address]
Address=10.100.2.108/20
[Route]
Gateway=10.100.1.1
[Link]
MTUBytes=1500
*cloudbr1.netdev*
[NetDev]
Name=cloudbr1
Kind=bridge
This /20 IPv4 is used for everything within CloudStack's communcation.
In *agent.properties* we have only set 'private.network.device=cloudbr1'
> I can see that the EVPN route with MAC address of cloudbr0 can be seen
> on both leaf switches. However, I can’t ping from the hypervisor to
> its gateway (.1) which is a firewall running somewhere that’s
> connected to a switchport tagged with VLAN 27.
>
You first need to make sure that the HV can ping all the other loopback
addresses of the Leaf and Spine switches and all HVs can connect to eachother
via their loopback addresses.
Can you check that? That's not EVPN nor VXLAN, just /32 (IPv4) routing with BGP.
Wido
