See my reply inline
Op 12/07/2024 om 04:33 schreef Muhammad Hanis Irfan Mohd Zaid:
I'm still trying to figure out the physical network setup based on your
talks, Wido.
Just to clarify that VXLAN isolation only works for Guest traffic types,
correct? Then how would Management and Public traffic type be configured
with a setup running fully L3 (BGP unnumbered etc) between the leaf
switches and KVM hosts?
No, you can isolate all traffic. Public/Management traffic just goes via
the loopback address of the hypervisor.
https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit
<https://docs.google.com/drawings/d/1oPWU5p_wUd9UPhXGZg7j4acu5XYPbkLWzmihi6Qbwl8/edit>
From the diagram I found here, it looks like they create a VLAN
interface on top of a bond as their VTEP. For the public network,
another VLAN interface is created for it.
There is no need to create a bond0 device. Via each uplink (p1p1 and
p1p2 in this example) you create a BGP session with the ToR. MLAG is not
required there.
On the hypervisor you then have p1p1 and p1p2 over which you create a
BGP (Unnumbered) session with the ToR.
We have then created only one bridge, cloudbr1:
root@hv-138-a13-37:~# ip addr show dev cloudbr1
6: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 12:5b:28:2a:a7:f4 brd ff:ff:ff:ff:ff:ff
inet 10.100.2.108/20 brd 10.100.15.255 scope global cloudbr1
valid_lft forever preferred_lft forever
root@hv-138-a13-37:~#
root@hv-138-a13-37:~# bridge link|grep cloudbr1
7: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr1
state forwarding priority 32 cost 100
root@hv-138-a13-37:~#
We have used VNI 100 as our internal POD network which is also carried
by Frr:
hv-138-a13-37.ams02.cldin.net# sh evpn vni 100
VNI: 100
Type: L2
Tenant VRF: default
VxLAN interface: vxlan
VxLAN ifIndex: 7
Local VTEP IP: 10.255.255.108
Mcast group: 0.0.0.0
Remote VTEPs for this VNI:
10.255.255.81 flood: HER
10.255.255.123 flood: HER
10.255.255.11 flood: HER
10.255.255.99 flood: HER
...
...
10.255.255.36 flood: HER
10.255.255.35 flood: HER
10.255.255.33 flood: HER
10.255.255.2 flood: HER
10.255.255.106 flood: HER
10.255.255.100 flood: HER
10.255.255.105 flood: HER
10.255.255.124 flood: HER
10.255.255.127 flood: HER
10.255.255.107 flood: HER
10.255.255.126 flood: HER
10.255.255.104 flood: HER
Number of MACs (local and remote) known for this VNI: 109
Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 107
Advertise-gw-macip: No
hv-138-a13-37.ams02.cldin.net#
The management server is a VM running somewhere on a Proxmox environment
(not really relevant) and has a NIC in VNI 100. This is how the Agents
communicate with the Management server.
The hypervisors have a /128 and /32 on their loopback which they use to
communicate to their storage (Ceph and TrueNAS NFS):
root@hv-138-a13-37:~# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.255.255.108/32 brd 10.255.255.108 scope global lo
valid_lft forever preferred_lft forever
inet6 2a05:xxxx:xxxx:2::108/128 scope global
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
root@hv-138-a13-37:~#
Wido
May I know how you achieve this Wido?
On Sat, 29 Jun 2024 at 12:53, Wido den Hollander <w...@widodh.nl
<mailto:w...@widodh.nl>> wrote:
Op 24/06/2024 om 09:19 schreef Muhammad Hanis Irfan Mohd Zaid:
> Then how would the other networks (guest and public) be configured
> during the zone creation process?
>
Just use the defaults labels, because the rest will be done by the
hypervisor.
Make sure you have selected VXLAN and you are good to go!
> For management, I've created a fixed VNI 10028. The management
server
> and the hosts can ping each other via this VNI. I'm using
cloudbr0 tho
> for the internal comm bridge name.
>
> I'm in need to understand how should physical network be
configured with
> the first zone creation wizard. I've added the modifyvxlan.sh
script to
> folder. And so far BGP EVPN works perfectly to the ToR switches.
>
Once the network is running, try to start the first VM and see what the
agent does. VXLAN devices and bridges will be created.
> I don't think I can join the conference because of financial
reasons 😂
>
Remotely :-)
> On Mon, 24 Jun 2024, 14:22 Wido den Hollander, <w...@widodh.nl
<mailto:w...@widodh.nl>
> <mailto:w...@widodh.nl <mailto:w...@widodh.nl>>> wrote:
>
>
>
> Op 21/06/2024 om 05:22 schreef Muhammad Hanis Irfan Mohd Zaid:
> > On the web UI, when setting up a zone, how would I create the
> physical
> > networks and assign the correct traffic types and labels?
> >
>
> You don't need to do that. No need for separate networks.
>
> Just make sure you have a cloudbr1 for internal
commmunication, this
> can
> be a fixed VNI where you create a network using
systemd-networkd for
> example.
>
> > You can refer to a simple diagram I drew below.
> >
>
https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link
<https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link>
<https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link
<https://drive.google.com/file/d/1_xGUxEu-U2mJltdIj94CMK0s4zAH-Ret/view?usp=drive_link>>
> >
> > So let's say based on the diagram, I create a physical
network named
> > "Management", an isolation method "VLAN", with traffic type
> "MANAGEMENT"
> > and label of "cloudbr0". Next, I create another physical
network
> named
> > "Public", an isolation method "VXLAN", with traffic type
"PUBLIC"
> and label
> > of "lo". Lastly, I create another physical network named
"Guest", an
> > isolation method "VXLAN", with traffic type "GUEST" and
label of
> "lo".
> >
> > Will this work? Is my understanding of physical networks
correct?
> >
>
> No, again. Not needed. The modifyvxlan.sh script creates
bridges on the
> fly and your Frrouting will see these bridges and start the VXLAN
> advertisement.
>
> Anything specific in your config you need help with?
>
> Might be good if I gave a VXLAN masterclass during the
conference in
> November ;-)
>
> Wido
>
> >
> >
> > On Thu, 20 Jun 2024 at 20:21, Wido den Hollander
> <w...@widodh.nl.invalid>
> > wrote:
> >
> >>
> >>
> >> Op 20/06/2024 om 11:15 schreef Alex Mattioli:
> >>> Hi Muhammad,
> >>>
> >>> Are you planning on using VXLAN or bridges? Those are
mutually
> >> exclusive, with VXLAN you have a single VLAN interface
with an
> IP which is
> >> the VTEP (Virtual Tunnel EndPoint) for your VXLAN
encapsulated
> traffic.
> >>
> >> Yes, however, keep in mind that the script
'modifyvxlan.sh' creates
> >> Linux bridges on the fly, one for each VNI.
> >>
> >> The administrator doesn't need to do anything though, this is
> all done
> >> by CS.
> >>
> >> Just make sure you use this script:
> >>
>
https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh
<https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>
>
<https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh <https://download.cloudstack.org/tools/scripts/vxlan/modifyvxlan.sh>>
> >>
> >> Wido
> >>
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Muhammad Hanis Irfan Mohd Zaid
<hanisirfan.w...@gmail.com <mailto:hanisirfan.w...@gmail.com>
> <mailto:hanisirfan.w...@gmail.com
<mailto:hanisirfan.w...@gmail.com>>>
> >>> Sent: Thursday, June 20, 2024 8:59 AM
> >>> To: users@cloudstack.apache.org
<mailto:users@cloudstack.apache.org>
> <mailto:users@cloudstack.apache.org
<mailto:users@cloudstack.apache.org>>
> >>> Cc: w...@widodh.nl <mailto:w...@widodh.nl>
<mailto:w...@widodh.nl <mailto:w...@widodh.nl>>
> >>> Subject: Physical network labels when using VXLAN
> >>>
> >>> Hi. We're trying to deploy a POC environment with VXLAN
EVPN. The
> >> underlay works perfectly and the overlay when creating a
bridge
> for the
> >> management network (cloudbr0) can ping without any issues
> between the hosts
> >> and management server.
> >>>
> >>> Now I'm trying to figure out how the bridges should be
> configured for
> >> the guest and public network. The hosts are fully running L3
> towards our
> >> leaf switches. I'm clueless when trying to configure the
> physical networks
> >> of the zone in the web UI.
> >>>
> >>> Any suggestions? Thanks
> >>
> >
>
