If I use a single Management server it works:
Question with the IP addresses assigned:
the initial VMs are running:
v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
I do not see the Public IP anywhere, not even advertised on vni1000 or any
other, the public VNI is there was created by the system, but nothing is there
- No IP addresses advertised
I used VLAN/VNI vxlan://1000
vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state
forwarding priority
There are 2 VNETS assigned
virsh domiflist s-285-VM
Interface Type Source Model MAC
--------------------------------------------------------------
vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
virsh domifaddr s-285-VM
Name MAC address Protocol Address
-------------------------------------------------------------------------------
----- Nothing here -----
the command arp -n, does not showPublic IPaddresses, just the private IP
addresses and the <control-ip-169.x.x.x>
sh bgp evpn route vni 1000
BGP table version is 1, local router ID is 10.1.1.14
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
*> [3]:[0]:[32]:[10.1.1.14]
10.1.1.14(hv1)
32768 i
ET:8 RT:5641:1000
Displayed 1 prefixes (1 paths)
Please Let me know if I should see the Public IP addresses advertised for this
VMs or any IP.
Tata Y.
> On Apr 1, 2025, at 9:35 AM, Wido den Hollander <[email protected]> wrote:
>
>
>
> Op 01-04-2025 om 13:16 schreef Chi vediamo:
>> Hello Wido,
>> Sorry I meant I only have cloudbr1, there is no cloudbr0 on the hypervisor
>> On the management server on the UI interface
>> when the VNI/VXLAN for Public is requested then I should enter vxlan://777
>> in your case ?
>
> Yes, for the first network you create for example.
>
>> And for the VNI/VXLAN Range for guest leave it blank ? or ALso use a full
>> range including VXLAN200 in your case vxlan://1 through vxlan://1000 ?
>
> I don't think we ever used a VNI range in our setups. We just manually add
> the VNI for every network we create.
>
> Wido
>
>> Tata Y.
>>> On Apr 1, 2025, at 7:06 AM, Wido den Hollander <[email protected]> wrote:
>>>
>>>
>>>
>>> Op 01-04-2025 om 12:16 schreef Chi vediamo:
>>>> Thank you Wido for jumping in, as always
>>>> So far I only Have cloudbr1 as several VXLAN e-mail trails you indicated.
>>>> Then I need to create the cloudbr0 with another VXLAN in addition to
>>>> cloudbr1 ? cloudbr1 should have the management per the video you posted
>>>> right?
>>>
>>> In our case we use a static VNI underneath cloudbr1 where it communicates
>>> with the other hosts, the mgmt server and the secondary storage (NFS). This
>>> is a /22 IPv4 we have allocated for that VNI.
>>>
>>> vxlan200 device which we created using networkd and is a slave of cloudbr1
>>>
>>> cloudbr0 is never used as the VXLAN script creates brvx-XXX devices on the
>>> fly.
>>>
>>>> Hope you recall, should I include in the VNI range the initial VNI/VXLANs
>>>> i am using for cloudbr1 and cloudbr0
>>>
>>> Not that I'm aware of. We simply specify the broadcast domain when creating
>>> a new network: vxlan://777 for example to use VNI 777 for that network.
>>>
>>> This will result in brvx-777 being created on a host when needed there and
>>> the VM being attached to that bridge.
>>>
>>> Wido
>>>
>>>> Thank you so much.
>>>> Tata Y.
>>>>> On Apr 1, 2025, at 6:00 AM, Wido den Hollander <[email protected]> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Op 29-03-2025 om 15:42 schreef Chi vediamo:
>>>>>> Hello Wei,
>>>>>> The VXLAN with EPVN is poorly documented, then I need all tthe guidance
>>>>>> as possible from the community.
>>>>>
>>>>> There is no silver bullet here as each EVPN+VXLAN setup will differ from
>>>>> the other :-)
>>>>>
>>>>>> Wido, please kindly allow me to use some of your multiple answers to
>>>>>> several members of the community as reference. As there is really not
>>>>>> much info.
>>>>>> * Wei what you suggesting I did try that, already, I did try several
>>>>>> options so far. The cloudstack-Agent keeps restarting itself, I copy
>>>>>> the logs in the e-mail.
>>>>>> *
>>>>>> * Saw countless videos, and read the manual, the Manual only states in
>>>>>> summary "VXLAN with EVPN is complex", there is nothing else.
>>>>>> *
>>>>>> * Manual states use and modify "modifyvxlan-evpn.sh" which does not
>>>>>> exists. Using the one form github. modifyvxlan.sh
>>>>>> *
>>>>>
>>>>> Yes, the Github one is good. It will be in packages from 4.21
>>>>>
>>>>>> * The manual indicates: showing a picture of Management,public and
>>>>>> storage on vlans. and indicates - The only thing you have to do is
>>>>>> configure the physical Network to use VXLAN as the isolation method
>>>>>> for Guest Network - We do not have VLANS at all.
>>>>>> *
>>>>>
>>>>> Don't worry about different networks, VXLAN creates it's own bridges on
>>>>> the fly.
>>>>>
>>>>>> * Per one of the e-mails iIread long time ago from Wido to another
>>>>>> member of the community: we do left the UI settings by default. That
>>>>>> didn't work. And we start experimenting, and none seems to be working.
>>>>>
>>>>> It has been a long time since I've set it up. But you should just have
>>>>> cloudbr0 and cloudbr1, nothing else.
>>>>>
>>>>>> Per Wido, we only need cloudbr1, with no VLANs.
>>>>>> I only need to know on the cludstack UI, What do we point to the
>>>>>> Cloudbr1 My understanding is This is going to be used for Migrations ad
>>>>>> that.
>>>>>> 4.20.0 does not allow me to add Management alone, nor the Guest Alone,
>>>>>> when creating the first zone.
>>>>>> I need Public and Management at least. Right? Or please indicate which
>>>>>> ones. I try to add only the Guest one, it also fails
>>>>>> Additionally, seems like the Cloudstack is not smart enough to see what
>>>>>> VNI VXLANs are already configured, then I have to provide a vni range
>>>>>> for the Guest.
>>>>>> Although the modifyvxlan.sh script should take care of it - that is
>>>>>> based on reading the script code.
>>>>>> So far we do have VXLAN-EVPN reachability and working.
>>>>>> I think we are almost there:
>>>>>> * I try adding Guest Only during the zone creation, we do have
>>>>>> management, public and storage on VXLANS, all are reacheable by
>>>>>> routing, and the cloudstack-agent server reach them and have
>>>>>> cloudbr1 created.
>>>>>> o Should I have only Managment and Guest on the cloudbr1?
>>>>>> * Need help Guidance with The cloudstack-agent keeps Bouncing comes up
>>>>>> and immediately down, and even reinstalled removing all files. Still
>>>>>> the same.
>>>>>> * Any example of the initial agent.properties: Or do not touch the
>>>>>> agent.properties
>>>>>
>>>>> Only set the network devices as you've done below. The guid should match
>>>>> what the hypervisor thinks the UUID of this Agent is. Again, long time
>>>>> ago since I've set this up for the first time.
>>>>>
>>>>> Wido
>>>>>
>>>>>> My agent.properties I have modified specifically 3 parameters:
>>>>>> host=<MAMGT-IP1>,<MGMT-IP2>
>>>>>> guest.cpu.mode=host-model
>>>>>> private.network.device=cloudbr1
>>>>>> Thank you for your guidance.
>>>>>> Thank you
>>>>>> Tata Y.
>>>>>>> On Mar 28, 2025, at 1:43 PM, Wei ZHOU <[email protected]> wrote:
>>>>>>>
>>>>>>> I think you do not need to change agent.properties
>>>>>>> Instead, you need to set the network traffic label of each traffic
>>>>>>> type, for example cloudbr1.
>>>>>>>
>>>>>>> By default, management traffic uses cloudbr0, public and guest traffic
>>>>>>> use cloudbr1, which might not be what you use.
>>>>>>>
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>> On Friday, March 28, 2025, Chi vediamo <[email protected]
>>>>>>> <mailto:[email protected]>> wrote:
>>>>>>>
>>>>>>> Hello Apache Cloudstack users,
>>>>>>>
>>>>>>>
>>>>>>> After attempted some options, I am recurring to what should work.
>>>>>>> As there is no info in the documentation about this deployment.
>>>>>>>
>>>>>>> I did replaced the agent server with private.network.device=cloudbr1
>>>>>>>
>>>>>>> I have my VXLAN 1 on Cloudbr1 that reaches management and SS, We
>>>>>>> only have cloudbr1 created, No other bridge, and everything is BGP.
>>>>>>>
>>>>>>>
>>>>>>> Then at the UI creating the ZONE I got errors:
>>>>>>>
>>>>>>> We have had tryed the following Options:
>>>>>>>
>>>>>>> *Option 1: *No tags
>>>>>>> I put only [Physical Network 1] [VXLAN] Management, throws a popup
>>>>>>> error " All required traffic types should be added and with
>>>>>>> multiple physical networks each traffic type should have a label"
>>>>>>>
>>>>>>> *Option 2:* No Tags, Throws and Error " All required traffic types
>>>>>>> should be added and with multiple physical networks each traffic
>>>>>>> type should have a label"
>>>>>>> I put :
>>>>>>> [Physical Network 1] [VXLAN] Management
>>>>>>> [Physical Network 2] [VXLAN] Public
>>>>>>> [Physical Network 3] [VXLAN] Guest
>>>>>>> [Physical Network 4] [VXLAN] Storage
>>>>>>>
>>>>>>>
>>>>>>> *Option 3: *
>>>>>>> I put :
>>>>>>> [Physical Network 1] [VXLAN] Management, Public, Guest, it goes to
>>>>>>> the next window - which is Great
>>>>>>> The the system throws exceptions like:
>>>>>>> *Something went wrong; please correct the following:*
>>>>>>> *Unable to add the host: Cannot find the server resources at
>>>>>>> http://<KVM-IP>*
>>>>>>>
>>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>>> *
>>>>>>> *
>>>>>>> *Management logs shows:*
>>>>>>> Succeeded to import certificate in the keystore for agent on the
>>>>>>> KVM host: <ip>. Agent secured and trusted
>>>>>>>
>>>>>>> *
>>>>>>> *
>>>>>>> *The Agent Logs:*
>>>>>>> **
>>>>>>> 2025-03-28 16:48:42,550 DEBUG [commons.beanutils.ConvertUtils]
>>>>>>> (main:[]) (logid:) Convert string '<IP-MGMT1>,<IP-MGMT2>' to class
>>>>>>> 'java.lang.String'
>>>>>>> 2025-03-28 16:48:42,550 DEBUG
>>>>>>> [beanutils.converters.StringConverter] (main:[]) (logid:)
>>>>>>> Converting 'String' value '<IP-MGMT1>,<IP-MGMT2>' to type 'String'
>>>>>>> 2025-03-28 16:48:42,551 DEBUG
>>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:)
>>>>>>> Property [guid] has empty or null value. Using default value [null].
>>>>>>> 2025-03-28 16:48:42,552 DEBUG
>>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:)
>>>>>>> Property [developer] has empty or null value. Using default value
>>>>>>> [false].
>>>>>>> 2025-03-28 16:48:53,897 INFO [cloud.agent.AgentShell] (main:[])
>>>>>>> (logid:) Agent started
>>>>>>> 2025-03-28 16:48:53,902 INFO [cloud.agent.AgentShell] (main:[])
>>>>>>> (logid:) Implementation Version is 4.20.0.0
>>>>>>> 2025-03-28 16:48:53,904 INFO [cloud.agent.AgentShell] (main:[])
>>>>>>> (logid:) agent.properties found at /etc/cloudstack/agent/
>>>>>>> agent.properties
>>>>>>> 2025-03-28 16:48:53,913 DEBUG
>>>>>>> [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:)
>>>>>>> Property [host] was altered. Now using the value
>>>>>>> [10.23.123.61,10.23.123.62].
>>>>>>> 2025-03-28 16:48:53,923 DEBUG
>>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:) Setting
>>>>>>> default value: false
>>>>>>> 2025-03-28 16:48:53,923 DEBUG
>>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:)
>>>>>>> Converting 'Boolean' value 'false' to type 'Boolean'
>>>>>>> 2025-03-28 16:48:53,923 DEBUG
>>>>>>> [beanutils.converters.BooleanConverter] (main:[]) (logid:) No
>>>>>>> conversion required, value is already a Boolean
>>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter]
>>>>>>> (main:[]) (logid:) Setting default value: 0
>>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter]
>>>>>>> (main:[]) (logid:) Converting 'Integer' value '0' to type 'Byte'
>>>>>>> 2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter]
>>>>>>> (main:[]) (logid:) Converted to Byte value '0'
>>>>>>> 2025-03-28 16:48:53,926 DEBUG
>>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:)
>>>>>>> Setting default value:
>>>>>>> 2025-03-28 16:48:53,926 DEBUG
>>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:)
>>>>>>> Converting 'Character' value ' ' to type 'Character'
>>>>>>> 2025-03-28 16:48:53,926 DEBUG
>>>>>>> [beanutils.converters.CharacterConverter] (main:[]) (logid:)
>>>>>>> No conversion required, value is already a Character
>>>>>>> 2025-03-28 16:48:53,926 DEBUG
>>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:) Setting
>>>>>>> default value: 0
>>>>>>> 2025-03-28 16:48:53,926 DEBUG
>>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:)
>>>>>>> Converting 'Integer' value '0' to type 'Double'
>>>>>>> 2025-03-28 16:48:53,927 DEBUG
>>>>>>> [beanutils.converters.DoubleConverter] (main:[]) (logid:)
>>>>>>> Converted to Double value '0.0'
>>>>>>> 2025-03-28 16:48:53,927 DEBUG
>>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:) Setting
>>>>>>> default value: 0
>>>>>>> 2025-03-28 16:48:53,927 DEBUG
>>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:)
>>>>>>> Converting 'Integer' value '0' to type 'Float'
>>>>>>> 2025-03-28 16:48:53,927 DEBUG
>>>>>>> [beanutils.converters.FloatConverter] (main:[]) (logid:)
>>>>>>> Converted to Float value '0.0'
>>>>>>> 2025-03-28 16:48:53,928 DEBUG
>>>>>>> [beanutils.converters.IntegerConverter] (main:[]) (logid:) Setting
>>>>>>> default value: 0
>>>>>>> *
>>>>>>> *
>>>>>>> AGENT out :
>>>>>>> Unable to find the guid
>>>>>>> Unable to find the guid
>>>>>>> Unable to find the guid
>>>>>>> Unable to find the guid
>>>>>>> Unable to find the guid
>>>>>>>
>>>>>>> There is nothing there
>>>>>>>
>>>>>>> And the agent come up, and immediately goes down!!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Tata Y.
>>>>>>>
>>>>>
>>>
>
