**### Regarding the VXLAN**
i found the Cloudstack does not link the Interface Loopback to the VXLAN
interface,
I did manually and now it propagates the routes
**### On the AGENT side:**
Not sure if this is just related to Ubuntu systems:
there is an error on the management server every time i reload it:
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults
DEBUG:root:Failed to execute:update-rc.d: **### error: unable to read
/etc/init.d/cloudstack-management**
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
DEBUG:root:Failed to execute:× cloudstack-management.service - CloudStack
Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service;
enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-02 13:13:48 UTC; 1
day 8h ago
Main PID: 537731 (code=exited, status=143)
CPU: 27min 54.476s
Obviously there is no such file: Should I create One?
ls -al /etc/init.d/
cryptdisks grub-common iwpmd nfs-common
plymouth rng-tools-debian ufw
apparmor cryptdisks-early hwclock.sh
keyboard-setup.sh ntp plymouth-log rpcbind
unattended-upgrades
apport dbus ipmievd kmod
open-iscsi procps rsync uuidd
console-setup.sh dpdk irqbalance lvm2
openipmi quota ssh x11-common
cron frr iscsid lvm2-lvmpolld
openvswitch-switch
**### On the management side**, only when i restart the agent:
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Starting Uncomplicated firewall...
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Finished Uncomplicated firewall.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopping Uncomplicated firewall...
Apr 03 23:30:14 kvmvcompatl2001 ufw-init[56782]: Skip stopping firewall: ufw
(not enabled)
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: ufw.service: Deactivated
successfully.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:**### Failed to execute:update-rc.d: error: unable to read
/etc/init.d/cloudstack-agent**
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:○ cloudstack-agent.service - CloudStack Agent
Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled;
vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Thu 2025-04-03 23:30:17
UTC; 31s ago
Docs: http://www.cloudstack.org/
Main PID: 56558 (code=exited, status=1/FAILURE)
CPU: 14.716s
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: Stopped CloudStack Agent.
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: cloudstack-agent.service: Consumed
14.716s CPU time.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start
**### Regarding the VXLAN** i found the Cloudstack does not link the Interface
Loopback to the VXLAN interface,
I did manually and now it propagates the routes
> On Apr 4, 2025, at 7:07 PM, Chi vediamo <tatay...@gmail.com> wrote:
>
> I created this one with the pictures,
>
> https://github.com/apache/cloudstack/issues/10659
>
>
> Pearl opened somethis similar:
> https://github.com/apache/cloudstack/issues/9920
>
> Which seems related to one Pearl was working, but if this is for 4.21 is
> unrealistic from the time line perspective for me !!! and Wido have this
> working Something is really messed up !
>
>
> Additionally: Issue 4) The VNI's I Choose for Guest are showing as VLANs,
> Wido what did you put on guest Vlan Range and Which Format Or nothing at all.
>
> Wido can you share which version of cloudstack are you running?
>
>
> Tata Y.
>
>
>
>
>> On Apr 4, 2025, at 6:27 AM, Chi vediamo <tatay...@gmail.com> wrote:
>>
>>
>> The cloud0 was added by the system, i did not created that.
>> I do not have a cloudbr0
>>
>> what does it show for your in your agent.properties: public.network.device
>> and guest.network.properties
>>
>>
>> My agent.properties
>> private.network.device=cloudbr1
>> guest.network.device=cloudbr1
>> hypervisor.type=kvm
>> public.network.device=cloudbr1
>> host=10.1.1.1@static
>> guid=579fafc9-5706-31f1-ae98-9f041ac90c80
>> workers=5
>>
>> the vxlan1000 I configured in the UI for the PUblic network and the
>> brvx-1000 that was added by Cloudstack, I did not do it manually
>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master
>>>> brvx-1000 state UNKNOWN group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state
>>>> UP group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>
>>
>> I created a vxlan100 for the management and the management IPs are being
>> propagates
>>
>> my VNi
>> ~# vtysh -c 'sh evpn vni'
>> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs
>> Tenant VRF
>> 1000 L2 vxlan1000 2 0 0
>> default <<< Added by Cloudstack Public but no Public IPs advertised
>>
>> 100 L2 vxlan100 7 11 4
>> default <<< created manually for the initial clourbr1 and management sec
>> storage
>> 222 L2 vxlan222 10 20 9
>> default <<< Separate vxlan for primary storage, its working
>>
>> My concerns are: Is normal the UI to show agent state gray for the 2 initial
>> SYSTEM-VMs the proxy and the secndarystorage
>>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being
>>>>> routed Advertised.***
>>>>> v-154-VM <public-IP-3> <Private-IP-2> <control-ip-169.x.x.x>
>>>>> [State-Running] [Agent State-Gray]
>>>>> s-285-VM <public-IP-2> <Private-IP-3> <control-ip-169.x.x.y>
>>>>> [State-Running] [Agent State-Gray]
>>
>> And where are my public IPs that should be added to the vxlan1000 ?
>>
>>
>> Also the UI does not allow me to add Public IPv6 address not even using
>> square brackets []
>>
>> I can send snapshots but apache.org removes them.
>>
>> I know it should be a little detail but I do not know what I am missing, or
>> what is the normal status for the UI buttons.
>>
>>
>> Can any of you send me a snapshot of the UI configuration for the ZONe
>>
>> and for your zone for example Attached a PDF with the snapshots of mine.
>> the network storage I did not add any, cloudstack added itself to the
>> cloudbr1
>>
>> Also How do I Assign Public IPv6 Addresses ?
>>
>> thank you
>>
>> Tata Y.
>>
>> see PDF attached
>>
>> <TataY-Cloudstack-VXLAN-UI-2025-V1.pdf>
>>
>>
>>
>>> On Apr 4, 2025, at 5:00 AM, Wido den Hollander <w...@widodh.nl> wrote:
>>>
>>> Let me post a couple of outputs of a hypervisor we run:
>>>
>>>
>>> 9: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>> master cloudbr1 state UNKNOWN group default qlen 1000
>>> link/ether 8e:7d:67:a7:c9:72 brd ff:ff:ff:ff:ff:ff
>>> inet6 fe80::8c7d:67ff:fea7:c972/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 10: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>> state UP group default qlen 1000
>>> link/ether 1e:11:99:1e:6d:46 brd ff:ff:ff:ff:ff:ff
>>> inet 10.100.2.2/20 scope global cloudbr1
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> As you can see there is a /20 IPv4 allocated there. The agent.properties
>>> shows:
>>>
>>> host=10.100.1.6@static
>>> control.cidr=169.254.240.0/20
>>> private.network.device=cloudbr1
>>> guid=44e94fad-4aec-414e-8cac-cea3264dd431
>>>
>>> Those are the relevant ones in this case.
>>>
>>>
>>> 6: enxb8599f206ba6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq
>>> state UP group default qlen 1000
>>> link/ether b8:59:9f:20:6b:a6 brd ff:ff:ff:ff:ff:ff
>>> inet6 2a05:xxxx:601:117:ba59:9fff:fe20:6ba6/64 scope global deprecated
>>> dynamic mngtmpaddr
>>> valid_lft 3592sec preferred_lft 0sec
>>> inet6 fe80::ba59:9fff:fe20:6ba6/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 7: enxb8599f206ba7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq
>>> state UP group default qlen 1000
>>> link/ether b8:59:9f:20:6b:a7 brd ff:ff:ff:ff:ff:ff
>>> inet6 2a05:xxxx:601:197:ba59:9fff:fe20:6ba7/64 scope global deprecated
>>> dynamic mngtmpaddr
>>> valid_lft 3594sec preferred_lft 0sec
>>> inet6 fe80::ba59:9fff:fe20:6ba7/64 scope link
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
>>> default qlen 1000
>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>> inet 127.0.0.1/8 scope host lo
>>> valid_lft forever preferred_lft forever
>>> inet 10.255.255.10/32 brd 10.255.255.10 scope global lo
>>> valid_lft forever preferred_lft forever
>>> inet6 2a05:xxxx:601:2::10/128 scope global
>>> valid_lft forever preferred_lft forever
>>> inet6 ::1/128 scope host
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> With two BGP sessions over these two uplinks:
>>>
>>> L2VPN EVPN Summary:
>>> BGP router identifier 10.255.255.10, local AS number 4200100124 VRF default
>>> vrf-id 0
>>> BGP table version 0
>>> RIB entries 4341, using 543 KiB of memory
>>> Peers 2, using 48 KiB of memory
>>> Peer groups 1, using 64 bytes of memory
>>>
>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
>>> State/PfxRcd PfxSnt Desc
>>> enxb8599f206ba6 4 4200100002 6027297 5839388 39828 0 0
>>> 28w1d10h 6635 6706 N/A
>>> enxb8599f206ba7 4 4200100002 6035334 5835598 39828 0 0
>>> 28w1d10h 6635 6706 N/A
>>>
>>>
>>>
>>> That's all! There is no cloudbr0 on the hypervisor at all.
>>>
>>> hv-138-a05-zz.xxxx.cldin.net# sh evpn vni
>>> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs
>>> Tenant VRF
>>> 539 L2 vxlan539 19 8 15 default
>>> 100 L2 vxlan100 107 106 103 default
>>> 528 L2 vxlan528 92 8 36 default
>>> 507 L2 vxlan507 43 8 22 default
>>> ....
>>> ....
>>> hv-138-a05-zz.xxxx.cldin.net#
>>>
>>>
>>> Wido
>>>
>>> On 4/4/25 04:07, Chi vediamo wrote:
>>>> I decided to reinstall everything clean again, drop the tables completely
>>>> and start all over again.
>>>> The management IPs I see them routed. even the ones for the s-VM and v-
>>>> VM. I see them on the transport switches.
>>>> The Public VXLAN is created, the Public IP address I am not able to see
>>>> them anywhere !
>>>> FROM THE UI: If I run a diagnostic Ping to the public IP for the
>>>> secondarystoragevm or for the consoleproxy - success !!!
>>>> Why or how is not being routed ? How can I troubleshoot this?
>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master
>>>> brvx-1000 state UNKNOWN group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state
>>>> UP group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> Any tip will be appreciated.
>>>> Thank you Teams.
>>>> Tata Y.
>>>>> On Apr 3, 2025, at 12:47 PM, Chi vediamo <tatay...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Using [Network 1] Only for all three [Management], [guest],[public]
>>>>> Tagged as VXLAN and with cloudbr1
>>>>>
>>>>> *- **Should I use a separate Network for each ? ? ***
>>>>>
>>>>> What should I check, in addition to the official manual: I reread again
>>>>> the instructions and saw videos
>>>>> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown
>>>>> both VMs are down'
>>>>> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
>>>>> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent
>>>>> State] green while the Vms are in starting mode with a bue dot.
>>>>> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>>>>>
>>>>> - Management servers are two physical separated servers. all Ubuntu 22.04
>>>>> - DB is mysql running on a separate server. Mysql 8
>>>>> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
>>>>> - Cloudstack 4.20.0.0
>>>>> - Primary Storage CEPH RDB
>>>>> - Secondary storage EMC NFS
>>>>>
>>>>>
>>>>> ***Issue 1) I can't add a second Management server, No idea why !!** *
>>>>> Thank you Wei, as soon i shut the second Management server the
>>>>> initial VMs proxy and secondary storage come up
>>>>>
>>>>> ***Issue 2) Management Network is Tagged as VXLAN, the UI shows
>>>>> vlan://untagged - "no typo vlan" every-time i use the UI to change to
>>>>> vxlan://untagged** *
>>>>> I did try vxlan://untagged parameter Got the message: Vlan parameter
>>>>> : vxlan://untagged is not in valid format
>>>>> I did try vxlan:// Got the
>>>>> message: Unable to convert to broadcast URI: vxlan://
>>>>> I did try vxlan:// Got the
>>>>> message: Vlan parameter : vxlan://100 is not in valid format'
>>>>>
>>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being
>>>>> routed Advertised.***
>>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>>>>> [State-Running] [Agent State-Gray]
>>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>>>> [State-Running] [Agent State-Gray]
>>>>> If both [State] and [Agent State] need to be green, What other
>>>>> debug can I enable ??
>>>>>
>>>>> **NOTE** The VXLAN parameter for the Public network was taken
>>>>> vxlan://1000, But are not routed.
>>>>>
>>>>> I have to use a Single management server as everything messes UP when I
>>>>> run a second Management server with the Same Database.
>>>>>
>>>>> I reiterate The initial VMs are running:
>>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>>>>
>>>>>
>>>>>
>>>>> I do not see the Public IP anywhere, not even advertised on vni1000 or
>>>>> any other, the public VNI is there was created by the system, but nothing
>>>>> is there - No IP addresses advertised
>>>>>
>>>>> I used VLAN/VNI vxlan://1000
>>>>>
>>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000
>>>>> state forwarding priority
>>>>> There are 2 VNETS assigned
>>>>>
>>>>> virsh domiflist s-285-VM
>>>>> Interface Type Source Model MAC
>>>>> --------------------------------------------------------------
>>>>> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
>>>>> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
>>>>> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>>>>>
>>>>> virsh domifaddr s-285-VM
>>>>> Name MAC address Protocol Address
>>>>> -------------------------------------------------------------------------------
>>>>> ----- Nothing here -----
>>>>>
>>>>>
>>>>> the command arp -n, does not show Public IP addresses neither the
>>>>> management IP addresses, just the private IP addresses and the
>>>>> <control-ip-169.x.x.x>
>>>>>
>>>>> sh bgp evpn route vni 1000
>>>>> BGP table version is 1, local router ID is 10.1.1.14
>>>>> Status codes: s suppressed, d damped, h history, * valid, > best, i -
>>>>> internal
>>>>> Origin codes: i - IGP, e - EGP, ? - incomplete
>>>>> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
>>>>> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
>>>>> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
>>>>> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
>>>>> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>>>>>
>>>>> Network Next Hop Metric LocPrf Weight Path
>>>>> *> [3]:[0]:[32]:[10.1.1.14]
>>>>> 10.1.1.14(hv1)
>>>>> 32768 i
>>>>> ET:8 RT:5641:1000
>>>>>
>>>>> Displayed 1 prefixes (1 paths)
>>>>>
>>>>> At the Agent side only WARNINGS I am not using uefi.
>>>>>
>>>>> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource]
>>>>> (main:[]) (logid:) uefi properties file not found due to: Unable to find
>>>>> file uefi.properties.
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: [])
>>>>> (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ]
>>>>> encountered the error: [127].
>>>>> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest-
>>>>> Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: [])
>>>>> (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v
>>>>> --version ] failed.
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: [])
>>>>> (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ]
>>>>> encountered the error: [127].
>>>>>
>>>>>
>>>>> AT the management the rror is:
>>>>> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl]
>>>>> (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing
>>>>> ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
>>>>> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl]
>>>>> (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from :
>>>>> https://10.1.1.1:9090/clusterservice, method : null, exception :
>>>>> [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException:
>>>>> Certificate for <10.1.1.1> doesn't match any of the subject alternative
>>>>> names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660,
>>>>> cs1.myinternaldomain.int, cloudstack.internal]
>>>>>
>>>>>
>>>>> Tata Y.
>>>
>>
>
