I have found new warning in the management-server.log. Please see new logh
entries below. The new warning says cloudbr0 cannot be found but it exists.
Please see the output of "ip a" and "brctl show" commands at the end.
agent.log:
2024-07-20 14:41:55,510 ERROR [kvm.resource.LibvirtConnection]
(Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
connection pointer in virConnectGetVersion
2024-07-20 14:41:55,459 ERROR [utils.linux.KVMHostInfo]
(Agent-Handler-1:null) (logid:) Unable to retrieve the CPU speed from lscpu.
2024-07-21 00:39:20,270 WARN [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Threw away a startup answer because we're reconnecting.
2024-07-21 00:39:20,289 WARN [cloud.agent.Agent] (Agent-Handler-4:null)
(logid:04ac4783) Unable to send response: null
management-server:
2024-07-21 00:38:27,899 WARN [c.c.a.AlertManagerImpl]
(AgentConnectTaskPool-101:ctx-113ad44d) (logid:a2169999) alertType=[7]
dataCenterId=[1] podId=[1] clusterId=[null] message=[Incorrect Network
setup on agent, Reinitialize agent after network names are setup, details :
Can not find network: cloudbr0].
2024-07-21 00:39:21,755 ERROR [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-19:null) (logid:) Failed to send server's
CLOSE message due to socket channel's failure.
I have setup network using nmcli with following commands:
nmcli connection add type vlan con-name vlan427 ifname vlan427 vlan.parent
eth0 vlan.id 427
nmcli con up vlan427
nmcli con add ifname cloudbr0 type bridge con-name cloudbr0 autoconnect yes
nmcli con add type bridge-slave ifname vlan427 master cloudbr0 autoconnect
yes con-name br-vlan427
nmcli connection up cloudbr0
Below is output of "ip a" and "brctl show" commands:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
group default qlen 1000
link/ether 3c:7c:3f:bb:16:88 brd ff:ff:ff:ff:ff:ff
altname enp5s0
inet 149.50.116.112/24 brd 149.50.116.255 scope global noprefixroute
eth0
valid_lft forever preferred_lft forever
inet6 2a03:cfc0:8000:2e::9532:7470/64 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::3e7c:3fff:febb:1688/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: cloudbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
state DOWN group default qlen 1000
link/ether e6:5e:4e:b9:48:e3 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global noprefixroute
cloudbr0
valid_lft forever preferred_lft forever
6: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN group default qlen 1000
link/ether 4e:63:4a:80:21:74 brd ff:ff:ff:ff:ff:ff
inet 169.254.0.1/16 scope global cloud0
valid_lft forever preferred_lft forever
inet6 fe80::4c63:4aff:fe80:2174/64 scope link
valid_lft forever preferred_lft forever
# brctl show
bridge name bridge id STP enabled interfaces
cloud0 8000.000000000000 no
cloudbr0 8000.000000000000 yes
Any help would be highly appreciated.
Regards.
On Sat, Jul 20, 2024 at 9:59 PM Fariborz Navidan <mdvlinqu...@gmail.com>
wrote:
> Is this a bug in 4.18.2.1?
>
> Is there any work arround for this issue?
>
> On Sat, Jul 20, 2024 at 4:15 PM Fariborz Navidan <mdvlinqu...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I have forcefully deleted host from the UI. Then I ran
>> cloudstack-setup-agent again successfully and host automatically appeared
>> in the UI but in Alert state, It seems management server cannot connect to
>> agent. I have following error in management-server.log:
>>
>> 2024-07-20 14:26:57,500 ERROR [c.c.u.n.Link]
>> (AgentManager-SSLHandshakeHandler-5:null) (logid:) Failed to send server's
>> CLOSE message due to socket channel's failure.
>>
>> Also I have following error in agent.log:
>>
>> 2024-07-20 14:41:35,066 ERROR [utils.linux.KVMHostInfo]
>> (Agent-Handler-1:null) (logid:) Unable to retrieve the CPU speed from file
>> [/sys/devices/system/cpu/cpu0/cpufreq/base_frequency]
>> 2024-07-20 14:41:35,120 ERROR [kvm.resource.LibvirtConnection]
>> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
>> connection pointer in virConnectGetVersion
>>
>> Management server and agent are running on the same node.
>>
>> Kindly someone help me. All our systems are down for a week as our old
>> provider has stopped providing service. We need to setup this server ASAP.
>>
>> On Sat, Jul 20, 2024 at 2:36 AM Fariborz Navidan <mdvlinqu...@gmail.com>
>> wrote:
>>
>>> I have successfully ran cloudstack-setup-agent but still getting error
>>> "connection with libvirtd is broken". What should I check?
>>>
>>> Please help me.
>>>
>>> On Fri, Jul 19, 2024 at 1:47 PM Fariborz Navidan <mdvlinqu...@gmail.com>
>>> wrote:
>>>
>>>> Any idea?
>>>>
>>>> On Fri, 19 Jul 2024, 11:18 Fariborz Navidan, <mdvlinqu...@gmail.com>
>>>> wrote:
>>>>
>>>>> I have two other errors in agent.lo. Please see below:
>>>>>
>>>>>
>>>>> 2024-07-19 01:47:57,960 ERROR [utils.nio.NioClient]
>>>>> (Agent-Handler-1:null) (logid:) SSL Handshake failed while connecting to
>>>>> host: 149.50.116.112 port: 8250
>>>>> 2024-07-19 01:47:57,960 ERROR [utils.nio.NioConnection]
>>>>> (Agent-Handler-1:null) (logid:) Unable to initialize the threads.
>>>>> 2024-07-19 01:47:59,514 ERROR [kvm.resource.LibvirtConnection]
>>>>> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
>>>>> invalid
>>>>> connection pointer in virConnectGetVersion
>>>>>
>>>>> Please help me.
>>>>>
>>>>> On Fri, Jul 19, 2024 at 11:01 AM Fariborz Navidan <
>>>>> mdvlinqu...@gmail.com> wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> I'm in a situation where I really need to put CS work on my new
>>>>>> server as my old server provider has stopped providing service. I have
>>>>>> migrated all VMs as templates to the new server but still cannot deploy
>>>>>> them. All my systems are down for a long time. Should I downgrade to
>>>>>> CentOS
>>>>>> 8?
>>>>>>
>>>>>> I'm really stuck. Please help me ASAP.
>>>>>>
>>>>>> On Fri, Jul 19, 2024 at 2:15 AM Muhammad Hanis Irfan Mohd Zaid <
>>>>>> hanisirfan.w...@gmail.com> wrote:
>>>>>>
>>>>>>> Actually, can someone point us to an article or documentation of a
>>>>>>> production ready deployment method and steps for the KVM agent? The
>>>>>>> CloudStack documentation seems to be a bit hard to follow and might
>>>>>>> not
>>>>>>> work with RHEL 9 derivatives.
>>>>>>>
>>>>>>> I can try to contribute in the docs however I can.
>>>>>>>
>>>>>>> On Fri, 19 Jul 2024, 05:47 Fariborz Navidan, <mdvlinqu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > I ran the command and rebooted the server but no luck :( :( I have
>>>>>>> the same
>>>>>>> > error in the agent.log:
>>>>>>> >
>>>>>>> > 2024-07-18 23:41:56,370 ERROR [kvm.resource.LibvirtConnection]
>>>>>>> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is
>>>>>>> broken: invalid
>>>>>>> > connection pointer in virConnectGetVersion
>>>>>>> >
>>>>>>> > On Thu, Jul 18, 2024 at 11:54 PM Wei ZHOU <ustcweiz...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > systemctl mask virtqemud.socket virtqemud-ro.socket
>>>>>>> > > virtqemud-admin.socket virtqemud virtnetworkd virtstoraged
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Thursday, July 18, 2024, Fariborz Navidan <
>>>>>>> mdvlinqu...@gmail.com>
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > > > Could you please send me commands for virtqemud.socket and
>>>>>>> others?
>>>>>>> > > >
>>>>>>> > > > On Thu, Jul 18, 2024 at 11:30 PM Wei ZHOU <
>>>>>>> ustcweiz...@gmail.com>
>>>>>>> > wrote:
>>>>>>> > > >
>>>>>>> > > > > The services to mark is not libvirtd, but virtqemud.socket
>>>>>>> and etc
>>>>>>> > > > >
>>>>>>> > > > > Have you tried the same command as mentioned before ?
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > >
>>>>>>> > > > > On Thursday, July 18, 2024, Fariborz Navidan <
>>>>>>> mdvlinqu...@gmail.com>
>>>>>>> > > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > > Hi
>>>>>>> > > > > >
>>>>>>> > > > > > Masking libvirtd service caused cloustack agent not to be
>>>>>>> able to
>>>>>>> > > start
>>>>>>> > > > > or
>>>>>>> > > > > > restart. I unmasked libvirtd service and now in dead
>>>>>>> status
>>>>>>> > > restarting
>>>>>>> > > > > it
>>>>>>> > > > > > does not bring it up.
>>>>>>> > > > > >
>>>>>>> > > > > > On Thu, 18 Jul 2024, 09:34 Wei ZHOU, <
>>>>>>> ustcweiz...@gmail.com>
>>>>>>> > wrote:
>>>>>>> > > > > >
>>>>>>> > > > > > > Hi,
>>>>>>> > > > > > >
>>>>>>> > > > > > > Can you try the following command ?
>>>>>>> > > > > > >
>>>>>>> > > > > > > systemctl mask virtqemud.socket virtqemud-ro.socket
>>>>>>> > > > > > > virtqemud-admin.socket virtqemud virtnetworkd
>>>>>>> virtstoraged
>>>>>>> > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > > > > -Wei
>>>>>>> > > > > > >
>>>>>>> > > > > > > On Wed, Jul 17, 2024 at 10:23 PM Fariborz Navidan <
>>>>>>> > > > > mdvlinqu...@gmail.com
>>>>>>> > > > > > >
>>>>>>> > > > > > > wrote:
>>>>>>> > > > > > > >
>>>>>>> > > > > > > > Thank you. cloudstack-setup-agent exists but it
>>>>>>> requires me to
>>>>>>> > > put
>>>>>>> > > > > > > network
>>>>>>> > > > > > > > configuration in /etc/sysconfig/network-scripts
>>>>>>> directory which
>>>>>>> > > > seems
>>>>>>> > > > > > > that
>>>>>>> > > > > > > > no longer supported in AlmaLinux 9. I already have
>>>>>>> bridges
>>>>>>> > > created
>>>>>>> > > > > > using
>>>>>>> > > > > > > > NetworkManager via nmcli command.
>>>>>>> > > > > > > >
>>>>>>> > > > > > > > What do you advise?
>>>>>>> > > > > > > >
>>>>>>> > > > > > > > On Wed, 17 Jul 2024, 21:56 Wei ZHOU, <
>>>>>>> ustcweiz...@gmail.com>
>>>>>>> > > > wrote:
>>>>>>> > > > > > > >
>>>>>>> > > > > > > > > The command should exist on kvm hosts
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > I meant is the systemctl mask command in the pull
>>>>>>> request
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > On Wednesday, July 17, 2024, Fariborz Navidan <
>>>>>>> > > > > mdvlinqu...@gmail.com
>>>>>>> > > > > > >
>>>>>>> > > > > > > > > wrote:
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > > Thank you Wei,
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Unfortunately your given commands cannot be found.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > bash: cloudstack-setup-agent:: command not found
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Any idea?
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Thanks.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > On Wed, 17 Jul 2024, 20:33 Wei ZHOU, <
>>>>>>> > ustcweiz...@gmail.com>
>>>>>>> > > > > > wrote:
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > > Maybe the following commands help
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > >
>>>>>>> https://github.com/apache/cloudstack/pull/7114/files
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > -Wei
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > On Wed, Jul 17, 2024 at 6:30 PM Fariborz Navidan
>>>>>>> <
>>>>>>> > > > > > > > > mdvlinqu...@gmail.com>
>>>>>>> > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > To add on this, my host OS is AlmaLinux 9. Any
>>>>>>> idea on
>>>>>>> > > this
>>>>>>> > > > > > > issue?
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > Please help me.
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > On Wed, 17 Jul 2024, 18:28 Muhammad Hanis
>>>>>>> Irfan Mohd
>>>>>>> > > Zaid,
>>>>>>> > > > <
>>>>>>> > > > > > > > > > > > hanisirfan.w...@gmail.com> wrote:
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > To add on this, I've also experienced the
>>>>>>> same error
>>>>>>> > > when
>>>>>>> > > > > > > adding a
>>>>>>> > > > > > > > > > > host. I
>>>>>>> > > > > > > > > > > > > can also confirm that the status of libvirtd
>>>>>>> from
>>>>>>> > > > systemctl
>>>>>>> > > > > > > that
>>>>>>> > > > > > > > > it's
>>>>>>> > > > > > > > > > > > > running fine. I can add the host only after
>>>>>>> doing a
>>>>>>> > > > couple
>>>>>>> > > > > of
>>>>>>> > > > > > > > > reboots
>>>>>>> > > > > > > > > > > to
>>>>>>> > > > > > > > > > > > > the while the add host process is running in
>>>>>>> the UI.
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > On Wed, 17 Jul 2024, 22:03 Fariborz Navidan,
>>>>>>> <
>>>>>>> > > > > > > > > mdvlinqu...@gmail.com>
>>>>>>> > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > Yes. Output of command systemctl status
>>>>>>> libvirtd
>>>>>>> > > shows
>>>>>>> > > > > > > service is
>>>>>>> > > > > > > > > > > active
>>>>>>> > > > > > > > > > > > > > and running.
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > Please advise.
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > Thanks
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > On Wed, 17 Jul 2024, 16:48 Wei ZHOU, <
>>>>>>> > > > > > ustcweiz...@gmail.com>
>>>>>>> > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > is libvirtd running well ?
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > On Wed, Jul 17, 2024 at 3:07 PM Fariborz
>>>>>>> Navidan
>>>>>>> > <
>>>>>>> > > > > > > > > > > > > mdvlinqu...@gmail.com>
>>>>>>> > > > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > Hello everyone,
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > I'm in a situation where I need to
>>>>>>> migrate my
>>>>>>> > > > > > > infrastructure
>>>>>>> > > > > > > > > to
>>>>>>> > > > > > > > > > > new
>>>>>>> > > > > > > > > > > > > > > servers
>>>>>>> > > > > > > > > > > > > > > > as soon as possible. I have installed
>>>>>>> CS 4.18
>>>>>>> > > > > > management
>>>>>>> > > > > > > and
>>>>>>> > > > > > > > > > > agent on
>>>>>>> > > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > > same node. I have gone through the
>>>>>>> installation
>>>>>>> > > > guide
>>>>>>> > > > > > and
>>>>>>> > > > > > > > > > > configured
>>>>>>> > > > > > > > > > > > > > > > libvirtd based on documentation.
>>>>>>> However, I am
>>>>>>> > > > > unable
>>>>>>> > > > > > to
>>>>>>> > > > > > > > > add a
>>>>>>> > > > > > > > > > > KVM
>>>>>>> > > > > > > > > > > > > > host.
>>>>>>> > > > > > > > > > > > > > > > Below is the error I am getting in the
>>>>>>> > agent.log
>>>>>>> > > > > file.
>>>>>>> > > > > > > Please
>>>>>>> > > > > > > > > > > note
>>>>>>> > > > > > > > > > > > > > that I
>>>>>>> > > > > > > > > > > > > > > > have set all required parameters in
>>>>>>> libvirt
>>>>>>> > > config
>>>>>>> > > > > > file.
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > 2024-07-17 14:54:59,872 ERROR
>>>>>>> > > > > [cloud.agent.AgentShell]
>>>>>>> > > > > > > > > > > (main:null)
>>>>>>> > > > > > > > > > > > > > > (logid:)
>>>>>>> > > > > > > > > > > > > > > > Unable to start agent:
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > com.cloud.utils.exception.CloudRuntimeException:
>>>>>>> > > > > > Failed
>>>>>>> > > > > > > to
>>>>>>> > > > > > > > > > > connect
>>>>>>> > > > > > > > > > > > > > socket
>>>>>>> > > > > > > > > > > > > > > > to '/var/run/libvirt/virtqemud-sock':
>>>>>>> > Connection
>>>>>>> > > > > > refused
>>>>>>> > > > > > > > > > > > > > > > at
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > >
>>>>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
>>>>>>> > > > configure(
>>>>>>> > > > > > > > > > LibvirtComputingResource.java:1097)
>>>>>>> > > > > > > > > > > > > > > > at
>>>>>>> com.cloud.agent.Agent.<init>(
>>>>>>> > > > > > Agent.java:191)
>>>>>>> > > > > > > > > > > > > > > > at
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > >
>>>>>>> com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452)
>>>>>>> > > > > > > > > > > > > > > > at
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > >
>>>>>>> > >
>>>>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
>>>>>>> > > > > > > > > > > > > > > > at
>>>>>>> > > > > > > > > > > > >
>>>>>>> > com.cloud.agent.AgentShell.launchAgent(AgentShell.java:
>>>>>>> > > > 415)
>>>>>>> > > > > > > > > > > > > > > > at com.cloud.agent.AgentShell.
>>>>>>> > > > > > > > > > start(AgentShell.java:511)
>>>>>>> > > > > > > > > > > > > > > > at com.cloud.agent.AgentShell.
>>>>>>> > > > > > > > > > main(AgentShell.java:541)
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > It says that agent is unable to connect
>>>>>>> > libvirt.
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > Please advise.
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > Thanks.
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > >
>>>>>>> > > > > > >
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> > > >
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
