Hi, the message " Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion " - is a false alarm and does NOT means any errors actually.
I can see that ACS agent sees different storage pools - namely "daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84" and "a2d455c6-68cb-303f-a7fa-287e62a5be9c" - and I don't see any explicit error message about these 2 pools (both RBD/Ceph) pools. Also I can see that the cloudstack agent says it's connected to the mgmt host - which means that all pools are in place (otherwise the agent would not connect) 1. Are you KVM hosts all green when checking in CloudStack UI (Connected/Up)? 2. You can always use virsh to list pools and see if they are there Best, On Wed, 13 Sept 2023 at 13:54, Mosharaf Hossain < mosharaf.hoss...@bol-online.com> wrote: > Hello Folks > We've recently performed an upgrade on our Cephadm cluster, transitioning > from Ceph Quiency to Reef. However, following the manual implementation of > a read balancer in the Reef cluster, we've experienced a significant > slowdown in client I/O operations within the Ceph cluster, affecting both > client bandwidth and overall cluster performance. > > This slowdown has resulted in unresponsiveness across all virtual machines > within the cluster, despite the fact that the cluster exclusively utilizes > SSD storage." > > In the CloudStack agent, we are getting libvirrt can't connect to CEPH pool > and generating an error message. > > 2023-09-13 16:57:51,660 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:) Lost connection to host: 10.10.11.61. Attempting reconnection > while we still have 1 command in progress. > 2023-09-13 16:57:51,661 INFO [utils.nio.NioClient] (Agent-Handler-4:null) > (logid:) NioClient connection closed > 2023-09-13 16:57:51,662 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:) Reconnecting to host:10.10.11.62 > 2023-09-13 16:57:51,662 INFO [utils.nio.NioClient] (Agent-Handler-4:null) > (logid:) Connecting to 10.10.11.62:8250 > 2023-09-13 16:57:51,663 INFO [utils.nio.Link] (Agent-Handler-4:null) > (logid:) Conf file found: /etc/cloudstack/agent/agent.properties > 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) > (logid:) SSL: Handshake done > 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) > (logid:) Connected to 10.10.11.62:8250 > 2023-09-13 16:57:51,815 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu". > 2023-09-13 16:57:51,836 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | > head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] > resulted in the value [2100] for CPU speed. > 2023-09-13 16:57:51,900 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > e205cf5f-ea32-46c7-ba18-d18f62772b80 (Filesystem) in libvirt > 2023-09-13 16:57:51,901 ERROR [kvm.resource.LibvirtConnection] > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid > connection pointer in virConnectGetVersion > 2023-09-13 16:57:51,903 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > e205cf5f-ea32-46c7-ba18-d18f62772b80, using it. > 2023-09-13 16:57:51,904 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool > e205cf5f-ea32-46c7-ba18-d18f62772b80 from libvirt > 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:) Process agent startup answer, agent id = 0 > 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:) Set agent id 0 > 2023-09-13 16:57:51,955 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:) Startup Response Received: agent id = 0 > 2023-09-13 16:57:52,047 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:e396a97c) Attempting to create storage > pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 (RBD) in libvirt > 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:e396a97c) Found existing defined > storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84, using it. > 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:e396a97c) Trying to fetch storage pool > daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 from libvirt > 2023-09-13 16:57:52,161 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-1:null) (logid:e396a97c) Attempting to create storage > pool a2d455c6-68cb-303f-a7fa-287e62a5be9c (RBD) in libvirt > 2023-09-13 16:57:52,163 WARN [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-1:null) (logid:e396a97c) Storage pool > a2d455c6-68cb-303f-a7fa-287e62a5be9c was not found running in libvirt. Need > to create it. > 2023-09-13 16:57:52,164 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-1:null) (logid:e396a97c) Didn't find an existing > storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c by UUID, checking for > pools with duplicate paths > 2023-09-13 16:57:56,780 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:) Connected to the host: 10.10.11.62 > ^C > > > Kindly guide us to move forward. > > > > Regards > Mosharaf Hossain > Manager, Product Development > IT Division > > Bangladesh Export Import Company Ltd. > > Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh > > Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 > > Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: > www.bol-online.com > < > https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A > > > -- Andrija Panić
