Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage." In the CloudStack agent, we are getting libvirrt can't connect to CEPH pool and generating an error message. 2023-09-13 16:57:51,660 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:) Lost connection to host: 10.10.11.61. Attempting reconnection while we still have 1 command in progress. 2023-09-13 16:57:51,661 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:) NioClient connection closed 2023-09-13 16:57:51,662 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:) Reconnecting to host:10.10.11.62 2023-09-13 16:57:51,662 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:) Connecting to 10.10.11.62:8250 2023-09-13 16:57:51,663 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:) SSL: Handshake done 2023-09-13 16:57:51,779 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:) Connected to 10.10.11.62:8250 2023-09-13 16:57:51,815 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu". 2023-09-13 16:57:51,836 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [2100] for CPU speed. 2023-09-13 16:57:51,900 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool e205cf5f-ea32-46c7-ba18-d18f62772b80 (Filesystem) in libvirt 2023-09-13 16:57:51,901 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion 2023-09-13 16:57:51,903 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool e205cf5f-ea32-46c7-ba18-d18f62772b80, using it. 2023-09-13 16:57:51,904 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool e205cf5f-ea32-46c7-ba18-d18f62772b80 from libvirt 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Process agent startup answer, agent id = 0 2023-09-13 16:57:51,924 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0 2023-09-13 16:57:51,955 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0 2023-09-13 16:57:52,047 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:e396a97c) Attempting to create storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 (RBD) in libvirt 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:e396a97c) Found existing defined storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84, using it. 2023-09-13 16:57:52,050 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:e396a97c) Trying to fetch storage pool daab90ad-42d3-3c48-a9e4-b4c3c7fcdc84 from libvirt 2023-09-13 16:57:52,161 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:e396a97c) Attempting to create storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c (RBD) in libvirt 2023-09-13 16:57:52,163 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:e396a97c) Storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c was not found running in libvirt. Need to create it. 2023-09-13 16:57:52,164 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:e396a97c) Didn't find an existing storage pool a2d455c6-68cb-303f-a7fa-287e62a5be9c by UUID, checking for pools with duplicate paths 2023-09-13 16:57:56,780 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:) Connected to the host: 10.10.11.62 ^C Kindly guide us to move forward. Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hoss...@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
