xuanyuanaosheng edited a comment on issue #5776: URL: https://github.com/apache/cloudstack/issues/5776#issuecomment-998465025
@wido I restart the VM，But It is also report the same errors. I do not know why this happen. ------ Today I add another kvm host into the cluster. Now the cluster has two nodes. - 10.26.246.5 (First added) - 10.26.246.6 (Later added) The additon info: 1. The ceph RBD primary storage scope is CLUSTER mode The First added node report errors， But the Later added node works fine. Now the first kvm node (10.26.246.5) has can not connect to the managerment node, the cloudstack agent error: **Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6** The details error as below:  ``` 2021-12-21 12:30:45,444 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt 2021-12-21 12:30:45,446 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it. 2021-12-21 12:30:45,446 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths 2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6 2021-12-21 12:30:45,451 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:5e42d071) Failed to create the RBD storage pool, cleaning up the libvirt secret 2021-12-21 12:30:45,451 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:5e42d071) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892 at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-12-21 12:30:45,462 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress. 2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed 2021-12-21 12:30:45,463 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173 2021-12-21 12:30:45,463 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250 2021-12-21 12:30:45,463 INFO [utils.nio.Link] (Agent-Handler-4:null) (logid:ed0d935b) Conf file found: /etc/cloudstack/agent/agent.properties 2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) SSL: Handshake done 2021-12-21 12:30:45,531 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connected to 10.25.2.173:8250 2021-12-21 12:30:45,560 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:ed0d935b) Failed to parse Script output: Host.OS.Version= 2021-12-21 12:30:45,595 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Attempting to create storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 (Filesystem) in libvirt 2021-12-21 12:30:45,595 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:ed0d935b) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion 2021-12-21 12:30:45,596 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Found existing defined storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1, using it. 2021-12-21 12:30:45,597 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:ed0d935b) Trying to fetch storage pool d8dabcb0-1a57-4e13-8a82-339b2052dec1 from libvirt 2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Proccess agent startup answer, agent id = 0 2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Set agent id 0 2021-12-21 12:30:45,606 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:ed0d935b) Startup Response Received: agent id = 0 2021-12-21 12:30:45,800 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Attempting to create storage pool 60b59087-7c53-3058-a50c-f50737e556bc (NetworkFilesystem) in libvirt 2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Found existing defined storage pool 60b59087-7c53-3058-a50c-f50737e556bc, using it. 2021-12-21 12:30:45,803 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:2e37d7fd) Trying to fetch storage pool 60b59087-7c53-3058-a50c-f50737e556bc from libvirt 2021-12-21 12:30:45,873 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Attempting to create storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff (NetworkFilesystem) in libvirt 2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Found existing defined storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff, using it. 2021-12-21 12:30:45,875 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:2e37d7fd) Trying to fetch storage pool c8e9ca6a-c004-3851-a074-19f4948b28ff from libvirt 2021-12-21 12:30:45,897 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:d1d208d9) Connected to the host: 10.25.2.173 2021-12-21 12:30:45,942 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Attempting to create storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa (NetworkFilesystem) in libvirt 2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Found existing defined storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa, using it. 2021-12-21 12:30:45,945 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:2e37d7fd) Trying to fetch storage pool c4355ed4-8833-381f-b3f7-2981782ee3fa from libvirt 2021-12-21 12:30:46,053 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Attempting to create storage pool b90eae9d-973c-362c-8afc-af88f0743892 (RBD) in libvirt 2021-12-21 12:30:46,056 WARN [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Storage pool b90eae9d-973c-362c-8afc-af88f0743892 was not found running in libvirt. Need to create it. 2021-12-21 12:30:46,056 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Didn't find an existing storage pool b90eae9d-973c-362c-8afc-af88f0743892 by UUID, checking for pools with duplicate paths 2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create RBD storage pool: org.libvirt.LibvirtException: operation failed: pool 'b90eae9d-973c-362c-8afc-af88f0743892' already exists with uuid 1e407a5d-cdc3-482a-8b5d-762e7883cfa6 2021-12-21 12:30:46,060 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:2e37d7fd) Failed to create the RBD storage pool, cleaning up the libvirt secret 2021-12-21 12:30:46,060 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:2e37d7fd) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: b90eae9d-973c-362c-8afc-af88f0743892 at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1648) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Lost connection to host: 10.25.2.173. Attempting reconnection while we still have 0 commands in progress. 2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) NioClient connection closed 2021-12-21 12:30:46,073 INFO [cloud.agent.Agent] (Agent-Handler-4:null) (logid:ed0d935b) Reconnecting to host:10.25.2.173 2021-12-21 12:30:46,073 INFO [utils.nio.NioClient] (Agent-Handler-4:null) (logid:ed0d935b) Connecting to 10.25.2.173:8250 ``` I have searched this error on google, But I can not get any help. Any idears? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
