So Charlie,
d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid disk? does it
exist on the backend nfs?
and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it exist both in
cloudstack and on the backend?
if both are answered with yes, you probably have a permissions issue, which
might be in the network.
On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky <
charlie.holeow...@gmail.com> wrote:
> Hi Daan and users,
> the infrastructure is based on the Linux environment. The management
> server, hosts and storage are all Ubuntu 16.04 except the new storage
> server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm with NFS
> to share the storage.
>
> We tried to add another primary storage and creating a VM that would use
> it we found no problems, the statistics update and no error messages appear.
>
> Gere is an excerpt of the logs of the most complete agent:
>
> 2020-01-14 09:01:45,749 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) (logid:c3851d3a) Trying to fetch storage pool
> 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt
> 2020-01-14 09:01:45,752 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) (logid:c3851d3a) Asking libvirt to refresh
> storage pool 171e90f4-511e-3b10-9310-b9eec0094be6
> 2020-01-14 09:01:46,641 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) (logid:c3851d3a) Trying to fetch storage pool
> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
> 2020-01-14 09:01:46,643 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) (logid:c3851d3a) Asking libvirt to refresh
> storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
> 2020-01-14 09:05:51,529 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) (logid:2765ff88) Trying to fetch storage pool
> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
> 2020-01-14 09:05:51,532 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) (logid:2765ff88) Asking libvirt to refresh
> storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
> 2020-01-14 09:10:47,286 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:6d27b740) Trying to fetch storage pool
> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
> 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:6d27b740) Caught:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
> at com.cloud.agent.Agent.processRequest(Agent.java:645)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-01-14 09:20:48,390 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) (logid:ec72387b) Trying to fetch storage pool
> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
> 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent]
> (agentRequest-Handler-4:null) (logid:ec72387b) Caught:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
> at com.cloud.agent.Agent.processRequest(Agent.java:645)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-01-14 09:25:15,259 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:1a7e082e) Trying to fetch storage pool
> 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt
> 2020-01-14 09:25:15,261 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:1a7e082e) Asking libvirt to refresh
> storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c
>
>
> And here the management server log:
>
> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075:
> Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat
> com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat
> com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
> com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
> java.lang.Thread.run(Thread.java:748)\n","wait":0}}] }
> 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request]
> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075:
> Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1,
> Flags: 10, { Answer } }
> 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl]
> (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing
> class com.cloud.agent.api.GetVolumeStatsCommand:
> com.cloud.utils.exception.CloudRuntimeException: Can't find
> volume:d93d3c0a-3859-4473-951d-9b5c5912c767
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
> at com.cloud.agent.Agent.processRequest(Agent.java:645)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> on 09/01/20 12:58, Daan Hoogland wrote:
>
> Charlie, I think you'll have to explain a bit more about your environment
> to get an answer. what type of storage is it? Where did you migrate the VM
> from and to? What types() of hypervisors are you using? Though saying **the**
> agent logs suggests KVM, you are still leaving people guessing a lot.
>
>
--
Daan
