sadhu suresh created CLOUDSTACK-4217:
----------------------------------------
Summary: ceph:addprimary storage is failed with exception
Key: CLOUDSTACK-4217
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4217
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: sadhu suresh
Assignee: Wido den Hollander
Priority: Critical
Steps:
1.created and configured the ceph cluster.
2. Install cloudstack agent on ubuntu13.4 and add it in the cloud-stack's kVM
cluster
3. Install librdos package on kvm host and also install ceph-common pakage
4. able to create the image from kvm host using qemu-img command
root@Rack1Pod1Host21:/etc/ceph# qemu-img create -f rbd rbd:data/foo 1G
Formatting 'rbd:data/foo', fmt=rbd size=1073741824 cluster_size=0
5.try to create a primay storage by selecting the rbd as protocol and
provinding cpeh cluster details
Actual result:
Addprimary storage "failed with delete storagepool failed error" and log shows
intially createstoragepool command successful and fail to access the storage
pool.
Agent.log:
:{"_command":{"hostType":"Routing","hostId":7,"wait":0},"result":true,"wait":0}}]
}
2013-08-09 22:13:16,905 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
Processing command: com.cloud.agent.api.GetHostStatsCommand
2013-08-09 22:13:16,905 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Executing: /bin/bash -c idle=$(top -b -n 1|grep
Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
2013-08-09 22:13:17,069 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Execution is successful.
2013-08-09 22:13:17,070 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
Seq 7-1066205442: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"empty
String","wait":0}}] }
2013-08-09 22:13:52,969 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Request:Seq 7-1066205443: { Cmd , MgmtId: 7296881000534, via: 7, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.CreateStoragePoolCommand":{"add":true,"pool":{"id":19,"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","host":"10.147.41.3","path":"data","userInfo":"ceph:QVFCVjdBRlNzTG1aRUJBQW1HaXZBay9CVjFpTXZjYVJIUWZpWGc9PQ==","port":6789,"type":"RBD"},"localPath":"/mnt//ed2dce31-1512-366e-84a8-a296cc5768f0","wait":0}}]
}
2013-08-09 22:13:52,969 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Processing command: com.cloud.agent.api.CreateStoragePoolCommand
2013-08-09 22:13:52,969 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Seq 7-1066205443: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"success","wait":0}}] }
2013-08-09 22:13:53,019 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Request:Seq 7-1066205444: { Cmd , MgmtId: 7296881000534, via: 7, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":19,"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","host":"10.147.41.3","path":"data","userInfo":"ceph:QVFCVjdBRlNzTG1aRUJBQW1HaXZBay9CVjFpTXZjYVJIUWZpWGc9PQ==","port":6789,"type":"RBD"},"localPath":"/mnt//ed2dce31-1512-366e-84a8-a296cc5768f0","wait":0}}]
}
2013-08-09 22:13:53,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
2013-08-09 22:13:53,022 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) createStoragePool didn't find existing running
pool: org.libvirt.LibvirtException: Storage pool not found: no storage pool
with matching uuid ,�^U0^_�:����;;?+^R, need to create it
2013-08-09 22:13:53,022 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Didn't find an existing storage pool
2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12 by UUID, checking for pools with duplicate
paths
2013-08-09 22:13:53,025 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Checking path of existing pool
99e0dae1-1ac8-46fa-a23e-20938b543c25 against pool we want to create
2013-08-09 22:13:53,029 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Attempting to create storage pool
2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12
2013-08-09 22:13:53,029 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <secret ephemeral='no' private='no'>
<uuid>2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12</uuid>
<usage type='ceph'>
<name>[email protected]:6789/data</name>
</usage>
</secret>
2013-08-09 22:13:53,031 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) org.libvirt.LibvirtException: internal error a
secret with UUID 2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12 is already defined for
use with [email protected]:6789/data
2013-08-09 22:13:53,031 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <pool type='rbd'>
<name>2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12</name>
<uuid>2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12</uuid>
<source>
<host name='10.147.41.3' port='6789'/>
<name>data</name>
<auth username='ceph' type='ceph'>
<secret uuid='2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12'/>
</auth>
</source>
</pool>
2013-08-09 22:13:53,036 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) org.libvirt.LibvirtException: An error occurred,
but the cause is unknown
2013-08-09 22:13:53,037 WARN [cloud.agent.Agent] (agentRequest-Handler-5:null)
Caught:
java.lang.NullPointerException
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:526)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:112)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:105)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2572)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1242)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-09 22:13:53,038 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Seq 7-1066205444: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:526)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:112)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:105)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2572)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1242)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-08-09 22:14:14,873 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-5:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
get_rule_logs_for_vms
2013-08-09 22:14:14,946 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-5:null)
Host:ubuntu13.4
virsh # version
Compiled against library: libvirt 1.0.2
Using library: libvirt 1.0.2
Using API: QEMU 1.0.2
Running hypervisor: QEMU 1.4.0
management.log:
cut -d, -f2);echo $idle
2013-08-09 18:11:14,696 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Execution is successful.
2013-08-09 18:11:14,698 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
Seq 7-1066205192: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"empty
String","wait":0}}] }
2013-08-09 18:11:14,822 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
get_rule_logs_for_vms
2013-08-09 18:11:14,892 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) Execution is successful.
2013-08-09 18:11:14,893 DEBUG [cloud.agent.Agent] (UgentTask-3:null) Sending
ping: Seq 7-3: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":7,"wait":0}}]
}
2013-08-09 18:11:14,904 DEBUG [cloud.agent.Agent] (Agent-Handler-5:null)
Received response: Seq 7-3: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1,
Flags: 100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":7,"wait":0},"result":true,"wait":0}}]
}
2013-08-09 18:11:20,624 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Request:Seq 7-1066205193: { Cmd , MgmtId: 7296881000534, via: 7, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.CreateStoragePoolCommand":{"add":true,"pool":{"id":13,"uuid":"2a5284e1-d490-34f1-a591-e3c8ea4b8f60","host":"10.147.41.3","path":"rbd","userInfo":"client.admin:AQBV7AFSsLmZEBAAmGivAk_BV1iMvcaRHQfiXg==","port":6789,"type":"RBD"},"localPath":"/mnt//81fb9a4e-e62f-3956-8164-f4b557bd08a6","wait":0}}]
}
2013-08-09 18:11:20,625 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Processing command: com.cloud.agent.api.CreateStoragePoolCommand
2013-08-09 18:11:20,625 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Seq 7-1066205193: { Ans: , MgmtId: 7296881000534, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"success","wait":0}}] }
2013-08-09 18:11:20,721 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Request:Seq 7-1066205194: { Cmd , MgmtId: 7296881000534, via: 7, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":13,"uuid":"2a5284e1-d490-34f1-a591-e3c8ea4b8f60","host":"10.147.41.3","path":"rbd","userInfo":"client.admin:AQBV7AFSsLmZEBAAmGivAk_BV1iMvcaRHQfiXg==","port":6789,"type":"RBD"},"localPath":"/mnt//81fb9a4e-e62f-3956-8164-f4b557bd08a6","wait":0}}]
}
2013-08-09 18:11:20,721 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Processing command: com.cloud.agent.api.ModifyStoragePoolCommand
2013-08-09 18:11:20,723 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) createStoragePool didn't find existing running
pool: org.libvirt.LibvirtException: Storage pool not found: no storage pool
with matching uuid *R��Ô4����K�`, need to create it
2013-08-09 18:11:20,723 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Didn't find an existing storage pool
2a5284e1-d490-34f1-a591-e3c8ea4b8f60 by UUID, checking for pools with duplicate
paths
2013-08-09 18:11:20,726 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Checking path of existing pool
99e0dae1-1ac8-46fa-a23e-20938b543c25 against pool we want to create
2013-08-09 18:11:20,730 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Attempting to create storage pool
2a5284e1-d490-34f1-a591-e3c8ea4b8f60
2013-08-09 18:11:20,731 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <secret ephemeral='no' private='no'>
<uuid>2a5284e1-d490-34f1-a591-e3c8ea4b8f60</uuid>
<uuid>2a5284e1-d490-34f1-a591-e3c8ea4b8f60</uuid>
<usage type='ceph'>
<name>[email protected]:6789/rbd</name>
</usage>
</secret>
2013-08-09 18:11:20,741 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <pool type='rbd'>
<name>2a5284e1-d490-34f1-a591-e3c8ea4b8f60</name>
<uuid>2a5284e1-d490-34f1-a591-e3c8ea4b8f60</uuid>
<source>
<host name='10.147.41.3' port='6789'/>
<name>rbd</name>
<auth username='client.admin' type='ceph'>
<secret uuid='2a5284e1-d490-34f1-a591-e3c8ea4b8f60'/>
</auth>
</source>
</pool>
2013-08-09 18:11:20,748 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) org.libvirt.LibvirtException: An error occurred,
but the cause is unknown
2013-08-09 18:11:20,749 WARN [cloud.agent.Agent] (agentRequest-Handler-5:null)
Caught:
java.lang.NullPointerException
at
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:526)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:112)
at
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:105)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2572)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1242)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTas
root@Rack1Pod1Host21:~# virsh pool-define ceph.xml
Pool 2a5284e1-d490-34f1-a591-e3c8ea4b8f60 defined from ceph.xml
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
