Well, I do not think that it is down to 2.6.32 kernel that is causing your issue as CS and RBD integration doesn't use kernel module. Please correct me if I am wrong, but I think it uses userspace libraries. Try asking people on OFTC irc #ceph, there should be people using centos/RH.
When I tried rbd with cs using centos 6.4 I have compiled the latest libvirt and qemu from sources and didn't have that many issues apart from heavy load server reboots. I was running fio with 4 files each threading 16 random reads/writes with block size of 4K. Larger block sizes didn't give me any issues. At that time I had to use centos because of the infiniband support issues which i had with ubuntu, but the latest ofed drivers sorted out the issue and I am happily back to ubuntu. Do you need to use Centos for a reason? Taking into account that Ceph is heavily developed and tested using Ubuntu servers, I would use it instead of Centos, unless there are issues with hardware, etc. which do not work in Ubuntu. Andrei ----- Original Message ----- From: "不坏阿峰" <onlydeb...@gmail.com> To: users@cloudstack.apache.org Sent: Tuesday, 20 August, 2013 3:18:36 AM Subject: Re: Fail to add RBD ceph as Primary storage on Cloudstack 4.1+Centos6.4 KVM host thank you for your response. i have done Widoh's way too and running well on ubuntu. so i know it is the centos some kernel and features too old. Hope have some one make a success deploy on Centos 6.4 2013/8/20 Andrei Mikhailovsky <and...@arhont.com>: > I've been testing centos 6.4 extensively with rbd and had a lot of stability > issues where the host server would freeze or reboot during high load > benchmarks, especially with small block sizes. I have switched to ubuntu > 12.04 with 3.5 kernel and didn't see this behaviour. Just my two cents. > > P.S. i've not had any issues with adding rbd cluster to cloudstack following > the Widoh's howto on the net. > > Andrei > ----- Original Message ----- > > From: "不坏阿峰" <onlydeb...@gmail.com> > To: users@cloudstack.apache.org > Sent: Monday, 19 August, 2013 1:40:46 PM > Subject: Fail to add RBD ceph as Primary storage on Cloudstack 4.1+Centos6.4 > KVM host > > Environment: > ################## > Cloudstack 4.1.1, Ceph 0.6.1.7, Qemu 0.12.1.2-2.355 with RBD enable > > I have added ceph RBD as primary successfully with ubuntu 12.04 kvm. > but fail on centos 6.4 kvm host in cloudstack. > > i did some test on centos kvm host, i think can use rbd to access ceph > (have the ceph.conf on KVM host) > ####################### > [root@centos-kvm01 ~]# qemu-img -v | grep rbd > Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat > qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device > file rbd > > [root@centos-kvm01 ~]# rbd -m 192.168.250.15 -p volumes ls > 08b376f0-5af4-4b46-b41e-d83dc0f93a70 > 1e9f8d76-94d9-4d56-a8a6-fdff32ba9d1b > b0f7fae3-716b-4210-a89d-af63c6e30859 > > [root@centos-kvm01 ~]# rados lspools > data > metadata > rbd > volumes > cloudstack > > [root@centos-kvm01 ~]# rbd create test --size 4096 > [root@centos-kvm01 ~]# rbd ls > test > > cloudstack log: > #################### > 2013-08-19 17:28:25,683 DEBUG [agent.transport.Request] > (AgentManager-Handler-9:null) Seq 26-368640010: Processing: { Ans: , > MgmtId: 345050143793, via: 26, Ver: v1, Flags: 10, > [{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:540)\n\tat > > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:111)\n\tat > > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:104)\n\tat > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2304)\n\tat > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1094)\n\tat > > com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat > com.cloud.utils.nio.Task.run(Task.java:83)\n\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)\n\tat > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat > > java.lang.Thread.run(Thread.java:679)\n","wait":0}}] } > 2013-08-19 17:28:25,683 DEBUG [agent.transport.Request] > (catalina-exec-1:null) Seq 26-368640010: Received: { Ans: , MgmtId: > 345050143793, via: 26, Ver: v1, Flags: 10, { Answer } } > 2013-08-19 17:28:25,683 DEBUG [agent.manager.AgentManagerImpl] > (catalina-exec-1:null) Details from executing class > com.cloud.agent.api.ModifyStoragePoolCommand: > java.lang.NullPointerException > at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:540) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:111) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:104) > > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2304) > > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1094) > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) > at com.cloud.utils.nio.Task.run(Task.java:83) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:679) > > 2013-08-19 17:28:25,689 WARN [cloud.storage.StorageManagerImpl] > (catalina-exec-1:null) Unable to establish a connection between > Host[-26-Routing] and Pool[207|RBD] > com.cloud.exception.StorageUnavailableException: Resource > [StoragePool:207] is unreachable: Unable establish connection from > storage head to storage pool 207 due to java.lang.NullPointerException > at > com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:540) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:111) > > at > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:104) > > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2304) > > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1094) > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) > at com.cloud.utils.nio.Task.run(Task.java:83) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:679) > at > com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1637) > > at > com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1402) > at > com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:147) > at > org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:123) > > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:505) > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:355) > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302) > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > at > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) > > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:679) > 2013-08-19 17:28:25,690 WARN [cloud.storage.StorageManagerImpl] > (catalina-exec-1:null) No host can access storage pool Pool[207|RBD] > on cluster 3 > 2013-08-19 17:28:25,693 INFO [cloud.api.ApiServer] > (catalina-exec-1:null) Failed to add storage pool >