Hi Wido, I'm in the project with Kimi and Nakajima-san.
[root@rx200s7-07m ~]# ceph -v ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404) Same version is installed into all the hosts (KVM host and Ceph nodes). Here is KVM Agent log. http://pastebin.com/5yG1uBuj I had set the log level "DEBUG" and failed to create 2 instances ,"RBDVM-shimazaki-1" and "RBDVM-shimazaki-2". Thanks, Satoshi Shimazaki 2013/7/23 Kimihiko Kitase <kimihiko.kit...@citrix.co.jp> > Hi Wido > > Thanks for you comment. > > If we create vm on the NFS primary storage and mount additional disk on > the RBD storage, it works fine. > If we check vm from virt manager, there is no virtual disk. So we believe > the problem should be vm configuration... > > We will check ceph version tomorrow. > > Thanks > Kimi > > -----Original Message----- > From: Wido den Hollander [mailto:w...@widodh.nl] > Sent: Monday, July 22, 2013 11:43 PM > To: dev@cloudstack.apache.org > Subject: Re: Problem in adding Ceph RBD storage to CloudStack > > Hi, > > On 07/22/2013 02:25 PM, Kimihiko Kitase wrote: > > Wido, Thank you very much. > > > > CloudStack: 4.1.0 > > QEMU: 1.5.50 > > Libvirt: 0.10.2 > > What version of Ceph on the nodes? > > $ ceph -v > > > > > We will set "DEBUG" on the agent tomorrow. But the following is command > CloudStack issue. We got this command at KVM host. > > > > [root@rx200s7-07m ~]# ps -ef|grep 1517 > > root 16099 1 27 19:36 ? 00:00:12 /usr/libexec/qemu-kvm > -name i-2-1517-VM -S -M pc-i440fx-1.6 -enable-kvm -m 256 -smp > 1,sockets=1,cores=1,threads=1 -uuid e67f1707-fe92-3426-978d-0441d5000d6a > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/i-2-1517-VM.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot dc -drive > file=rbd:libvirt-pool/cd3688ab-e37b-4866-9ea7-4051b670a323:id=libvirt:key=AQC7OuZReMndFxAAY/qUwLbvfod6EMvgVWU21g==:auth_supported=cephx\;none:mon_host=192.168.10.20\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none > -device > virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 > -drive > if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,cache=none > -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:b9:00:16 > ,bus=pci. > 0,addr=0x3 -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 > -vnc 0.0.0.0:3 -vga cirrus -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > > > > The argument to Qemu seems just fine, so I think the problem is not in > CloudStack. > > Wido > > > Thanks > > Kimi > > > > -----Original Message----- > > From: Wido den Hollander [mailto:w...@widodh.nl] > > Sent: Monday, July 22, 2013 7:47 PM > > To: dev@cloudstack.apache.org > > Subject: Re: Problem in adding Ceph RBD storage to CloudStack > > > > Hi, > > > > On 07/22/2013 12:43 PM, Kimihiko Kitase wrote: > >> It seems secondary storage vm could copy template to primary storage > successfully, but created VM doesn't point this vol.. > >> If we create vm manually and add this vol as boot vol, it works fine.. > >> > > > > Which version of CloudStack are you using? > > > > What is the Qemu version running on your hypervisor and what libvirt > version? > > > > If you set the logging level on the Agent to "DEBUG", does it show > deploying the VM with the correct XML parameters? > > > > I haven't seen the things you are reporting. > > > > Wido > > > >> So it seems cloudstack cannot configure VM correctly in ceph rbd > environment. > >> > >> Any idea? > >> > >> Thanks > >> Kimi > >> > >> -----Original Message----- > >> From: Kimihiko Kitase [mailto:kimihiko.kit...@citrix.co.jp] > >> Sent: Monday, July 22, 2013 7:11 PM > >> To: dev@cloudstack.apache.org > >> Subject: RE: Problem in adding Ceph RBD storage to CloudStack > >> > >> Hello > >> > >> I am in the project with Nakajima san. > >> > >> We succeeded to add RBD storage to primary storage. > >> But when we try to boot centos as user instance, it fail during system > logger process. > >> It works fine when we boot centos using NFS storage. > >> It works fine when we boot centos using NFS storage and add additional > disk from RBD storage. > >> > >> Do you have any idea to resolve this issue? > >> > >> Thanks > >> Kimi > >> > >> -----Original Message----- > >> From: Takuma Nakajima [mailto:penguin.trance.2...@gmail.com] > >> Sent: Saturday, July 20, 2013 12:23 PM > >> To: dev@cloudstack.apache.org > >> Subject: Re: Problem in adding Ceph RBD storage to CloudStack > >> > >> I'm sorry but I forgot to tell you that the environment does not have > the internet connection. > >> It is not allowed to make a direct connection to the internet because > of the security policy. > >> > >> Wido, > >>> No, it works for me like a charm :) > >>> > >>> Could you set the Agent logging to DEBUG as well and show the output > >>> of > >> that log? Maybe paste the log on pastebin. > >>> > >>> I'm interested in the XMLs the Agent is feeding to libvirt when > >>> adding > >> the RBD pool. > >> > >> I thought the new libvirt overwrites the old one, but actually both > libvirt (with RBD and without RBD) were installed to the system. qemu was > installed from the package and so it might have the dependency to the > libvirt installed from the package. After deleting the both libvirt > installed from source and package, then installed it from rpm package with > RBD support, RBD storage was registered to the CloudStack successfully. > >> > >> David, > >>> Why not 6.4? > >> > >> Because of no internet connection, packages in the local mirror > repository may be old. > >> I checked /etc/redhat-release and it showed the version is 6.3. > >> > >> In current state, although the RBD storage was installed, system VMs > >> won't start with "Unable to get vms > >> org.libvirt.LibvirtException: Domain not found: no domain with > >> matching uuid 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'" error like > >> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201303.mbox > >> / > >> %3CD2EE6B3265AD864EB3EA4F5C670D256F3546C1@EXMBX01L-CRP-03.webmdhealth. > >> net%3E The uuid in the error message was not in the database of the > >> management server nor in ceph storage node. > >> > >> I tried removing host from the CloudStack and cleaning up the computing > node, but it cannot be added again to the CloudStack. > >> agent log says it attempted to connect to localhost:8250 though the > management server address is set to 10.40.1.190 in global settings. > >> management server log is here: http://pastebin.com/muGz73c0 > >> (10.40.1.24 is the address of the computing node) > >> > >> Now the computing node is under rebuilding. > >> > >> Takuma Nakajima > >> > >> 2013/7/19 David Nalley <da...@gnsa.us>: > >>> On Thu, Jul 18, 2013 at 12:09 PM, Takuma Nakajima > >>> <penguin.trance.2...@gmail.com> wrote: > >>>> Hi, > >>>> > >>>> I'm building a CloudStack 4.1 with Ceph RBD storage using RHEL 6.3 > >> recently > >>>> but it fails when adding RBD storage to primary storage. > >>>> Does anybody know about the problem? > >>> > >>> > >>> Why not 6.4? >
