Note that if I replace the client.libvirt ceph key with the
client.admin key I get a different error:
qemu-kvm: -drive file=rbd:one/one-19-59-0:id=admin:key=AQAGNMlTED9CHhAAxWyfQvJkM
R1PyTEZ+1lFUQ==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca0
4a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=non
e: could not open disk image
rbd:one/one-19-59-0:id=admin:key=AQAGNMlTED9CHhAAxW
yfQvJkMR1PyTEZ+1lFUQ==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;st
kendca04a\:6789\;stkendca02a\:6789: Invalid argument
as opposed to:
qemu-kvm: -drive
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
error reading header from one-19-58-0
qemu-kvm: -drive
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image
rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789:
Operation not permitted
Anybody who has knowledge of the internals of qemu-kvm and the arguments
it wants, please let me know.. in particular a working deployment.0 and/or
ps output from the qemu-kvm process would be very valuable indeed.
Steve Timm
On Wed, 17 Sep 2014, Steven Timm wrote:
This is the kvm command our libvirt is trying to do, from
/var/log/libvirt/qemu/one-58.log
Clearly it is an authentication error, the only question is
what and why.. we have made the client.libvirt secret
as directed at
http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html
The example there suggests to set CEPH_USER="libvirt"
but I don't have a "libvirt" user defined on my system (most RHEL6/SL6
systems don't.). Should we use a different user?
2014-09-17 18:43:51.579+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -name one-58 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp
2,sockets=2,cores=1,threads=1 -uuid 01ef1f2d-58e2-e089-0012-537426cd1bad
-nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-58.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/one//datastores/102/58/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/var/lib/one//datastores/102/58/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-vnc 127.0.0.1:58 -k en-us -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/3
qemu-kvm: -drive
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
error reading header from one-19-58-0
On Wed, 17 Sep 2014, Steven Timm wrote:
Does anyone have a working Ceph datastore that would
be willing to send me a copy of what the deployment.0 of
your successfully-launched Ceph VM looks like? Likewise
a ps -ef of the corresponding running KVM process?
We are getting close to being able to launch a virtual machine from
Ceph but it is still not quite clear what libvirt is doing
under the covers to talk to rbd, and if it can be tested manually.
Steve Timm
On Wed, 10 Sep 2014, Steven Timm wrote:
> The first and most obvious problem below was that we were running an
> old version of qemu-img and qemu-kvm that ships with RHEL6/Centos6/SL6
> that doesn't support the "rbd" format. We were able to find
> a modified version that the Ceph people had back-ported and now
> we can import an image into the datastore and have
> gotten as far to getting a deployment.0 written on the hypervisor.
> It can't contact Ceph as yet, is getting connection refused but
> we think that is an authentication issue.
>
> http://ceph.com/packages/qemu-kvm/redhat/x86_64/
> is where these packages for 6.2 live.
>
> Hopefully this all gets easier pretty soon now that RedHat has bought
> Ceph
> and all the right packages will be in RHEL7. Or will they be only
> proprietarily available in Redhat Enterprise Virtualization? Has
> anyone
> tried yet?
>
> Steve
>
>
>
> On Wed, 10 Sep 2014, Steven Timm wrote:
>
> >
> >
> > I have configured a Ceph datastore on one 4.6 and have gotten as
> > far as to get opennebula to accept the datastore. But when we
> > try to do the first oneimage create into the datastore we get the
> > following error in oned.log :
> >
> > Wed Sep 10 13:10:44 2014 [ImM][I]: Command execution fail:
> > /var/lib/one/remotes/datastore/ceph/cp
> >
PERTX0RSSVZFUl9BQ1RJT05fREFUQT48SU1BR0U+PElEPjE4PC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+Y2VwaHRlc3Q8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjA8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjM8L0RJU0tfVFlQRT48UEVSU0lTVEVOVD4wPC9QRVJTSVNURU5UPjxSRUdUSU1FPjE0MTAzNzI1NTU8L1JFR1RJTUU+PFNPVVJDRT48L1NPVVJDRT48UEFUSD4vY2xvdWQvaW1hZ2VzL3RpbW0vNDBnYi5xY293MjwvUEFUSD48RlNUWVBFPjwvRlNUWVBFPjxTSVpFPjQwOTYwPC9TSVpFPjxTVEFURT40PC9TVEFURT48UlVOTklOR19WTVM+MDwvUlVOTklOR19WTVM+PENMT05JTkdfT1BTPjA8L0NMT05JTkdfT1BTPjxDTE9OSU5HX0lEPi0xPC9DTE9OSU5HX0lEPjxEQVRBU1RPUkVfSUQ+MTAzPC9EQVRBU1RPUkVfSUQ+PERBVEFTVE9SRT5jZXBoX2RhdGFzdG9yZTwvREFUQVNUT1JFPjxWTVM+PC9WTVM+PENMT05FUz48L0NMT05FUz48VEVNUExBVEU+PERF
> > U0N
> > SSV
> > BU
> S
> >
U9OPjwhW0NEQVRBW3Rlc3QgY2VwaCBnb2xkZW4gaW1nXV0+PC9ERVNDUklQVElPTj48REVWX1BSRUZJWD48IVtDREFUQVtoZF1dPjwvREVWX1BSRUZJWD48RFJJVkVSPjwhW0NEQVRBW3Fjb3cyXV0+PC9EUklWRVI+PC9URU1QTEFURT48L0lNQUdFPjxEQVRBU1RPUkU+PElEPjEwMzwvSUQ+PFVJRD4wPC9VSUQ+PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+b25lYWRtaW48L0dOQU1FPjxOQU1FPmNlcGhfZGF0YXN0b3JlPC9OQU1FPjxQRVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09XTkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT48R1JPVVBfVT4xPC9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09USEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+Y2VwaDwvRFNfTUFEPjxUTV9NQUQ+Y2VwaDwvVE1fTUFEPjxCQVNFX1BBVEg+L3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwMzwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+PERJU0tfVFlQRT4zPC9ESVNLX1RZUEU+PENMVVNURVJfSUQ+LTE8L0NMVVNURVJfSUQ+PENMVVNURVI+PC9DTFVTVEVSPjxUT1RBTF9NQj42MTAyNzMyODwvVE9UQUxfTUI+PEZSRUVfTUI+NjA4NTE1NTI8L0ZSRUVfTUI+PFVTRURfTUI+MTc1Nzc2PC9VU0VEX01CPjxJTUFHRVM+P
> > C9J
> > TUF
> > HR
> V
> >
M+PFRFTVBMQVRFPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy9kYXRhc3RvcmVzL11dPjwvQkFTRV9QQVRIPjxCUklER0VfTElTVD48IVtDREFUQVtvbmU0ZGV2XV0+PC9CUklER0VfTElTVD48Q0VQSF9IT1NUPjwhW0NEQVRBW3N0a2VuZGNhMDFhIHN0a2VuZGNhMDRhIHN0a2VuZGNhMDJhXV0+PC9DRVBIX0hPU1Q+PENFUEhfU0VDUkVUPjwhW0NEQVRBWy9ldGMvY2VwaC9jZXBoLmNsaWVudC5hZG1pbi5rZXlyaW5nXV0+PC9DRVBIX1NFQ1JFVD48Q0xPTkVfVEFSR0VUPjwhW0NEQVRBW1NFTEZdXT48L0NMT05FX1RBUkdFVD48REFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjwhW0NEQVRBW3llc11dPjwvREFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjxESVNLX1RZUEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+PFBPT0xfTkFNRT48IVtDREFUQVtvbmVdXT48L1BPT0xfTkFNRT48U1RBR0lOR19ESVI+PCFbQ0RBVEFbL3Zhci9saWIvb25lL2NlcGgtdG1wXV0+PC9TVEFHSU5HX0RJUj48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01BRD48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg==
> > 18
> > Wed Sep 10 13:10:44 2014 [ImM][I]: cp: Copying local image
> > /cloud/images/timm/40gb.qcow2 to the image repository
> > Wed Sep 10 13:10:44 2014 [ImM][E]: cp: Command " set -e
> > Wed Sep 10 13:10:44 2014 [ImM][I]:
> > Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "" = "2" ]; then
> > Wed Sep 10 13:10:44 2014 [ImM][I]: FORMAT=$(qemu-img info
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c | grep "^file
> > format:"
> > | awk '{print }')
> > Wed Sep 10 13:10:44 2014 [ImM][I]:
> > Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "$FORMAT" != "raw" ]; then
> > Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert -O raw
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
> > Wed Sep 10 13:10:44 2014 [ImM][I]: mv
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
> > Wed Sep 10 13:10:44 2014 [ImM][I]: fi
> > Wed Sep 10 13:10:44 2014 [ImM][I]:
> > Wed Sep 10 13:10:44 2014 [ImM][I]: rbd import --format 2
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c one/one-18
> > Wed Sep 10 13:10:44 2014 [ImM][I]: else
> > Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
> > rbd:one/one-18
> > Wed Sep 10 13:10:44 2014 [ImM][I]: fi
> > Wed Sep 10 13:10:44 2014 [ImM][I]:
> > Wed Sep 10 13:10:44 2014 [ImM][I]: # remove original
> > Wed Sep 10 13:10:44 2014 [ImM][I]: rm -f
> > /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c" failed:
> > Unknown
> > protocol 'rbd:one/one-18'
> > Wed Sep 10 13:10:44 2014 [ImM][E]: Error registering one/one-18 in
> > one4dev
> > Wed Sep 10 13:10:44 2014 [ImM][I]: ExitCode: 1
> > Wed Sep 10 13:10:44 2014 [ImM][E]: Error copying image in the
> > datastore:
> > Error registering one/one-18 in one4dev
> >
> >
> >
> >
> > ---
> >
> > Clear we are afailed to register the rbd, but not clear why.. any
> > or clues on why we failed are helpful.
> > Several places in the docs refer to libvirt 1.x. Has anyone
> > made this work on RHEL6/Centos 6 (we are running the newer kernel so
> > we do have the rbd.o kernel module available, and rbd import/export
> > works from the command line.)
> >
> > Steve Timm
> >
> >
> > ------------------------------------------------------------------
> > Steven C. Timm, Ph.D (630) 840-8525
> > t...@fnal.gov http://home.fnal.gov/~timm/
> > Fermilab Scientific Computing Division, Scientific Computing
> > Services
> > Quad.
> > Grid and Cloud Services Dept., Associate Dept. Head for Cloud
> > Computing
> >
>
> ------------------------------------------------------------------
> Steven C. Timm, Ph.D (630) 840-8525
> t...@fnal.gov http://home.fnal.gov/~timm/
> Fermilab Scientific Computing Division, Scientific Computing Services
> Quad.
> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
> _______________________________________________
> Users mailing list
> Users@lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
t...@fnal.gov http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services
Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
_______________________________________________
Users mailing list
Users@lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
t...@fnal.gov http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
_______________________________________________
Users mailing list
Users@lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
t...@fnal.gov http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
_______________________________________________
Users mailing list
Users@lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
