Re: GlusterFS QEMU libgfapi
Hi John, On 07/21/2013 03:32 PM, John Mark Walker wrote: Greetings, CloudStackers. I noticed there was a question re: libgfapi as it pertains to libvirt and QEMU. As of last January/February, all new versions of the KVM/QEMU/libvirt/oVirt virtualization stack have been glusterized and use the libgfapi client library when you specify the gluster protocol in QEMU. I noticed, but there is one side note. Libvirt doesn't support libgfapi storage pools: http://www.libvirt.org/storage.html Do not assume the Network Filesystem Pool uses libgfapi, since that simply runs mount with glusterfs as the filesystem type. Right now CloudStack leans on libvirt to manage it's storage pools. Marcus Sorensen correctly noted that we don't HAVE to use libvirt for the storage pool management, but I do prefer to do so. GlusterFS is a RedHat thing just like libvirt, so I'd like to see libgfapi storage pool support implemented in libvirt. There are more things standing out in libvirt though what should be fixed, which makes sense now. I ran into this when implementing RBD support: http://www.redhat.com/archives/libvir-list/2012-February/msg00503.html If that would be implemented both the RBD (Ceph) and libgfapi could leverage from that. You can see this old presentation here that goes over some of these topics: http://www.gluster.org/community/documentation/images/9/9d/QEMU_GlusterFS.pdf That presentation is almost a year old and much of what was listed as in-progress is now in an official release. What this integration means for you is that you should be able to spin up and manage VM images on Gluster volumes directly through libgfapi, which bypasses the fuse mount. This is especially useful for use cases such as VM image hosting, and the performance gains are pretty dramatic. So yes, that would be possible, but for CloudStack (and probably other projects) it would be very useful if we could let libvirt do all the storage handling. Wido I've CC'd the Gluster-devel list, as well as some of the engineers who were working on the integration. If you have specific questions, make sure to keep gluster-devel on the CC list so that they can respond. Thanks! John Mark Walker Gluster Community Leader
Re: GlusterFS QEMU libgfapi
On a side note, we aren't limited to libvirt-supported storage. KVMStoragePoolManager allows for multiple storage adaptors, so you can just add yours to the adaptor map in there and create a MyStorageAdaptor that implements the various methods like CreatePhysicalDisk, etc , similar to LibvirtStorageAdaptor. You would also have to tweak the XML stuff as well if it can't be connected via the existing raw or file based XML disk definitions. On Jul 15, 2013 8:57 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:52 PM, John Skinner wrote: Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. It won't be only Java code, but also C code to manage the GlusterFS storage pool in libvirt: http://libvirt.org/storage.**htmlhttp://libvirt.org/storage.html Currently GlusterFS isn't supported in libvirt as a storage pool, but the CloudStack agent relies on that. It might be possible to do without libvirt, but I'm not sure how that would work out. Ceph and RBD are my thing, I'm not a GlusterFS expert. Wido Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
On 07/15/2013 06:38 PM, John Skinner wrote: I dug a little bit deeper and found that it IS a supported storage type of NETFS, as a valid poor format type (see below from libvirt.org). Now, not being familiar with how CloudStack handles storage; I was think that under cloud-plugin-hypervisoer-kvm src com.cloud.hypervisor.kvm.storage LibvirtStorageAdaptor.java I create a pool type similar to the one for NFS with the required information for using GlusterFS. Is this assumption correct? Yes, if that's the case, you could simply change netfs into glusterfs, but I'm not sure if that uses libgfapi, although that wouldn't matter much. It's when building the domain XML and actually invoking Qemu that you want it to use libgfapi. Wido Thanks, John Valid volume format types The valid volume types are the same as for the directory pool type. Network filesystem pool This is a variant of the filesystem pool. Instead of requiring a local block device as the source, it requires the name of a host and path of an exported directory. It will mount this network filesystem and manage files within the directory of its mount point. It will default to using NFS as the protocol. Example pool input pool type=netfs namevirtimages/name source host name=nfs.example.com/ dir path=/var/lib/virt/images/ /source target path/var/lib/virt/images/path /target /pool Valid pool format types The network filesystem pool supports the following formats: auto - automatically determine format nfs glusterfs cifs On Jul 15, 2013, at 10:08 AM, John Skinner john.skin...@appcore.com wrote: Wido, Are you sure on that? I know the libgfapi is in C. But I thought GlusterFS was now supported in both libvirt and qemu (1.0.1+, 1.3; respectively). 1.0.1: Dec 17 2012 Features: Introduce virtlockd daemon (Daniel P. Berrange), parallels: add disk and network device support (Dmitry Guryanov), Add virDomainSendProcessSignal API (Daniel P. Berrange), Introduce virDomainFSTrim() public API (Michal Privoznik), add fuse support for libvirt lxc (Gao feng), Add Gluster protocol as supported network disk backend (Harsh Prateek Bora), various snapshot improvements (Peter Krempa, Eric Blake) Thanks, John On Jul 15, 2013, at 9:56 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:52 PM, John Skinner wrote: Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. It won't be only Java code, but also C code to manage the GlusterFS storage pool in libvirt: http://libvirt.org/storage.html Currently GlusterFS isn't supported in libvirt as a storage pool, but the CloudStack agent relies on that. It might be possible to do without libvirt, but I'm not sure how that would work out. Ceph and RBD are my thing, I'm not a GlusterFS expert. Wido Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
Hi John, On 07/15/2013 04:52 PM, John Skinner wrote: Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. It won't be only Java code, but also C code to manage the GlusterFS storage pool in libvirt: http://libvirt.org/storage.html Currently GlusterFS isn't supported in libvirt as a storage pool, but the CloudStack agent relies on that. It might be possible to do without libvirt, but I'm not sure how that would work out. Ceph and RBD are my thing, I'm not a GlusterFS expert. Wido Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
Wido, Are you sure on that? I know the libgfapi is in C. But I thought GlusterFS was now supported in both libvirt and qemu (1.0.1+, 1.3; respectively). 1.0.1: Dec 17 2012 Features: Introduce virtlockd daemon (Daniel P. Berrange), parallels: add disk and network device support (Dmitry Guryanov), Add virDomainSendProcessSignal API (Daniel P. Berrange), Introduce virDomainFSTrim() public API (Michal Privoznik), add fuse support for libvirt lxc (Gao feng), Add Gluster protocol as supported network disk backend (Harsh Prateek Bora), various snapshot improvements (Peter Krempa, Eric Blake) Thanks, John On Jul 15, 2013, at 9:56 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:52 PM, John Skinner wrote: Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. It won't be only Java code, but also C code to manage the GlusterFS storage pool in libvirt: http://libvirt.org/storage.html Currently GlusterFS isn't supported in libvirt as a storage pool, but the CloudStack agent relies on that. It might be possible to do without libvirt, but I'm not sure how that would work out. Ceph and RBD are my thing, I'm not a GlusterFS expert. Wido Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
Re: GlusterFS QEMU libgfapi
I dug a little bit deeper and found that it IS a supported storage type of NETFS, as a valid poor format type (see below from libvirt.org). Now, not being familiar with how CloudStack handles storage; I was think that under cloud-plugin-hypervisoer-kvm src com.cloud.hypervisor.kvm.storage LibvirtStorageAdaptor.java I create a pool type similar to the one for NFS with the required information for using GlusterFS. Is this assumption correct? Thanks, John Valid volume format types The valid volume types are the same as for the directory pool type. Network filesystem pool This is a variant of the filesystem pool. Instead of requiring a local block device as the source, it requires the name of a host and path of an exported directory. It will mount this network filesystem and manage files within the directory of its mount point. It will default to using NFS as the protocol. Example pool input pool type=netfs namevirtimages/name source host name=nfs.example.com/ dir path=/var/lib/virt/images/ /source target path/var/lib/virt/images/path /target /pool Valid pool format types The network filesystem pool supports the following formats: auto - automatically determine format nfs glusterfs cifs On Jul 15, 2013, at 10:08 AM, John Skinner john.skin...@appcore.com wrote: Wido, Are you sure on that? I know the libgfapi is in C. But I thought GlusterFS was now supported in both libvirt and qemu (1.0.1+, 1.3; respectively). 1.0.1: Dec 17 2012 Features: Introduce virtlockd daemon (Daniel P. Berrange), parallels: add disk and network device support (Dmitry Guryanov), Add virDomainSendProcessSignal API (Daniel P. Berrange), Introduce virDomainFSTrim() public API (Michal Privoznik), add fuse support for libvirt lxc (Gao feng), Add Gluster protocol as supported network disk backend (Harsh Prateek Bora), various snapshot improvements (Peter Krempa, Eric Blake) Thanks, John On Jul 15, 2013, at 9:56 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:52 PM, John Skinner wrote: Thanks, Wido. I am not a programmer per se, but I am going to pull the code down and have a look to see if I can figure it out. I know some java guys so may be able to get some help on that end. It won't be only Java code, but also C code to manage the GlusterFS storage pool in libvirt: http://libvirt.org/storage.html Currently GlusterFS isn't supported in libvirt as a storage pool, but the CloudStack agent relies on that. It might be possible to do without libvirt, but I'm not sure how that would work out. Ceph and RBD are my thing, I'm not a GlusterFS expert. Wido Thanks, John On Jul 15, 2013, at 9:36 AM, Wido den Hollander w...@widodh.nl wrote: Hi John, On 07/15/2013 04:31 PM, John Skinner wrote: Is there any way to use GlusterFS with the native QEMU libgfapi so we do not have to use Fuse to access the shares? Or are there any plans to build libgfapi QEMU support into CloudStack in the future? As for now there is no way to use libgfapi with Qemu/KVM in CloudStack, nor are there any plans to implement this. Patches are welcome though! Would be great to see this be written. Wido Thanks, John
