Re: [Openstack] ZFS/ZVol + iscsi for volume
As Diego pointed out, this should all work already. You just point your nova-volume at your Solaris-like box, and it runs all the commands for you (over SSH). I wrote the original way-back-when as a stepping-stone to support for HP SANs (as I had much easier access to Solaris than real SANs), and it sounds like Diego fixed it up in Diablo, so it hasn't totally bit-rotted. Have you tried this? Did you encounter any problems? Is there something extra you need? Justin On Mon, Jun 11, 2012 at 12:06 PM, Nicolas de BONFILS openst...@nicolas2bonfils.com wrote: ** In fact, my question is more about how to integrate it within nova workflow. I know how do each step (zfs pool, iscsi export, iscsi attach in vm) but this way is not register by nova db because it's not nova commands. I was looking for people who have replace the LVM creation and iscsiadm export part with zfs (pool+export) . --- Nicolas On 11-06-2012 17:05, Alberto Molina Coballes wrote: What about Debian/KFreeBSD for nova-volume? Debian testing (wheezy) with FreeBSD kernel provides support for both OpenStack (nova-volume) and ZFS. We are deploying a little cloud for testing purposes and it will probably be one of the alternatives we will test for storage. Cheers ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
How integrated is the network target support for zfs on freebsd? One of the most compelling features (IMHO) of ZFS on illumos is the whole comstar stack. On the zfs linux port at least, there are just integration hooks out to the standard linux methods (kernel-nfs, etc) for nfs, iscsi, etc. I'm really interested to hear how this goes on FreeBSD. I've been playing with zfs on linux as well, but the results haven't been so good for me. -nld On Mon, Jun 11, 2012 at 10:05 AM, Alberto Molina Coballes alb.mol...@gmail.com wrote: What about Debian/KFreeBSD for nova-volume? Debian testing (wheezy) with FreeBSD kernel provides support for both OpenStack (nova-volume) and ZFS. We are deploying a little cloud for testing purposes and it will probably be one of the alternatives we will test for storage. Cheers ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
Nicholas de BONFILS asked: I use zfs for other project, and I'm very interested in using it everywhere (when possible). One functionality zfs bring is zvol : it allows to create a block device from a zfs pool (a zfs property allow to share this device with iscsi). Mirantis and Nexenta developed a nova driver to make ZFS ZVOLs mountable on demand. A webinar describing it can be found at http://vimeo.com/32498061 We'd be glad to hear any suggestions on additional functionality here. We also have a set of patches that is pending to bring ZFS data integrity to Swift objects. https://review.openstack.org/#/c/7524/ Both bring the benefits of ZFS data integrity to Openstack users at a much lower cost than solutions that rely on excessive network replication. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
El lun, 11-06-2012 a las 10:26 -0500, Narayan Desai escribió: How integrated is the network target support for zfs on freebsd? One of the most compelling features (IMHO) of ZFS on illumos is the whole comstar stack. On the zfs linux port at least, there are just integration hooks out to the standard linux methods (kernel-nfs, etc) for nfs, iscsi, etc. I'm really interested to hear how this goes on FreeBSD. I've been playing with zfs on linux as well, but the results haven't been so good for me. -nld AFAIK, support for ZFS on FreeBSD is good. The main problem for using Debian/KFreeBSD at this moment is nova-volume: it depends on lvm2 and tgt [1], packages that are not available in Debian/KfreeBSD port. [1] http://qa.debian.org/debcheck.php?dist=unstablepackage=nova ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
Also, there is a serious problem with the divergence of EOL schedules between the FreeBSD and Debian camps. Basically, the FreeBSD kernel and other bits will go EOL before the Debian bits will. You need to have a certain amount of faith that the small Debian/kFreeBSD team and/or your own team will be able to sufficiently provide the necessary security backports and hotfixes. For this reason, the support for stable releases of the kFreeBSD releases, in a sense, may be considered significantly shortened compared to standard Debian releases. -- Eric Windisch On Monday, June 11, 2012 at 13:49 PM, Alberto Molina Coballes wrote: El lun, 11-06-2012 a las 10:26 -0500, Narayan Desai escribió: How integrated is the network target support for zfs on freebsd? One of the most compelling features (IMHO) of ZFS on illumos is the whole comstar stack. On the zfs linux port at least, there are just integration hooks out to the standard linux methods (kernel-nfs, etc) for nfs, iscsi, etc. I'm really interested to hear how this goes on FreeBSD. I've been playing with zfs on linux as well, but the results haven't been so good for me. -nld AFAIK, support for ZFS on FreeBSD is good. The main problem for using Debian/KFreeBSD at this moment is nova-volume: it depends on lvm2 and tgt [1], packages that are not available in Debian/KfreeBSD port. [1] http://qa.debian.org/debcheck.php?dist=unstablepackage=nova ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
In fact, my question is more about how to integrate it within nova workflow. I know how do each step (zfs pool, iscsi export, iscsi attach in vm) but this way is not register by nova db because it's not nova commands. I was looking for people who have replace the LVM creation and iscsiadm export part with zfs (pool+export) . --- Nicolas On 11-06-2012 17:05, Alberto Molina Coballes wrote: What about Debian/KFreeBSD for nova-volume? Debian testing (wheezy) with FreeBSD kernel provides support for both OpenStack (nova-volume) and ZFS. We are deploying a little cloud for testing purposes and it will probably be one of the alternatives we will test for storage. Cheers ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
We're definitely interested in this sort of thing. So much so that we've already hacked support into nova-volume to run directly on top of an illumos box with zfs. ;) We've only gotten the basics working, and we haven't done any serious torture testing of it yet. Our real goal is to get things working properly using iSER. We tried it on Stuart Stent's recommendation after the summit and it performs considerably better than stock iscsi over infiniband. Our eventual plan is to get the code ready for upstream merge, but it isn't nearly ready for that yet. Ping me offline and I can send you the code, if you're interested. (this is very much a caveat emptor thing at the moment; we're not done any hardening work yet) -nld On Sun, Jun 10, 2012 at 4:15 AM, Nicolas de BONFILS openst...@nicolas2bonfils.com wrote: Hi Stackers, I got an idea/question about iscsi volume. Actually the workflow is : create an LVM pool create a volume in this pool (with nova command or horizon UI) share it with an iscsi target (the compute/volume node) attach it with an iscsi initiator (the VM) I use zfs for other project, and I'm very interested in using it everywhere (when possible). One functionality zfs bring is zvol : it allows to create a block device from a zfs pool (a zfs property allow to share this device with iscsi). With this, a new workflow will be : create a zfs pool (zpool create name device) create a zvol block device (zfs create -V size poolname/volumename) set the zfs property to share it with iscsi (zfs set shareiscsi=on poolname/volumename) attach it with an iscsi initiator (the VM) From what I read, the Nexenta guys do a lot of work around zfs, but for volume use I only found code to plug a Nexenta san (I do not have the hardware to build a san), but no solution to use zfs drive/pool directly. So my 2 questions are : are some people interested in it ? does it seem possible to do it (maybe some one already try) ? Nova volume driver [github.com] got subclasses for specific cases, maybe a new zfs one. Please discuss, submit ideas ! --- Nicolas ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
Nicolas, it makes a lot of sense what you want to do. ZFS+iSCSI is a perfect fit for vm storage, some other IaaS solutions already use this combination (and I have developed some). Keep in mind that Nexenta is only a distro based upon OpenSolaris: OpenIndiana, FreeBSD... there are a lot of options with ZFS. There are several things you should consider to implement this feature: 1) The code that matters is mostly in nova/virt/libvirt/connection.py for libvirt 2) You need to implement a new storage type: iscsi ( http://libvirt.org/storage.html#StorageBackendISCSI) 3) You need to implement a feature to block-copy the source image to the backend iscsi storage 4) _cache_image method and most of _create_image should be developed thinking in 3) I think the only reason to implement this feature is to develop a highly scalable storage system for instances, so a single storage system does not make sense. It should be able to manage a list storage systems as a pool and load balance them. Some quick thoughts, good luck! Diego -- Diego Parrilla http://www.stackops.com/*CEO* *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ * * On Sun, Jun 10, 2012 at 11:15 AM, Nicolas de BONFILS openst...@nicolas2bonfils.com wrote: ** Hi Stackers, I got an idea/question about iscsi volume. Actually the workflow is : 1. create an LVM pool 2. create a volume in this pool (with nova command or horizon UI) 3. share it with an iscsi target (the compute/volume node) 4. attach it with an iscsi initiator (the VM) I use zfs for other project, and I'm very interested in using it everywhere (when possible). One functionality zfs bring is zvol : it allows to create a block device from a zfs pool (a zfs property allow to share this device with iscsi). With this, a new workflow will be : 1. create a zfs pool (zpool create name device) 2. create a zvol block device (zfs create -V size poolname/volumename) 3. set the zfs property to share it with iscsi (zfs set shareiscsi=on poolname/volumename) 4. attach it with an iscsi initiator (the VM) From what I read, the Nexenta guys do a lot of work around zfs, but for volume use I only found code to plug a Nexenta san (I do not have the hardware to build a san), but no solution to use zfs drive/pool directly. So my 2 questions are : 1. are some people interested in it ? 2. does it seem possible to do it (maybe some one already try) ? Nova volume driver [github.comhttps://github.com/openstack/nova/blob/master/nova/volume/driver.py] got subclasses for specific cases, maybe a new zfs one. Please discuss, submit ideas ! --- Nicolas ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
Hi Diego, Thanks for your ideas about vm storage.In my proposal/question, I got in mind (for now) volume more than vm storage (with the zfs property allowing nfs sharing, I believe that vm storage can already be done with zfs backend). --- Nicolas On 2012-06-10 17:58, Diego Parrilla Santamara wrote: Nicolas, it makes a lot of sense what you want to do. ZFS+iSCSI is a perfect fit for vm storage, some other IaaS solutions already use this combination (and I have developed some). Keep in mind that Nexenta is only a distro based upon OpenSolaris: OpenIndiana, FreeBSD... there are a lot of options with ZFS. There are several things you should consider to implement this feature: 1) The code that matters is mostly innova/virt/libvirt/connection.py for libvirt 2) You need to implement a new storage type: iscsi (http://libvirt.org/storage.html#StorageBackendISCSI) 3) You need to implement a feature to block-copy the source image to the backend iscsi storage 4) _cache_image method and most of _create_image should be developed thinking in 3) I think the only reason to implement this feature is to develop a highly scalable storage system for instances, so a single storage system does not make sense. It should be able to manage a list storage systems as a pool and load balance them. Some quick thoughts, good luck! Diego -- Diego ParrillaCEOwww.stackops.com|diego.parri...@stackops.com|+34 649 94 43 29 |skype:diegoparrilla ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] ZFS/ZVol + iscsi for volume
Hi Nicolas, have you seen this? https://github.com/StackOps/nova/blob/master/nova/volume/san.py It has a Solaris implementation that works on top ZFS. And it works. We modified it work with Nexenta in Diablo: https://github.com/StackOps/nova/blob/stable/diablo/nova/volume/san.py Cheers Diego -- Diego Parrilla http://www.stackops.com/*CEO* *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ * * On Sun, Jun 10, 2012 at 8:07 PM, Nicolas de BONFILS openst...@nicolas2bonfils.com wrote: ** Hi Diego, Thanks for your ideas about vm storage. In my proposal/question, I got in mind (for now) volume more than vm storage (with the zfs property allowing nfs sharing, I believe that vm storage can already be done with zfs backend). --- Nicolas On 2012-06-10 17:58, Diego Parrilla Santamaría wrote: Nicolas, it makes a lot of sense what you want to do. ZFS+iSCSI is a perfect fit for vm storage, some other IaaS solutions already use this combination (and I have developed some). Keep in mind that Nexenta is only a distro based upon OpenSolaris: OpenIndiana, FreeBSD... there are a lot of options with ZFS. There are several things you should consider to implement this feature: 1) The code that matters is mostly in nova/virt/libvirt/connection.py for libvirt 2) You need to implement a new storage type: iscsi ( http://libvirt.org/storage.html#StorageBackendISCSI) 3) You need to implement a feature to block-copy the source image to the backend iscsi storage 4) _cache_image method and most of _create_image should be developed thinking in 3) I think the only reason to implement this feature is to develop a highly scalable storage system for instances, so a single storage system does not make sense. It should be able to manage a list storage systems as a pool and load balance them. Some quick thoughts, good luck! Diego -- Diego Parrilla *CEO* *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ ** blocked.gif___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
