Well, the default KVM one does, but only because the hypervisor is playing the part or storage API. My plugin certainly doesn't have to talk to the hypervisor, and I don't think I've instructed you to. On Sep 27, 2013 7:29 PM, "Marcus Sorensen" <shadow...@gmail.com> wrote:
> The plugin itself doesn't talk to the hypervisor. > On Sep 27, 2013 7:28 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> > wrote: > >> Yeah, we should bring John Burwell into this conversation. He had a strong >> opposition to storage plug-ins talking to hypervisors. Perhaps he was just >> referring to shared storage, though (not local disks). >> >> Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate >> your >> SR/datastore (there was a minor exception for XenServer). CloudStack >> wasn't >> involved in that process at all. It was a manual admin task. >> >> The storage plug-in framework didn't really solve that problem, though. >> All >> it did was allow you to create your storage volume and make it available >> to >> CloudStack. There was no way to get XenServer or VMware to actually use it >> because they didn't want an IQN or whatever, they wanted an SR/datastore. >> >> That being the case, I had to write code to detect this "managed" >> situation >> in XenServer and VMware (in the attach/detach methods). >> >> For example, for XenServer I had to first create an SR and a VDI (which >> the >> SolidFire plug-in probably should have no knowledge of). Then the attach >> logic would work. >> >> After a detach, I had to have XenServer "forget" the SR. >> >> What I was thinking is that the attach/detach methods aren't really the >> right place for this special logic. >> >> We could have VolumeApiServiceImpl know about the special needs of >> "managed" storage. >> >> If you want to do an attach of a volume and you're managed storage, >> VolumeApiService Imple could first tell the hypervisor to create the >> necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just >> establish the connection to the iSCSI target). >> >> It true that there are commands like CreateObjectCommand in XenServer and >> VMware now, but they are only used when the storage is preallocated (not >> when the storage is managed by CloudStack). >> >> In the end, we kind of have two different storage models in CloudStack: >> Non-managed (preallocated storage by admin) and managed (plug-in >> dynamically creates storage on a storage system). >> >> >> On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen <shadow...@gmail.com >> >wrote: >> >> > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen <shadow...@gmail.com> >> > wrote: >> > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski >> > > <mike.tutkow...@solidfire.com> wrote: >> > >> createAsync is just for creating the SAN (or whatever storage) >> volume. >> > >> deleteAsync is the reverse. >> > > >> > > Exactly. It used to be that the hypervisor created the disk/lun/file >> > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin >> > > supports it. Ideally, only calls that are required for utilizing the >> > > storage (or perhaps things like copy to NFS, where a server need be >> > > involved if your SAN can't do it directly) will go to the hypervisor, >> > > for external plugins. >> > > >> > > So mgmt server creates the LUN on the SAN, then calls the hypervisor >> > > to attach it to the host, so that a VM can make use of it. The >> > > createAsync is hypervisor agnostic, it just creates a LUN, and then >> > > when you go to start up a VM or attach it to one it calls the >> > > hypervisor-specific code to make it available. >> > >> > I think this is the same with the other hypervisors as well. For >> > example with Xen, you create the LUN via your plugin, and then a call >> > is made to Xen to register it as an SR. It's basically the same, only >> > you're now coding the 'register' part yourself (and it's ephemeral, >> > it's re-registered whenever it's used). You basically take over some >> > of the API coding that already exists on the other platforms, which >> > means you can do whatever you want instead of just what's supported, >> > but you have to do the work as well. >> > >> > > >> > >> >> > >> Technically even the default plug-in should not call into the >> hypervisor >> > >> layer. >> > > >> > > There's no way you can create a local storage file to use as a volume, >> > > or CLVM volume, or other types of libvirt storage without calling a >> > > service that runs on the hypervisor. Those things exist only on the >> > > hypervisor, and are controlled by the hypervisor. For NFS, you could >> > > create a separate API for your NFS server that creates qcow2 images on >> > > your NFS primary, I suppose. >> > > >> > > One of the really nice things about KVM is that we can do whatever a >> > > Linux box is capable of, it was one of the draws we had to it. We >> > > wouldn't be able to do the storage we do with Xen or VMWare. >> > > >> > >> >> > >> The storage layer should probably not be aware of the hypervisor >> layer. >> > > >> > > That's fine, but there's no reason why a storage plugin can't talk to >> > > the agent that happens to be running on the hypervisor for >> > > implementation, if that's what the plugin intends. I don't see the >> > > distinction between utilizing the kvm agent as you storage API or >> > > talking to a custom SAN API, or some other concocted service. That's >> > > sort of the point of the plugin, people can do whatever they want. >> > > >> > >> >> > >> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski < >> > >> mike.tutkow...@solidfire.com> wrote: >> > >> >> > >>> Well, from what I saw with XenServer and VMware, that hypervisor >> > logic's >> > >>> attachVolume command also assumed a VDI/VMDK was created in advance. >> > >>> >> > >>> I had to put logic in those attachVolume methods to create the >> SR/VDI >> > or >> > >>> datastore/VMDK. >> > >>> >> > >>> However, thinking back on it, it might have made more sense for the >> > >>> storage framework to detect if the storage in question was managed >> and >> > - >> > >>> before calling attach - call create. >> > >>> >> > >>> If that logic was in place, I could have left attach/detachVolume >> alone >> > >>> and implemented create and delete in the hypervisor code to create >> my >> > >>> SR/VDI or datastore/VMDK. >> > >>> >> > >>> That makes sense to me because it is CloudStack-managed storage (so >> > >>> CloudStack is calling into the hypervisor to create and delete these >> > types >> > >>> of objects...it's managing them). >> > >>> >> > >>> >> > >>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen < >> shadow...@gmail.com >> > >wrote: >> > >>> >> > >>>> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski >> > >>>> <mike.tutkow...@solidfire.com> wrote: >> > >>>> > Sure, sounds good - let me know when it's up on Review Board and >> I >> > can >> > >>>> take >> > >>>> > a look. >> > >>>> > >> > >>>> > I made most of the changes you and I talked about: >> > >>>> > >> > >>>> > >> > >>>> >> > >> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1 >> > >>>> > >> > >>>> > I have a new idea, though, that I think will simplify this: >> > >>>> > >> > >>>> > The main "weirdness" right now is when attachVolume is called >> that >> > the >> > >>>> > original logic assumed createVolume had been called already. >> > >>>> > >> > >>>> > In my case, this doesn't apply, so we had to place extra logic in >> > >>>> > attachVolume to essentially "create" a volume. We decided to >> make a >> > >>>> connect >> > >>>> > method, which establishes the iSCSI connection and creates a >> > >>>> > KVMPhysicalDisk that can be returned when attachVolume calls >> > >>>> > getPhysicalDisk. >> > >>>> > >> > >>>> > The "normal" place where you'd create a KVMPhysicalDisk, however, >> > would >> > >>>> be >> > >>>> > in the createVolume method. Since I don't currently "create" a >> > volume, >> > >>>> my >> > >>>> > only chance to note the size of the volume is in the connect >> method. >> > >>>> >> > >>>> I don't think createVolume applies to plugins. My impression wash >> that >> > >>>> createAsync is called on the mgmt server side. If createVolume IS >> > >>>> being called, that's weird. The idea here is that mgmt server >> creates >> > >>>> the LUN, and then on the KVM side attach is called (or >> StartCommand if >> > >>>> it's a root volume and vm is being started), and it assumes that >> the >> > >>>> LUN is already there, so we call connectPhysicalDisk to attach it >> to >> > >>>> the KVM host. >> > >>>> >> > >>>> > >> > >>>> > It ends up being kind of weird to pass a size into the connect >> > method, >> > >>>> as >> > >>>> > you've noted. >> > >>>> > >> > >>>> > What if we essentially left the attachVolume and detachVolume >> > methods >> > >>>> alone >> > >>>> > (as in how they were before my changes)? We could have >> > >>>> > VolumeApiServiceImpl, before sending the AttachCommand, detect if >> > the >> > >>>> > storage in question is managed. If it is, VolumeApiServiceImpl >> could >> > >>>> send a >> > >>>> > CreateObjectCommand. I would then implement createPhysicalDisk to >> > >>>> connect >> > >>>> > my iSCSI target and create a KVMPhysicalDisk. >> > >>>> > >> > >>>> > On the reverse side, VolumeApiServiceImpl, after sending the >> > >>>> DetachCommand, >> > >>>> > could detect if the storage in question is managed. If it is, >> > >>>> > VolumeApiServiceImpl could send a DeleteCommand. I would then >> > implement >> > >>>> the >> > >>>> > deletePhysicalDisk method to disconnect my iSCSI session. >> > >>>> > >> > >>>> > What do you think? >> > >>>> >> > >>>> Maybe I'm just confused, but I thought the create and delete on the >> > >>>> KVM side only apply to the default storage plugin, which has to >> pass >> > >>>> everything on the agent. I thought the creation/deletion of LUNs >> > >>>> occured via createAsync and deleteAsync in your plugin. >> > >>>> >> > >>>> > >> > >>>> > >> > >>>> > On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen < >> > shadow...@gmail.com >> > >>>> >wrote: >> > >>>> > >> > >>>> >> Ok, I've got our plugin working against 4.2. Tested start vm, >> stop >> > vm, >> > >>>> >> migrate vm, attach volume, detach volume. Other functions that >> we >> > >>>> >> already had in our StorageAdaptor implementation, such as >> copying >> > >>>> >> templates to primary storage, just worked without any >> modification >> > >>>> >> from our 4.1 version. >> > >>>> >> >> > >>>> >> I'll post a patch to reviewboard with the applicable changes. I >> was >> > >>>> >> correct that attachVolume and dettachVolume only apply to >> > >>>> >> adding/removing disks from running VMs, so there were some more >> > >>>> >> changes to LibvirtComputingResource. I don't intend for this >> patch >> > to >> > >>>> >> be applied (for one it's against 4.2), but I want you to take a >> > look >> > >>>> >> and see if it will work for you as well. If it does, then it's a >> > good >> > >>>> >> indicator that it should work for other plugins too, or if it >> > needs to >> > >>>> >> be tweaked we can work it out. >> > >>>> >> >> > >>>> >> The gist is that we needed a connectPhysicalDisk call that can >> > accept >> > >>>> >> the pool/volume info (which we've discussed), but also a >> version of >> > >>>> >> connectPhysicalDisk that can take a vm specification >> > >>>> >> (VirtualMachineTO) and figure out which pools/disks are needed >> and >> > >>>> >> attach them. I largely copied the code we had custom inserted >> into >> > our >> > >>>> >> 4.1 and put it into KVMStoragePoolManager so that it will be >> > adaptor >> > >>>> >> agnostic. >> > >>>> >> >> > >>>> >> Same goes for disconnectPhysicalDisk. >> > >>>> >> >> > >>>> >> We also needed to pass the VirtualMachineTO in a few other >> places >> > like >> > >>>> >> MigrateCommand and StopCommand, it's otherwise hard to know >> which >> > >>>> >> storage adaptors we need to deal with when all we have is a vm >> > name or >> > >>>> >> something like that. >> > >>>> >> >> > >>>> >> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski >> > >>>> >> <mike.tutkow...@solidfire.com> wrote: >> > >>>> >> > Thanks for the clarification on how that works. >> > >>>> >> > >> > >>>> >> > Also, yeah, I think CHAP only grants you access to a volume. >> If >> > >>>> multiple >> > >>>> >> > hosts are using the CHAP credentials for a single volume, it's >> > up to >> > >>>> >> those >> > >>>> >> > hosts to make sure they don't step on each other's toes (and >> > this is >> > >>>> - to >> > >>>> >> > my understanding - how it works with XenServer and VMware). >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen < >> > >>>> shadow...@gmail.com >> > >>>> >> >wrote: >> > >>>> >> > >> > >>>> >> >> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski >> > >>>> >> >> <mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> > Maybe I should seek a little clarification as to how live >> > >>>> migration >> > >>>> >> works >> > >>>> >> >> > in CS with KVM. >> > >>>> >> >> > >> > >>>> >> >> > Before we do a live migration of VM 1 from Host 1 to Host >> 2, >> > do we >> > >>>> >> detach >> > >>>> >> >> > all disks from VM1? >> > >>>> >> >> > >> > >>>> >> >> > If so, then we're good to go there. >> > >>>> >> >> > >> > >>>> >> >> > I'm not as clear with HA. >> > >>>> >> >> >> > >>>> >> >> During live migration this is what we currently do in our >> > modified >> > >>>> >> >> 4.1, I'm not sure if the new framework is set up for this, >> but >> > it >> > >>>> >> >> should be made to do this if not. >> > >>>> >> >> >> > >>>> >> >> PrepareForMigrationCommand is called on destination host. In >> > >>>> >> >> PrepareForMigrationCommand we added a few lines to call >> > >>>> >> >> connectPhysicalDisk. This host connects the SAN disks to this >> > new >> > >>>> >> >> host, then creates a paused VM. >> > >>>> >> >> >> > >>>> >> >> MigrateCommand is called on the source host. This sends the >> > proper >> > >>>> >> >> command to transfer VM memory, then atomically cuts over to >> the >> > >>>> >> >> destination host. During this time, the disks are attached on >> > both >> > >>>> >> >> sides, but the VM is still the only thing that is using them, >> > and it >> > >>>> >> >> atomically cuts over. There's no caching on the host (qemu is >> > using >> > >>>> >> >> directio), so this is safe. >> > >>>> >> >> >> > >>>> >> >> After MigrateCommand completes it's VM passoff, we detach the >> > disks >> > >>>> >> >> before returning. >> > >>>> >> >> >> > >>>> >> >> > >> > >>>> >> >> > If VM 1 goes down because Host 1 crashes, is the >> attach-volume >> > >>>> command >> > >>>> >> >> > invoked as many times as need be (depending on how many >> > volumes >> > >>>> need >> > >>>> >> to >> > >>>> >> >> be >> > >>>> >> >> > attached) when VM 1 is restarted on Host 2? >> > >>>> >> >> >> > >>>> >> >> From what I can tell, the attachVolume and dettachVolume >> seemed >> > to >> > >>>> >> >> only be for attaching disks to existing, running VMs (i.e. >> > inserting >> > >>>> >> >> new XML into an existing domain definition). Normally when >> > >>>> starting a >> > >>>> >> >> vm from scratch the vm definition, along with any currently >> > attached >> > >>>> >> >> disks, is passed in to StartCommand (which would also be >> called >> > >>>> during >> > >>>> >> >> HA restart of a VM). In our 4.1 branch we also have a call to >> > >>>> >> >> connectPhysicalDisk here, where we loop through the disk >> > definitions >> > >>>> >> >> that were passed. >> > >>>> >> >> >> > >>>> >> >> Again, I should be able to flesh out the differences in 4.2 >> and >> > how >> > >>>> to >> > >>>> >> >> go about making this suitable for everyone in the coming >> days, >> > so >> > >>>> long >> > >>>> >> >> as you and anyone else writing plugins agree with the >> changes. >> > >>>> >> >> >> > >>>> >> >> These processes would make sure the disks are available on >> the >> > hosts >> > >>>> >> >> they need to be, but they don't really provide locking or >> ensure >> > >>>> that >> > >>>> >> >> only the necessary hosts can write to or see the disks at any >> > given >> > >>>> >> >> time. I don't think CHAP does that either. We currently >> generate >> > >>>> ACLs >> > >>>> >> >> via our SAN api during connectPhysicalDisk as a safety >> measure, >> > but >> > >>>> if >> > >>>> >> >> CloudStack is working properly it will be in charge of >> > controlling >> > >>>> >> >> that the disks are only being used where they should be. The >> > ACLs >> > >>>> just >> > >>>> >> >> ensure that if the VM somehow gets started in two different >> > places >> > >>>> >> >> (e.g. HA malfunction), only one of them will have access to >> the >> > >>>> disks. >> > >>>> >> >> >> > >>>> >> >> > >> > >>>> >> >> > >> > >>>> >> >> > On Fri, Sep 27, 2013 at 12:17 AM, Mike Tutkowski < >> > >>>> >> >> > mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> > >> > >>>> >> >> >> Let me clarify this line a bit: >> > >>>> >> >> >> >> > >>>> >> >> >> "We get away without this with XenServer and VMware >> because >> > - as >> > >>>> far >> > >>>> >> as >> > >>>> >> >> I >> > >>>> >> >> >> know - CS delegates HA and live migration to those >> clusters >> > and >> > >>>> they >> > >>>> >> >> handle >> > >>>> >> >> >> it most likely with some kind of locking protocol on the >> > >>>> >> SR/datastore." >> > >>>> >> >> >> >> > >>>> >> >> >> When I set up a XenServer or a VMware cluster, all nodes >> in >> > the >> > >>>> >> cluster >> > >>>> >> >> >> have the proper CHAP credentials and can access a shared >> > >>>> >> SR/datastore. >> > >>>> >> >> >> >> > >>>> >> >> >> HA and live migrations are OK here because the cluster >> > controls >> > >>>> >> access >> > >>>> >> >> to >> > >>>> >> >> >> the VDI on the SR (or VMDK on the datastore) with some >> kind >> > of >> > >>>> >> locking >> > >>>> >> >> >> protocol, I expect. >> > >>>> >> >> >> >> > >>>> >> >> >> Since KVM isn't really in a cluster outside of the >> CloudStack >> > >>>> world, >> > >>>> >> >> >> CloudStack has to handle these intricacies. >> > >>>> >> >> >> >> > >>>> >> >> >> In my case, I'm just presenting a raw disk to a VM on a >> KVM >> > host. >> > >>>> >> >> >> >> > >>>> >> >> >> In that case, HA and live migration depend on the storage >> > plug-in >> > >>>> >> being >> > >>>> >> >> >> able to grant and revoke access to the volume for hosts as >> > >>>> needed. >> > >>>> >> >> >> >> > >>>> >> >> >> I'd actually rather not even bother with CHAP when using >> KVM. >> > >>>> >> >> >> >> > >>>> >> >> >> >> > >>>> >> >> >> On Fri, Sep 27, 2013 at 12:06 AM, Mike Tutkowski < >> > >>>> >> >> >> mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >> >> > >>>> >> >> >>> Hey Marcus, >> > >>>> >> >> >>> >> > >>>> >> >> >>> I agree that CHAP does not fulfill the same role as >> fencing. >> > >>>> >> >> >>> >> > >>>> >> >> >>> I think we're going to have trouble with HA and live >> > migrations >> > >>>> on >> > >>>> >> KVM >> > >>>> >> >> if >> > >>>> >> >> >>> the storage plug-in doesn't have a way of knowing when a >> > host >> > >>>> wants >> > >>>> >> to >> > >>>> >> >> >>> access a volume and when we want to revoke access to that >> > >>>> volume. >> > >>>> >> >> >>> >> > >>>> >> >> >>> We get away without this with XenServer and VMware >> because >> > - as >> > >>>> far >> > >>>> >> as >> > >>>> >> >> I >> > >>>> >> >> >>> know - CS delegates HA and live migration to those >> clusters >> > and >> > >>>> they >> > >>>> >> >> handle >> > >>>> >> >> >>> it most likely with some kind of locking protocol on the >> > >>>> >> SR/datastore. >> > >>>> >> >> >>> >> > >>>> >> >> >>> As far as the path field is concerned, I should be able >> to >> > >>>> populate >> > >>>> >> it >> > >>>> >> >> >>> with the IQN of the volume in question. >> > >>>> >> >> >>> >> > >>>> >> >> >>> One problem I do see, however, is in the getPhysicalDisk >> > method. >> > >>>> >> >> >>> >> > >>>> >> >> >>> How are you envisioning I keep track of KVMPhysicalDisks >> > that I >> > >>>> >> create >> > >>>> >> >> in >> > >>>> >> >> >>> my connect method? >> > >>>> >> >> >>> >> > >>>> >> >> >>> Initially I was thinking I'd just keep them in a map. >> > Storage >> > >>>> pool >> > >>>> >> UUID >> > >>>> >> >> >>> to KVMPhysicalDisks. >> > >>>> >> >> >>> >> > >>>> >> >> >>> The problem is, how do I reconstruct that map if the >> agent >> > is >> > >>>> >> restarted >> > >>>> >> >> >>> (say the host crashes or is restarted). >> > >>>> >> >> >>> >> > >>>> >> >> >>> For storage pools, we get a message >> > (ModifyStoragePoolCommand) >> > >>>> from >> > >>>> >> the >> > >>>> >> >> >>> CS MS to tell us about all of the relevant storage pools. >> > With >> > >>>> this >> > >>>> >> >> >>> message, I can reconstruct my cache of storage pools. No >> > problem >> > >>>> >> there. >> > >>>> >> >> >>> >> > >>>> >> >> >>> But how will I know which volumes belong to a given >> storage >> > >>>> pool if >> > >>>> >> I >> > >>>> >> >> >>> have to rebuild that map? How will I even know which >> volumes >> > >>>> are in >> > >>>> >> >> use at >> > >>>> >> >> >>> all? >> > >>>> >> >> >>> >> > >>>> >> >> >>> Thanks >> > >>>> >> >> >>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> On Thu, Sep 26, 2013 at 11:37 PM, Marcus Sorensen < >> > >>>> >> shadow...@gmail.com >> > >>>> >> >> >wrote: >> > >>>> >> >> >>> >> > >>>> >> >> >>>> On Thu, Sep 26, 2013 at 10:23 PM, Mike Tutkowski >> > >>>> >> >> >>>> <mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >>>> > My comments are inline: >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> > On Thu, Sep 26, 2013 at 9:10 PM, Marcus Sorensen < >> > >>>> >> >> shadow...@gmail.com >> > >>>> >> >> >>>> >wrote: >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> >> Ok, let me digest this a bit. I got the github >> responses >> > >>>> but I'd >> > >>>> >> >> also >> > >>>> >> >> >>>> >> like to keep it on-list as well. >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> My initial thoughts are: >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> 1) I don't think disk format and size are necessary >> > >>>> parameters >> > >>>> >> for >> > >>>> >> >> >>>> >> connectPhysicalDisk, as the format can be determined >> by >> > the >> > >>>> >> >> adaptor, >> > >>>> >> >> >>>> >> and the size is set during the createAsync call in >> the >> > >>>> plugin. >> > >>>> >> We >> > >>>> >> >> >>>> >> really just need the disk path and the pool. >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> > [Mike T.] I agree, format is not needed. The only >> reason >> > I >> > >>>> have >> > >>>> >> size >> > >>>> >> >> >>>> passed >> > >>>> >> >> >>>> > in is because I need to create a KVMPhysicalDisk at >> the >> > end >> > >>>> of >> > >>>> >> the >> > >>>> >> >> >>>> connect >> > >>>> >> >> >>>> > method. Since this KVMPhysicalDisk is (in the code on >> > GitHub) >> > >>>> >> being >> > >>>> >> >> >>>> used to >> > >>>> >> >> >>>> > create our XML to attach the disk, I figured we'd need >> > that >> > >>>> size. >> > >>>> >> >> The >> > >>>> >> >> >>>> > KVMPhysicalDisk I produce from my implementation of >> > >>>> >> getPhysicalDisk >> > >>>> >> >> is >> > >>>> >> >> >>>> not >> > >>>> >> >> >>>> > as good because I don't know the size of the disk at >> this >> > >>>> point >> > >>>> >> (I >> > >>>> >> >> >>>> don't >> > >>>> >> >> >>>> > keep that information around). The reason I don't keep >> > that >> > >>>> info >> > >>>> >> >> >>>> around is >> > >>>> >> >> >>>> > because I don't have a good way to reproduce that >> info if >> > >>>> the KVM >> > >>>> >> >> host >> > >>>> >> >> >>>> is >> > >>>> >> >> >>>> > rebooted. We get info about storage pools in the form >> of >> > a >> > >>>> >> >> >>>> > ModifyStoragePoolCommand, but nothing about the >> volumes >> > >>>> inside of >> > >>>> >> >> the >> > >>>> >> >> >>>> > storage pool. >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> getPhysicalDisk is called in a bunch of places. I'd rely >> > on the >> > >>>> >> >> >>>> connectPhysicalDisk to only make the block device appear >> > on the >> > >>>> >> host, >> > >>>> >> >> >>>> and then getPhysicalDisk to find that block device and >> > fill out >> > >>>> >> things >> > >>>> >> >> >>>> like disk size and path (the real path to the local >> block >> > >>>> device) >> > >>>> >> for >> > >>>> >> >> >>>> passing and creating the disk XML. Trust me, unless >> things >> > have >> > >>>> >> >> >>>> changed significantly you need to be able to identify a >> > given >> > >>>> >> device >> > >>>> >> >> >>>> as a specific local disk by whatever you are setting the >> > 'path' >> > >>>> >> >> >>>> attribute to be. getPhysicalDisk will be called on your >> > >>>> storage >> > >>>> >> pool >> > >>>> >> >> >>>> with simply the path attribute, and via your adaptor >> with >> > the >> > >>>> pool >> > >>>> >> and >> > >>>> >> >> >>>> path. >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> So you may set path as some combination of iqn and >> > target/pool >> > >>>> >> info, >> > >>>> >> >> >>>> or if iqn is enough to identify a unique block device >> (in >> > >>>> >> >> >>>> /dev/disk/by-id maybe?) on a host then just use that. >> Path >> > just >> > >>>> >> needs >> > >>>> >> >> >>>> to be something, anything, to identify the disk on the >> > host. In >> > >>>> >> >> >>>> getPhysicalDisk, identify the local block device >> matching >> > the >> > >>>> info, >> > >>>> >> >> >>>> create a new KVMPhysicalDisk with the local path, size, >> > etc, >> > >>>> and >> > >>>> >> >> >>>> return it. >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> 2) I thought this access group thing you mention are >> the >> > >>>> >> >> grantAccess >> > >>>> >> >> >>>> >> and revokeAccess calls in the storage plugin 2.0 >> design >> > >>>> doc. Was >> > >>>> >> >> that >> > >>>> >> >> >>>> >> not implemented? >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> > [Mike T.] Yeah, as I mentioned in an e-mail way back, >> > those >> > >>>> >> methods >> > >>>> >> >> >>>> were >> > >>>> >> >> >>>> > never implemented in 4.2. I think you said you were >> > going to >> > >>>> get >> > >>>> >> >> around >> > >>>> >> >> >>>> > them not being implemented by keeping certain logic >> that >> > >>>> talks to >> > >>>> >> >> the >> > >>>> >> >> >>>> SAN >> > >>>> >> >> >>>> > in the agent. I don't think we want any >> > SolidFire-specific >> > >>>> code >> > >>>> >> in >> > >>>> >> >> the >> > >>>> >> >> >>>> > agent, however, so I can't go that route. If those >> > methods >> > >>>> do not >> > >>>> >> >> get >> > >>>> >> >> >>>> > implemented in 4.3, then I will need to use CHAP >> > credentials >> > >>>> for >> > >>>> >> KVM >> > >>>> >> >> >>>> (just >> > >>>> >> >> >>>> > like I did with XenServer and VMware). >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> I initially figured your StorageAdaptor implementation >> > would >> > >>>> be all >> > >>>> >> >> >>>> solidfire specific, just like the mgmt server side >> plugin >> > is. >> > >>>> If it >> > >>>> >> >> >>>> can be generic to all iscsi storage then that's great. I >> > agree >> > >>>> that >> > >>>> >> >> >>>> ideally the agent wouldn't be making API calls to your >> > SAN. I >> > >>>> don't >> > >>>> >> >> >>>> think it should be necessary given that you're not >> going to >> > >>>> use the >> > >>>> >> >> >>>> ACL route. I'm not sure CHAP fills the same purpose of >> > fencing. >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> I see you've added getters/setters for the attach >> cmd >> > to >> > >>>> pass >> > >>>> >> the >> > >>>> >> >> >>>> >> iscsi info you need. Would it perhaps be possible to >> > send a >> > >>>> >> details >> > >>>> >> >> >>>> >> Map<String, String> instead? Then any plugin >> implementer >> > >>>> could >> > >>>> >> >> attach >> > >>>> >> >> >>>> >> arbitrary data they need. So it might be >> > >>>> >> >> >>>> >> connectPhysicalDisk(StoragePoolType type, String >> > poolUuid, >> > >>>> >> String >> > >>>> >> >> >>>> >> volPath, Map<String, String> details)? I'll have to >> > look >> > >>>> and >> > >>>> >> see >> > >>>> >> >> >>>> >> where those cmd. attributes are set, ideally it >> would be >> > >>>> all the >> > >>>> >> >> way >> > >>>> >> >> >>>> >> back in the plugin to avoid custom code for every >> > adaptor >> > >>>> that >> > >>>> >> >> wants >> > >>>> >> >> >>>> >> to set details. >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> > [Mike T.] If I'm not using the volumes.path field for >> > >>>> anything, I >> > >>>> >> >> can >> > >>>> >> >> >>>> stick >> > >>>> >> >> >>>> > the IQN in volumes.path (as well as leaving it in >> > >>>> >> >> volumes.iscsi_name, >> > >>>> >> >> >>>> which >> > >>>> >> >> >>>> > is used elsewhere). That way we only have to ask for >> > >>>> getPath(). >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> Yeah, whatever it is that you'd need to find the right >> > block >> > >>>> device >> > >>>> >> >> >>>> should go in the path. If you look through >> > >>>> LibvirtComputingResource >> > >>>> >> >> >>>> you'll see stuff like this sprinkled around: >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> KVMPhysicalDisk volume = >> > >>>> >> >> primaryPool.getPhysicalDisk(cmd >> > >>>> >> >> >>>> .getVolumePath()); >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> or: >> > >>>> >> >> >>>> String volid = cmd.getPath(); >> > >>>> >> >> >>>> KVMPhysicalDisk vol = >> pool.getPhysicalDisk(volid); >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> or: >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> KVMPhysicalDisk physicalDisk = >> > >>>> >> >> >>>> _storagePoolMgr.getPhysicalDisk( store.getPoolType(), >> > >>>> >> >> >>>> store.getUuid(), >> > >>>> >> >> >>>> data.getPath()); >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> Maybe some of it is short-circuited by the new >> > >>>> KVMStorageProcessor, >> > >>>> >> >> >>>> but I'd still implement a working one, and then >> > attachVolume >> > >>>> can >> > >>>> >> call >> > >>>> >> >> >>>> getPhysicalDisk after connectPhysicalDisk, even on your >> > >>>> >> adaptor/pool. >> > >>>> >> >> >>>> >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> On Thu, Sep 26, 2013 at 7:35 PM, Mike Tutkowski >> > >>>> >> >> >>>> >> <mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >>>> >> > Also, if we went the non-CHAP route, before >> attaching >> > a >> > >>>> volume >> > >>>> >> >> to a >> > >>>> >> >> >>>> VM, >> > >>>> >> >> >>>> >> > we'd have to tell the plug-in to set up a volume >> > access >> > >>>> group. >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > When a volume is detached from a VM, we'd have to >> > tell the >> > >>>> >> >> plug-in >> > >>>> >> >> >>>> to >> > >>>> >> >> >>>> >> > delete the volume access group. >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > On Thu, Sep 26, 2013 at 7:32 PM, Mike Tutkowski < >> > >>>> >> >> >>>> >> > mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> >> I mention this is my comments on GitHub, as well, >> but >> > >>>> CHAP >> > >>>> >> info >> > >>>> >> >> is >> > >>>> >> >> >>>> >> >> associated with an account - not a storage pool. >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> Ideally we could do without CHAP info entirely if >> we >> > had >> > >>>> a >> > >>>> >> >> >>>> reliable way >> > >>>> >> >> >>>> >> to >> > >>>> >> >> >>>> >> >> tell the storage plug-in that a given host wants >> to >> > >>>> access a >> > >>>> >> >> given >> > >>>> >> >> >>>> >> volume. >> > >>>> >> >> >>>> >> >> In this case, my storage plug-in could create >> what we >> > >>>> call a >> > >>>> >> >> Volume >> > >>>> >> >> >>>> >> Access >> > >>>> >> >> >>>> >> >> Group on the SAN. It would essentially say, "The >> host >> > >>>> with >> > >>>> >> IQN >> > >>>> >> >> <x> >> > >>>> >> >> >>>> can >> > >>>> >> >> >>>> >> >> access the volume with IQN <y> without using CHAP >> > >>>> >> credentials." >> > >>>> >> >> Of >> > >>>> >> >> >>>> >> course >> > >>>> >> >> >>>> >> >> we'd need a way to revoke this privilege in the >> > event of >> > >>>> a >> > >>>> >> live >> > >>>> >> >> >>>> >> migration >> > >>>> >> >> >>>> >> >> of a VM. Right now, I do not believe such a >> facility >> > is >> > >>>> >> >> supported >> > >>>> >> >> >>>> with >> > >>>> >> >> >>>> >> the >> > >>>> >> >> >>>> >> >> storage plug-ins. >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> On Thu, Sep 26, 2013 at 4:56 PM, Marcus Sorensen < >> > >>>> >> >> >>>> shadow...@gmail.com >> > >>>> >> >> >>>> >> >wrote: >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >>> Looking at your code, is the chap info stored >> with >> > the >> > >>>> pool, >> > >>>> >> >> so we >> > >>>> >> >> >>>> >> >>> could pass the pool to the adaptor? That would be >> > more >> > >>>> >> >> agnostic, >> > >>>> >> >> >>>> >> >>> anyone implementing a plugin could pull the >> > specifics >> > >>>> they >> > >>>> >> need >> > >>>> >> >> >>>> for >> > >>>> >> >> >>>> >> >>> their stuff out of the pool on the adaptor side, >> > rather >> > >>>> than >> > >>>> >> >> >>>> creating >> > >>>> >> >> >>>> >> >>> custom signatures. >> > >>>> >> >> >>>> >> >>> >> > >>>> >> >> >>>> >> >>> Also, I think we may want to consider >> implementing >> > >>>> >> >> >>>> connect/disconnect >> > >>>> >> >> >>>> >> >>> as just dummy methods in LibvirtStorageAdaptor, >> so >> > we >> > >>>> don't >> > >>>> >> >> have >> > >>>> >> >> >>>> to be >> > >>>> >> >> >>>> >> >>> picky about which adaptors/types in every single >> > place >> > >>>> we >> > >>>> >> may >> > >>>> >> >> >>>> want to >> > >>>> >> >> >>>> >> >>> connect/disconnect (in 4.1 there were several, >> I'm >> > not >> > >>>> sure >> > >>>> >> if >> > >>>> >> >> >>>> >> >>> everything goes through this in 4.2). We can just >> > call >> > >>>> >> >> >>>> >> >>> adaptor.connectPhysicalDisk and the adaptor can >> > decide >> > >>>> if it >> > >>>> >> >> >>>> needs to >> > >>>> >> >> >>>> >> >>> do anything. >> > >>>> >> >> >>>> >> >>> >> > >>>> >> >> >>>> >> >>> Comments are attached to your commit, I just >> wanted >> > to >> > >>>> echo >> > >>>> >> >> them >> > >>>> >> >> >>>> here >> > >>>> >> >> >>>> >> >>> on-list. >> > >>>> >> >> >>>> >> >>> >> > >>>> >> >> >>>> >> >>> On Thu, Sep 26, 2013 at 4:32 PM, Mike Tutkowski >> > >>>> >> >> >>>> >> >>> <mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >>>> >> >>> > Oh, SnapshotTestWithFakeData is just modified >> > because >> > >>>> the >> > >>>> >> >> code >> > >>>> >> >> >>>> wasn't >> > >>>> >> >> >>>> >> >>> > building until I corrected this. It has nothing >> > >>>> really to >> > >>>> >> do >> > >>>> >> >> >>>> with my >> > >>>> >> >> >>>> >> >>> real >> > >>>> >> >> >>>> >> >>> > changes. >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> > On Thu, Sep 26, 2013 at 4:31 PM, Mike >> Tutkowski < >> > >>>> >> >> >>>> >> >>> > mike.tutkow...@solidfire.com> wrote: >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> >> Hey Marcus, >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> I implemented your recommendations regarding >> > adding >> > >>>> >> connect >> > >>>> >> >> and >> > >>>> >> >> >>>> >> >>> disconnect >> > >>>> >> >> >>>> >> >>> >> methods. It is not yet checked in (as you >> know, >> > >>>> having >> > >>>> >> >> trouble >> > >>>> >> >> >>>> with >> > >>>> >> >> >>>> >> my >> > >>>> >> >> >>>> >> >>> KVM >> > >>>> >> >> >>>> >> >>> >> environment), but it is on GitHub here: >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> >> > >>>> >> >> >> > >>>> >> >> > >>>> >> > >> https://github.com/mike-tutkowski/incubator-cloudstack/commit/f2897c65689012e6157c0a0c2ed7e5355900c59a >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> Please let me know if you have any more >> comments. >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> Thanks! >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> On Thu, Sep 26, 2013 at 4:05 PM, Marcus >> Sorensen >> > < >> > >>>> >> >> >>>> >> shadow...@gmail.com >> > >>>> >> >> >>>> >> >>> >wrote: >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >>> Mike, everyone, >> > >>>> >> >> >>>> >> >>> >>> As I've mentioned on the board, I'm >> working >> > on >> > >>>> >> getting >> > >>>> >> >> our >> > >>>> >> >> >>>> own >> > >>>> >> >> >>>> >> >>> >>> internal KVM storage plugin working on 4.2. >> In >> > the >> > >>>> >> >> interest of >> > >>>> >> >> >>>> >> making >> > >>>> >> >> >>>> >> >>> >>> it forward compatible, I just wanted to >> confirm >> > >>>> what you >> > >>>> >> >> were >> > >>>> >> >> >>>> doing >> > >>>> >> >> >>>> >> >>> >>> with the solidfire plugin as far as attaching >> > your >> > >>>> iscsi >> > >>>> >> >> >>>> LUNs. We >> > >>>> >> >> >>>> >> had >> > >>>> >> >> >>>> >> >>> >>> discussed a new connectPhysicalDisk method >> for >> > the >> > >>>> >> >> >>>> StorageAdaptor >> > >>>> >> >> >>>> >> >>> >>> class, something perhaps like: >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >>> public boolean connectPhysicalDisk(String >> > >>>> volumeUuid, >> > >>>> >> >> >>>> >> KVMStoragePool >> > >>>> >> >> >>>> >> >>> >>> pool); >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >>> then added to KVMStoragePoolManager: >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >>> public boolean >> > connectPhysicalDisk(StoragePoolType >> > >>>> type, >> > >>>> >> >> >>>> String >> > >>>> >> >> >>>> >> >>> >>> poolUuid, String volPath) { >> > >>>> >> >> >>>> >> >>> >>> StorageAdaptor adaptor = >> > >>>> >> getStorageAdaptor(type); >> > >>>> >> >> >>>> >> >>> >>> KVMStoragePool pool = >> > >>>> >> >> >>>> adaptor.getStoragePool(poolUuid); >> > >>>> >> >> >>>> >> >>> >>> return >> > adaptor.connectPhysicalDisk(volPath, >> > >>>> >> pool); >> > >>>> >> >> >>>> >> >>> >>> } >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >>> Something similar to this for disconnect as >> > well. >> > >>>> Then >> > >>>> >> in >> > >>>> >> >> the >> > >>>> >> >> >>>> >> >>> >>> KVMStorageProcessor these can be called as >> > needed >> > >>>> for >> > >>>> >> >> >>>> >> attach/detach. >> > >>>> >> >> >>>> >> >>> >>> We can probably stub out one in >> > >>>> LibvirtStorageAdaptor so >> > >>>> >> >> >>>> there's no >> > >>>> >> >> >>>> >> >>> >>> need to switch or if/else for pool types, for >> > >>>> example in >> > >>>> >> >> >>>> >> >>> >>> KVMStorageProcessor.attachVolume. >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >>> I have debated on whether or not it should >> just >> > be >> > >>>> >> rolled >> > >>>> >> >> into >> > >>>> >> >> >>>> >> >>> >>> getPhysicalDisk, having it connect the disk >> if >> > it's >> > >>>> not >> > >>>> >> >> >>>> already >> > >>>> >> >> >>>> >> >>> >>> connected. getPhysicalDisk is called a lot, >> and >> > I'm >> > >>>> not >> > >>>> >> >> sure >> > >>>> >> >> >>>> it >> > >>>> >> >> >>>> >> always >> > >>>> >> >> >>>> >> >>> >>> needs to connect the disk when it does. In >> past >> > >>>> >> iterations >> > >>>> >> >> >>>> >> >>> >>> getPhysicalDisk has simply spoken to our SAN >> > api and >> > >>>> >> >> returned >> > >>>> >> >> >>>> the >> > >>>> >> >> >>>> >> disk >> > >>>> >> >> >>>> >> >>> >>> details, nothing more. So it seemed more >> > flexible >> > >>>> and >> > >>>> >> >> >>>> granular to >> > >>>> >> >> >>>> >> do >> > >>>> >> >> >>>> >> >>> >>> the connectPhysicalDisk (we have one now in >> our >> > 4.1 >> > >>>> >> >> version). >> > >>>> >> >> >>>> >> >>> >>> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> >> -- >> > >>>> >> >> >>>> >> >>> >> *Mike Tutkowski* >> > >>>> >> >> >>>> >> >>> >> *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>>> >> >>> >> e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>>> >> >>> >> o: 303.746.7302 >> > >>>> >> >> >>>> >> >>> >> Advancing the way the world uses the cloud< >> > >>>> >> >> >>>> >> >>> >> http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >>>> >> >>> >> *™* >> > >>>> >> >> >>>> >> >>> >> >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> > >> > >>>> >> >> >>>> >> >>> > -- >> > >>>> >> >> >>>> >> >>> > *Mike Tutkowski* >> > >>>> >> >> >>>> >> >>> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>>> >> >>> > e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>>> >> >>> > o: 303.746.7302 >> > >>>> >> >> >>>> >> >>> > Advancing the way the world uses the >> > >>>> >> >> >>>> >> >>> > cloud< >> > >>>> http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >>>> >> >>> > *™* >> > >>>> >> >> >>>> >> >>> >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> >> -- >> > >>>> >> >> >>>> >> >> *Mike Tutkowski* >> > >>>> >> >> >>>> >> >> *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>>> >> >> e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>>> >> >> o: 303.746.7302 >> > >>>> >> >> >>>> >> >> Advancing the way the world uses the cloud< >> > >>>> >> >> >>>> >> http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >>>> >> >> *™* >> > >>>> >> >> >>>> >> >> >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > >> > >>>> >> >> >>>> >> > -- >> > >>>> >> >> >>>> >> > *Mike Tutkowski* >> > >>>> >> >> >>>> >> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>>> >> > e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>>> >> > o: 303.746.7302 >> > >>>> >> >> >>>> >> > Advancing the way the world uses the >> > >>>> >> >> >>>> >> > cloud< >> > http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >>>> >> > *™* >> > >>>> >> >> >>>> >> >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> > >> > >>>> >> >> >>>> > -- >> > >>>> >> >> >>>> > *Mike Tutkowski* >> > >>>> >> >> >>>> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>>> > e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>>> > o: 303.746.7302 >> > >>>> >> >> >>>> > Advancing the way the world uses the >> > >>>> >> >> >>>> > cloud< >> http://solidfire.com/solution/overview/?video=play >> > > >> > >>>> >> >> >>>> > *™* >> > >>>> >> >> >>>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> -- >> > >>>> >> >> >>> *Mike Tutkowski* >> > >>>> >> >> >>> *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >>> e: mike.tutkow...@solidfire.com >> > >>>> >> >> >>> o: 303.746.7302 >> > >>>> >> >> >>> Advancing the way the world uses the cloud< >> > >>>> >> >> http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >>> *™* >> > >>>> >> >> >>> >> > >>>> >> >> >> >> > >>>> >> >> >> >> > >>>> >> >> >> >> > >>>> >> >> >> -- >> > >>>> >> >> >> *Mike Tutkowski* >> > >>>> >> >> >> *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> >> e: mike.tutkow...@solidfire.com >> > >>>> >> >> >> o: 303.746.7302 >> > >>>> >> >> >> Advancing the way the world uses the cloud< >> > >>>> >> >> http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> >> *™* >> > >>>> >> >> >> >> > >>>> >> >> > >> > >>>> >> >> > >> > >>>> >> >> > >> > >>>> >> >> > -- >> > >>>> >> >> > *Mike Tutkowski* >> > >>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> >> > e: mike.tutkow...@solidfire.com >> > >>>> >> >> > o: 303.746.7302 >> > >>>> >> >> > Advancing the way the world uses the >> > >>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play> >> > >>>> >> >> > *™* >> > >>>> >> >> >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > -- >> > >>>> >> > *Mike Tutkowski* >> > >>>> >> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> >> > e: mike.tutkow...@solidfire.com >> > >>>> >> > o: 303.746.7302 >> > >>>> >> > Advancing the way the world uses the >> > >>>> >> > cloud<http://solidfire.com/solution/overview/?video=play> >> > >>>> >> > *™* >> > >>>> >> >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > -- >> > >>>> > *Mike Tutkowski* >> > >>>> > *Senior CloudStack Developer, SolidFire Inc.* >> > >>>> > e: mike.tutkow...@solidfire.com >> > >>>> > o: 303.746.7302 >> > >>>> > Advancing the way the world uses the >> > >>>> > cloud<http://solidfire.com/solution/overview/?video=play> >> > >>>> > *™* >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> *Mike Tutkowski* >> > >>> *Senior CloudStack Developer, SolidFire Inc.* >> > >>> e: mike.tutkow...@solidfire.com >> > >>> o: 303.746.7302 >> > >>> Advancing the way the world uses the cloud< >> > http://solidfire.com/solution/overview/?video=play> >> > >>> *™* >> > >>> >> > >> >> > >> >> > >> >> > >> -- >> > >> *Mike Tutkowski* >> > >> *Senior CloudStack Developer, SolidFire Inc.* >> > >> e: mike.tutkow...@solidfire.com >> > >> o: 303.746.7302 >> > >> Advancing the way the world uses the >> > >> cloud<http://solidfire.com/solution/overview/?video=play> >> > >> *™* >> > >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the >> cloud<http://solidfire.com/solution/overview/?video=play> >> *™* >> >
