Re: [Openstack] lunr reference iSCSI target driver
Thanks for the explanation! On Mon, 2 May 2011 21:12:22 -0700 Michael Barton mike-launch...@weirdlooking.com wrote: What I've been playing with is having a manifest that contains hashes of (4mb) chunks for the volume's backups. When a user initiates a new backup, dm-snapshot does its thing and gives me a block device. I read and hash chunks from that block device and compare them to the manifest, uploading any that differ to Swift, then update the manifest with the new backup. Ah, you read and hash chunks from the whole block device. I misunderstood the point. The restore uses fuse with some basic bitmap logic to lazy load chunks from Swift on demand, plus a background thread that fills them in autonomously. I've been pretty happy with fuse's performance and stability (python-fuse that is; fusepy is really slow). You create a loopback device of the fuse file and set up a dm-snap device by using it, right? Tomo ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On Mon, 2 May 2011 21:19:43 -0700 Michael Barton mike-launch...@weirdlooking.com wrote: Oh, and I don't know if keeping track of dirty chunks so backups are less work is worth putting an indirection layer on top of volumes. I think that it depends on volume capacity and the frequency of snapshot creation. It's probably something we can discuss more fully and do some testing around later. Sure. I like the idea to try a simple design first (like something you explained). I'm really looking forward to trying the code. Tomo ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
Hello Mike, Tomo: I invite you to take a look at Livebackup for kvm: http://wiki.qemu.org/Features/Livebackup This is a feature that I am currently developing. In brief, livebackup enables full and incremental backups of running VMs. It has enhancements to qemu that enable it to keep an in-memory dirty blocks bitmap. The advantage of this approach is that you never have to calculate hashes in order to determine which blocks are dirty. Livebackup includes a rudimentary network protocol for transferring the dirty blocks over to a backup server. The client livebackup_client can be used to connect to the running VM, create a livebackup snapshot - this is a special purpose snapshot that tracks the state of all disks of a virtual machine at the time of snapshot, and then creates a COW file of blocks that the VM modifies during the snapshot. My intent is to develop this in qemu/qemu-kvm, then add support in libvirt if necessary, and then build support for this in Openstack. At that point Openstack will be in a position to offer three types of VMs: 1. ephemeral VMs with virtual disks that can be lost if the VM server's hard disk dies, 2. Fully highly available external storage (NAS or SAN) based VMs 3. Hourly backuped up VMs - livebackup_client running on another VM server can be utilized to backup the VM every hour, and the VM can be re-started from the backup if the primary VM server dies. Let me know what you think. Thanks, Jagane On 5/3/2011 3:05 AM, FUJITA Tomonori wrote: Thanks for the explanation! On Mon, 2 May 2011 21:12:22 -0700 Michael Bartonmike-launch...@weirdlooking.com wrote: What I've been playing with is having a manifest that contains hashes of (4mb) chunks for the volume's backups. When a user initiates a new backup, dm-snapshot does its thing and gives me a block device. I read and hash chunks from that block device and compare them to the manifest, uploading any that differ to Swift, then update the manifest with the new backup. Ah, you read and hash chunks from the whole block device. I misunderstood the point. The restore uses fuse with some basic bitmap logic to lazy load chunks from Swift on demand, plus a background thread that fills them in autonomously. I've been pretty happy with fuse's performance and stability (python-fuse that is; fusepy is really slow). You create a loopback device of the fuse file and set up a dm-snap device by using it, right? Tomo ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On 05/02/2011 01:46 PM, Chuck Thier wrote: This leads to another interesting question. While our reference implementation may not directly expose snapshot functionality, I imagine other storage implementations could want to. I'm interested to hear what use cases others would be interested in with snapshots. The obvious ones are things like creating a volume based on a snapshot, or rolling a volume back to a previous snapshot. I would like others' input here to shape what the snapshot API might look like. For RBD we only need the obvious ones: - create/list/remove snapshots - create volume from a snapshot - rollback to a snapshot ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
At Tue, 03 May 2011 12:19:50 -0700, Josh Durgin wrote: On 05/02/2011 01:46 PM, Chuck Thier wrote: This leads to another interesting question. While our reference implementation may not directly expose snapshot functionality, I imagine other storage implementations could want to. I'm interested to hear what use cases others would be interested in with snapshots. The obvious ones are things like creating a volume based on a snapshot, or rolling a volume back to a previous snapshot. I would like others' input here to shape what the snapshot API might look like. For RBD we only need the obvious ones: - create/list/remove snapshots - create volume from a snapshot - rollback to a snapshot These are the same for Sheepdog too. Thanks, Kazutaka ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On May 2, 2011, at 12:50 PM, FUJITA Tomonori wrote: Hello, Chuck told me at the conference that lunr team are still working on the reference iSCSI target driver design and a possible design might exploit device mapper snapshot feature. You're involved in the tgt project and it is the tgt project's purgative to add features as seen fit, but are you sure that you want to support this feature? I can see the advantages of having Swift support in tgt; However, is considerably complex. You're mapping a file-backed block device on a remote filesystem to a remote block storage protocol (iSCSI). Might it not be better, albeit less integrated, to improve the Swift FUSE driver and use a standard loopback device? This loopback device would be supported by the existing AIO driver in tgt. To clarify on the subject of snapshots: The naming of snapshots in Nova and their presence on disk is more confusing than it should be. There was some discussion of attempting to clarify the naming conventions. Storage snapshots as provided by the device mapper are copy-on-write block devices, while Nova will also refer to file-backing stores as snapshots. This latter definition is also used by EC2, but otherwise unknown and unused in the industry. I foresee that Lunr could use dm-snapshot to facilitate backups and/or to provide a COW against dm-zero for the purpose of preventing information disclosure. Both of these use-cases would be most applicable to local storage, whereas most iSCSI targets would provide these as part of their filer API and it would probably not be very useful at all for Swift. The only reasons to perform storage-snapshots/COW for iSCSI targets would be for relatively dumb filers that cannot do this internally, or for deployments where their smart filers have edge-cases preventing or breaking the use of these features. Regards, Eric Windisch e...@cloudscaling.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
You're involved in the tgt project and it is the tgt project's purgative to add features as seen fit, but are you sure that you want to support this feature? Major spell check fail: prerogative ;-) Regards, Eric Windisch e...@cloudscaling.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On Mon, May 2, 2011 at 2:45 PM, Eric Windisch e...@cloudscaling.com wrote: On May 2, 2011, at 12:50 PM, FUJITA Tomonori wrote: Hello, Chuck told me at the conference that lunr team are still working on the reference iSCSI target driver design and a possible design might exploit device mapper snapshot feature. To clarify on the subject of snapshots: The naming of snapshots in Nova and their presence on disk is more confusing than it should be. There was some discussion of attempting to clarify the naming conventions. Storage snapshots as provided by the device mapper are copy-on-write block devices, while Nova will also refer to file-backing stores as snapshots. This latter definition is also used by EC2, but otherwise unknown and unused in the industry. One of the things that was made very evident at the conference was the confusion around snapshots in Lunr. We were just talking about this in the office, and we are considering renaming snapshots in the Lunr API to backups to better indicate its intentions. Backups will be made from a volume, and a user will be able to create new volumes based on a backup. This leads to another interesting question. While our reference implementation may not directly expose snapshot functionality, I imagine other storage implementations could want to. I'm interested to hear what use cases others would be interested in with snapshots. The obvious ones are things like creating a volume based on a snapshot, or rolling a volume back to a previous snapshot. I would like others' input here to shape what the snapshot API might look like. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On Mon, 2 May 2011 15:45:20 -0400 Eric Windisch e...@cloudscaling.com wrote: You're involved in the tgt project and it is the tgt project's purgative to add features as seen fit, but are you sure that you want to support this feature? I'm the maintainer so I can add anything useful unless I upset the existing users. Swift support can be implemented as one of tgt I/O drivers so it's unlikely that the addition can hurt the existing code. I can see the advantages of having Swift support in tgt; However, is considerably complex. You're mapping a file-backed block device on a remote filesystem to a remote block storage protocol (iSCSI). Might it not be better, albeit less integrated, to improve the Swift FUSE driver and use a standard loopback device? This loopback device would be supported by the existing AIO driver in tgt. The implementation of the snapshot itself is complicated but integrating it into tgt isn't so complicated to me. tgt already supports things complicated than a file-backed block device. For example, it uses the own image format for the virtual tape library feature. It also support sending SCSI commands via sg char devices. Surely, FUSE is another possible option, I think. I heard that lunr team was thinking about the approach too. I foresee that Lunr could use dm-snapshot to facilitate backups and/or to provide a COW against dm-zero for the purpose of preventing information disclosure. As I wrote in the previous mail, the tricky part of the dm-snapshot approach is getting the delta of snaphosts (I assume that we want to store only deltas on Swift). dm-snapshot doesn't provide the user-space API to get the deltas. So Lunr needs to access to dm-snapshot volume directly. It's sorta backdoor approach (getting the information that Linux kernel doesn't provide to user space). As a Linux kernel developer, I would like to shout at people who do such :) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
Surely, FUSE is another possible option, I think. I heard that lunr team was thinking about the approach too. I'm concerned about the performance/stability of FUSE, but I'm not sure if using iSCSI is a significantly better option when the access is likely to be local. If I had to choose something in-between, I'd evaluate if NBD was any better of a solution. I expect there will be great demand for an implementation of a Swift as a block device client. Care should be made in deciding what will be the best-supported method/implementation. That said, you have an implementation, and that goes a long way versus the alternatives which don't currently exist. As I wrote in the previous mail, the tricky part of the dm-snapshot approach is getting the delta of snaphosts (I assume that we want to store only deltas on Swift). dm-snapshot doesn't provide the user-space API to get the deltas. So Lunr needs to access to dm-snapshot volume directly. It's sorta backdoor approach (getting the information that Linux kernel doesn't provide to user space). As a Linux kernel developer, I would like to shout at people who do such :) With dm-snapshot, the solution is to look at the device mapper table (via the device mapper API) and access the backend volume. I don't see why this is a bad solution. In fact, considering that the device mapper table could be arbitrarily complex and some backend volumes might be entirely virtual, i.e. dm-zero, this seems fairly reasonable to me. I really don't see at all how Swift-as-block-device relates at all to (storage) snapshots, other than the fact that this makes it possible to use Swift with dm-snapshot. Regards, Eric Windisch e...@cloudscaling.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
Is Swift as a Block device a real option? It looks to me that performance will be a big problem. Also how the three copies of Swift will be presented as iSCSI? Only one? Each one with its own iSCSI target? Who serialize the writes in this scenario? Nelson On Mon, May 2, 2011 at 6:11 PM, Eric Windisch e...@cloudscaling.com wrote: Surely, FUSE is another possible option, I think. I heard that lunr team was thinking about the approach too. I'm concerned about the performance/stability of FUSE, but I'm not sure if using iSCSI is a significantly better option when the access is likely to be local. If I had to choose something in-between, I'd evaluate if NBD was any better of a solution. I expect there will be great demand for an implementation of a Swift as a block device client. Care should be made in deciding what will be the best-supported method/implementation. That said, you have an implementation, and that goes a long way versus the alternatives which don't currently exist. As I wrote in the previous mail, the tricky part of the dm-snapshot approach is getting the delta of snaphosts (I assume that we want to store only deltas on Swift). dm-snapshot doesn't provide the user-space API to get the deltas. So Lunr needs to access to dm-snapshot volume directly. It's sorta backdoor approach (getting the information that Linux kernel doesn't provide to user space). As a Linux kernel developer, I would like to shout at people who do such :) With dm-snapshot, the solution is to look at the device mapper table (via the device mapper API) and access the backend volume. I don't see why this is a bad solution. In fact, considering that the device mapper table could be arbitrarily complex and some backend volumes might be entirely virtual, i.e. dm-zero, this seems fairly reasonable to me. I really don't see at all how Swift-as-block-device relates at all to (storage) snapshots, other than the fact that this makes it possible to use Swift with dm-snapshot. Regards, Eric Windisch e...@cloudscaling.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On Mon, 2 May 2011 21:11:22 -0400 Eric Windisch e...@cloudscaling.com wrote: I expect there will be great demand for an implementation of a Swift as a block device client. Care should be made in deciding what will Surely. I also modified tgt to simply store data on Swift. It doesn't work well due to Swift's week consistency. To implement a block device on the top of Swift, you need to sorta a log structure file system, that is, never over-write the existing objects. Updating the data means creating new objects. As I wrote in the previous mail, the tricky part of the dm-snapshot approach is getting the delta of snaphosts (I assume that we want to store only deltas on Swift). dm-snapshot doesn't provide the user-space API to get the deltas. So Lunr needs to access to dm-snapshot volume directly. It's sorta backdoor approach (getting the information that Linux kernel doesn't provide to user space). As a Linux kernel developer, I would like to shout at people who do such :) With dm-snapshot, the solution is to look at the device mapper table (via the device mapper API) and access the backend volume. I don't see why this is a bad solution. In fact, conside Hmm, seems that we aren't on the same page. I'm talking about how to get an incremental changes from the original volume. They are stored in the exception table in dm-snapshot. IIRC, the information doesn't exported to user space. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
We have no current plans to make an iSCSI target for swift. Not only would there be performance issues, but also consistency issues among other things. For Lunr, swift will only be a target for backups from block devices. I think some of this confusion stems from the confusion around snapshots, and Fujita's proposal would make an interesting case if we were going to use swift for more traditional snapshots. But since we are looking to use swift as backups for volumes, we will not need that type of functionality initially. Eric: Our current snapshot prototype uses FUSE since that is very simple to do in python, but we are also considering using a NBD (among other options). Once we have this nailed down a bit more, we will send out more details. -- Chuck On Mon, May 2, 2011 at 8:29 PM, Nelson Nahum nel...@zadarastorage.comwrote: Is Swift as a Block device a real option? It looks to me that performance will be a big problem. Also how the three copies of Swift will be presented as iSCSI? Only one? Each one with its own iSCSI target? Who serialize the writes in this scenario? Nelson On Mon, May 2, 2011 at 6:11 PM, Eric Windisch e...@cloudscaling.com wrote: Surely, FUSE is another possible option, I think. I heard that lunr team was thinking about the approach too. I'm concerned about the performance/stability of FUSE, but I'm not sure if using iSCSI is a significantly better option when the access is likely to be local. If I had to choose something in-between, I'd evaluate if NBD was any better of a solution. I expect there will be great demand for an implementation of a Swift as a block device client. Care should be made in deciding what will be the best-supported method/implementation. That said, you have an implementation, and that goes a long way versus the alternatives which don't currently exist. As I wrote in the previous mail, the tricky part of the dm-snapshot approach is getting the delta of snaphosts (I assume that we want to store only deltas on Swift). dm-snapshot doesn't provide the user-space API to get the deltas. So Lunr needs to access to dm-snapshot volume directly. It's sorta backdoor approach (getting the information that Linux kernel doesn't provide to user space). As a Linux kernel developer, I would like to shout at people who do such :) With dm-snapshot, the solution is to look at the device mapper table (via the device mapper API) and access the backend volume. I don't see why this is a bad solution. In fact, considering that the device mapper table could be arbitrarily complex and some backend volumes might be entirely virtual, i.e. dm-zero, this seems fairly reasonable to me. I really don't see at all how Swift-as-block-device relates at all to (storage) snapshots, other than the fact that this makes it possible to use Swift with dm-snapshot. Regards, Eric Windisch e...@cloudscaling.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
What I've been playing with is having a manifest that contains hashes of (4mb) chunks for the volume's backups. When a user initiates a new backup, dm-snapshot does its thing and gives me a block device. I read and hash chunks from that block device and compare them to the manifest, uploading any that differ to Swift, then update the manifest with the new backup. The restore uses fuse with some basic bitmap logic to lazy load chunks from Swift on demand, plus a background thread that fills them in autonomously. I've been pretty happy with fuse's performance and stability (python-fuse that is; fusepy is really slow). The NBD solution isn't really any different logic-wise from the fuse version, but requires a lot more wrangling of server and client processes. And actually we weren't too impressed with the performance of a basic NBD server in some (non-scientific) tests. All of this is sort of at the proof of concept stage at the moment. -- Michael Barton ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] lunr reference iSCSI target driver
On Mon, May 2, 2011 at 9:12 PM, Michael Barton mike-launch...@weirdlooking.com wrote: What I've been playing with is having a manifest that contains hashes of (4mb) chunks for the volume's backups. When a user initiates a new backup, dm-snapshot does its thing and gives me a block device. I read and hash chunks from that block device and compare them to the manifest, uploading any that differ to Swift, then update the manifest with the new backup. Oh, and I don't know if keeping track of dirty chunks so backups are less work is worth putting an indirection layer on top of volumes. It's probably something we can discuss more fully and do some testing around later. -- Mike ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
