Re: backups of bhyve images
On 12/10/14 11:40 AM, Pete Wright wrote: On 12/10/14 11:29, John-Mark Gurney wrote: Pete Wright wrote this message on Mon, Dec 08, 2014 at 15:44 -0800: On 12/08/14 15:30, Craig Rodrigues wrote: (3) When you want to backup the VM, do a "zfs snapshot" take take a snapshot of the ZFS zvol. will this ensure that your zvol is consistent, or rather will the filesystem overlaid on the zvol device be ensured it is consistent when the hypervisor issues a snapshot command? That's the beauty of FreeBSD... UFS provides this w/ soft updates, and ZFS does this through COW... In both cases, as far as I understand it, it is safe to snapshot the FS... hrm not sure I think that is a reliable methodology. You'll be forced to recover you VM's filesystem at best, and potentially have corrupt data if blocks were still sitting in cache and had not returned a filehande. My guess is that this method will work with out issues %80-%90 of the time - but you still expose yourself to possible data loss or corruption. I think you and others have mentioned the proper way to do this - snapshot the guess filesystem from with-in the guest VM itself. cheers, -pete Hence why in the original email I wrote: *option 3 is the least safe imo as you can wind up with filesystem "angry".* in case 1 and 2 you'll have UFS snapshots that should be "OK" to restore from. in case 4 you are also doing snapshot, but you switch to ZFS. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
Pete Wright wrote this message on Wed, Dec 10, 2014 at 11:40 -0800: > On 12/10/14 11:29, John-Mark Gurney wrote: > > Pete Wright wrote this message on Mon, Dec 08, 2014 at 15:44 -0800: > >> > >> > >> On 12/08/14 15:30, Craig Rodrigues wrote: > >>> (3) When you want to backup the VM, do a "zfs snapshot" take take a > >>> snapshot of the ZFS zvol. > >> > >> will this ensure that your zvol is consistent, or rather will the > >> filesystem overlaid on the zvol device be ensured it is consistent when > >> the hypervisor issues a snapshot command? > > > > That's the beauty of FreeBSD... UFS provides this w/ soft updates, and > > ZFS does this through COW... In both cases, as far as I understand it, > > it is safe to snapshot the FS... > > hrm not sure I think that is a reliable methodology. You'll be forced > to recover you VM's filesystem at best, and potentially have corrupt > data if blocks were still sitting in cache and had not returned a > filehande. My guess is that this method will work with out issues > %80-%90 of the time - but you still expose yourself to possible data > loss or corruption. If the application(s) do not properly handle data ordering issues itself (making sure previous writes are completed using fsync), then that application(s) will have issues if you suffer power loss at the wrong moment, and you'll still have issues... > I think you and others have mentioned the proper way to do this - > snapshot the guess filesystem from with-in the guest VM itself. Doing it in the VM is only marginally better... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
On 12/10/14 11:29, John-Mark Gurney wrote: > Pete Wright wrote this message on Mon, Dec 08, 2014 at 15:44 -0800: >> >> >> On 12/08/14 15:30, Craig Rodrigues wrote: >>> (3) When you want to backup the VM, do a "zfs snapshot" take take a >>> snapshot of the ZFS zvol. >> >> will this ensure that your zvol is consistent, or rather will the >> filesystem overlaid on the zvol device be ensured it is consistent when >> the hypervisor issues a snapshot command? > > That's the beauty of FreeBSD... UFS provides this w/ soft updates, and > ZFS does this through COW... In both cases, as far as I understand it, > it is safe to snapshot the FS... > hrm not sure I think that is a reliable methodology. You'll be forced to recover you VM's filesystem at best, and potentially have corrupt data if blocks were still sitting in cache and had not returned a filehande. My guess is that this method will work with out issues %80-%90 of the time - but you still expose yourself to possible data loss or corruption. I think you and others have mentioned the proper way to do this - snapshot the guess filesystem from with-in the guest VM itself. cheers, -pete -- Pete Wright p...@nomadlogic.org twitter => @nomadlogicLA ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
Pete Wright wrote this message on Mon, Dec 08, 2014 at 15:44 -0800: > > > On 12/08/14 15:30, Craig Rodrigues wrote: > > (3) When you want to backup the VM, do a "zfs snapshot" take take a > > snapshot of the ZFS zvol. > > will this ensure that your zvol is consistent, or rather will the > filesystem overlaid on the zvol device be ensured it is consistent when > the hypervisor issues a snapshot command? That's the beauty of FreeBSD... UFS provides this w/ soft updates, and ZFS does this through COW... In both cases, as far as I understand it, it is safe to snapshot the FS... > it's been a while since i've done this - but IIRC on NetApp WAFL systems > (which are similar to zfs in terms of being a COW filesystem) you need > to ensure the guest filesystem is in a consistent state before issuing a > snapshot from it's parent. > > my data may be out of date since it's been several years since i've done > this though... With either UFS and ZFS, create a snapshot, and then do the copy of the image from the snapshot... This will get you a consistant copy of the image as if you had "powered off" the vm at that moment.. You'll still have to fsck a UFS file system, but that is to be expected... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
On 12/9/14, 7:44 AM, Pete Wright wrote: On 12/08/14 15:30, Craig Rodrigues wrote: (3) When you want to backup the VM, do a "zfs snapshot" take take a snapshot of the ZFS zvol. will this ensure that your zvol is consistent, or rather will the filesystem overlaid on the zvol device be ensured it is consistent when the hypervisor issues a snapshot command? it's been a while since i've done this - but IIRC on NetApp WAFL systems (which are similar to zfs in terms of being a COW filesystem) you need to ensure the guest filesystem is in a consistent state before issuing a snapshot from it's parent. my data may be out of date since it's been several years since i've done this though... there are three layers to consider. The client must have layed down a consistent set of blocks. Bhyve needs to have written them all to the file in question. The Host needs to have made sure that they are all consistent on the media. cheers, -pete ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
Hi John, Basically, what I want to do is to run accurate backups without shutting down and restarting the VM. Is this possible? If it isn't, I think the only alternative is to make a script that shuts the vm down, copies it, restarts the vm then runs its compression and backup-over-ssh routine. It's not possible in the general case to take a snapshot of the underlying image since as others have pointed out, it may not be consistent on disk since there are still data/metadata from the guest's filesystem that haven't made it out to disk. Commercial hypervisors provide guest tools that allow a filesystem quiesce/sync that lock-steps with external snapshotting machinery. However, one option that could be worth investigating is using ZFS in the guest with disk images backed by a zvol on the host. ZFS guarantees on-disk consistency, and a zvol provides instantaneous snapshots. Worst case is that some writes are not picked up at snapshot time, but that seems preferable to having to force a full sync in the guest. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
On 12/08/14 15:30, Craig Rodrigues wrote: > (3) When you want to backup the VM, do a "zfs snapshot" take take a > snapshot of the ZFS zvol. will this ensure that your zvol is consistent, or rather will the filesystem overlaid on the zvol device be ensured it is consistent when the hypervisor issues a snapshot command? it's been a while since i've done this - but IIRC on NetApp WAFL systems (which are similar to zfs in terms of being a COW filesystem) you need to ensure the guest filesystem is in a consistent state before issuing a snapshot from it's parent. my data may be out of date since it's been several years since i've done this though... cheers, -pete -- Pete Wright p...@nomadlogic.org twitter => @nomadlogicLA ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
On Dec 8, 2014, at 8:33 AM, John wrote: > > I have each image on its own (external to the image) ZFS filesystem. > Internally the image is using ufs if freebsd, ext3fs if linux. Would > using some ZFS method of duplication be better? In this case, would the > image become inconsistent? I recommend that you do the following: (1) Learn about ZFS zvol: http://zfsonlinux.org/example-zvol.html (2) Instead of creating a big disk image to hold your bhyve VM, use a ZFS zvol (3) When you want to backup the VM, do a "zfs snapshot" take take a snapshot of the ZFS zvol. (4) You can backup the zvol to another host by using "zfs send", and onthe receiving host, you do "zfs receive" The content of your VM can be any file system that you want (UFS, ext4, zfs), but you can backup the ZFS zvol using zfs commands. I've been doing it, and it works really nicely. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: backups of bhyve images
On Dec 8, 2014, at 8:33 AM, John wrote: > Hello list, > > I have a few questions about creating backups to be stored offsite. > > If a guest is running, can I compress the image without it becoming > inconsistent? If not, can it be copied without it becoming inconsistent? > By inconsistent, I mean will I see weird effects and broken files if the > backup is restored? Previously I've shut the VM down to avoid this, > before archiving. > > I have each image on its own (external to the image) ZFS filesystem. > Internally the image is using ufs if freebsd, ext3fs if linux. Would > using some ZFS method of duplication be better? In this case, would the > image become inconsistent? > > Basically, what I want to do is to run accurate backups without shutting > down and restarting the VM. Is this possible? If it isn't, I think the > only alternative is to make a script that shuts the vm down, copies it, > restarts the vm then runs its compression and backup-over-ssh routine. [[ adding f...@freebsd.org in case I'm wrong ]] If you are using UFS internally to the VMs then you'll need to send a snapshot that is consistent. If you are just copying the files out from under a running vm you are going to get spaghettios for a filesystem if you try to recover as you need a true point in time snapshot. I think a few better options would be: 1) Inside the VM create a UFS snapshot then dump that externally using tools. 2) Create the UFS snapshot, then make sure that the file/vzol is snapshotted using zfs. 3) Just snapshot the underlying zvol you've made the UFS image on and send that (you'll get a dirty FS on restore, but it *should* be recoverable with a simple fsck) 4) Use zfs internally to the vm and send/receive the internal zfs. option 3 is the least safe imo as you can wind up with filesystem "angry". in case 1 and 2 you'll have UFS snapshots that should be "OK" to restore from. in case 4 you are also doing snapshot, but you switch to ZFS. -Alfred ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
backups of bhyve images
Hello list, I have a few questions about creating backups to be stored offsite. If a guest is running, can I compress the image without it becoming inconsistent? If not, can it be copied without it becoming inconsistent? By inconsistent, I mean will I see weird effects and broken files if the backup is restored? Previously I've shut the VM down to avoid this, before archiving. I have each image on its own (external to the image) ZFS filesystem. Internally the image is using ufs if freebsd, ext3fs if linux. Would using some ZFS method of duplication be better? In this case, would the image become inconsistent? Basically, what I want to do is to run accurate backups without shutting down and restarting the VM. Is this possible? If it isn't, I think the only alternative is to make a script that shuts the vm down, copies it, restarts the vm then runs its compression and backup-over-ssh routine. thanks, -- John ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
