Re: [zfs-discuss] Snapshots and Data Loss
>-Original Message- >From: Ross Walker [mailto:rswwal...@gmail.com] >Sent: Friday, April 23, 2010 7:08 AM >> >> We are currently porting over our existing Learning Lab Infrastructure >> platform from MS Virtual Server to VBox + ZFS. When students >> connect into >> their lab environment it dynamically creates their VMs and load >> balances >> them across physical servers. > >You can also check out OpenSolaris' Xen implementation, which if you >use Linux VMs will allow PV VMs as well as hardware assisted full >virtualized Windows VMs. There are public domain Windows Xen drivers >out there. > >The advantage of using Xen is it's VM live migration and XMLRPC >management API. As it runs as a bare metal hypervisor it also allows >fine granularity of CPU schedules, between guests and the host VM, but >unfortunately it's remote display technology leaves something to be >desired. For Windows VMs I use the built-in remote desktop, and for >Linux VMs I use XDM and use something like 'thinstation' on the client >side. > >-Ross Hi Ross. We decided to use a hosted hypervisor like VirtualBox because our customers use a variety of different platforms and they don't run high end workloads. We want a lot of flexibility on configuration and OS support (both host and guest). Remote control is a challenge. In our scenario students are going to spin up exact copies of a lab environment and we need to isolate their machines in separate networks so you can't directly connect to the VMs. We don't know what Guest OS they are going to run so we can't rely on the guest OS remote control tools. We want students to be able to have "console" access and they need to be able to share it out with an instructor. We want students to be able to connect from any type of device. We don't want to rely on other connection broker software to coordinate access. VirtualBox is great because it provides console level access via RDP. RDP performs well enough and is pretty much on everything. This is probably getting a bit off topic now :) Geoff ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
On Apr 22, 2010, at 11:03 AM, Geoff Nordli wrote: From: Ross Walker [mailto:rswwal...@gmail.com] Sent: Thursday, April 22, 2010 6:34 AM On Apr 20, 2010, at 4:44 PM, Geoff Nordli wrote: If you combine the hypervisor and storage server and have students connect to the VMs via RDP or VNC or XDM then you will have the performance of local storage and even script VirtualBox to take a snapshot right after a save state. A lot less difficult to configure on the client side, and allows you to deploy thin clients instead of full desktops where you can get away with it. It also allows you to abstract the hypervisor from the client. Need a bigger storage server with lots of memory, CPU and storage though. Later, if need be, you can break out the disks to a storage appliance with an 8GB FC or 10Gbe iSCSI interconnect. Right, I am in the process now of trying to figure out what the load looks like with a central storage box and how ZFS needs to be configured to support that load. So far what I am seeing is very exciting :) We are currently porting over our existing Learning Lab Infrastructure platform from MS Virtual Server to VBox + ZFS. When students connect into their lab environment it dynamically creates their VMs and load balances them across physical servers. You can also check out OpenSolaris' Xen implementation, which if you use Linux VMs will allow PV VMs as well as hardware assisted full virtualized Windows VMs. There are public domain Windows Xen drivers out there. The advantage of using Xen is it's VM live migration and XMLRPC management API. As it runs as a bare metal hypervisor it also allows fine granularity of CPU schedules, between guests and the host VM, but unfortunately it's remote display technology leaves something to be desired. For Windows VMs I use the built-in remote desktop, and for Linux VMs I use XDM and use something like 'thinstation' on the client side. -Ross ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
>From: Ross Walker [mailto:rswwal...@gmail.com] >Sent: Thursday, April 22, 2010 6:34 AM > >On Apr 20, 2010, at 4:44 PM, Geoff Nordli wrote: > > >If you combine the hypervisor and storage server and have students >connect to the VMs via RDP or VNC or XDM then you will have the >performance of local storage and even script VirtualBox to take a >snapshot right after a save state. > >A lot less difficult to configure on the client side, and allows you >to deploy thin clients instead of full desktops where you can get away >with it. > >It also allows you to abstract the hypervisor from the client. > >Need a bigger storage server with lots of memory, CPU and storage >though. > >Later, if need be, you can break out the disks to a storage appliance >with an 8GB FC or 10Gbe iSCSI interconnect. > Right, I am in the process now of trying to figure out what the load looks like with a central storage box and how ZFS needs to be configured to support that load. So far what I am seeing is very exciting :) We are currently porting over our existing Learning Lab Infrastructure platform from MS Virtual Server to VBox + ZFS. When students connect into their lab environment it dynamically creates their VMs and load balances them across physical servers. Geoff ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
On Apr 20, 2010, at 4:44 PM, Geoff Nordli wrote: From: matthew patton [mailto:patto...@yahoo.com] Sent: Tuesday, April 20, 2010 12:54 PM Geoff Nordli wrote: With our particular use case we are going to do a "save state" on their virtual machines, which is going to write 100-400 MB per VM via CIFS or NFS, then we take a snapshot of the volume, which guarantees we get a consistent copy of their VM. maybe you left out a detail or two but I can't see how your ZFS snapshot is going to be consistent UNLESS every VM on that ZFS volume is prevented from doing any and all I/O from the time it finishes "save state" and you take your ZFS snapshot. If by "save state" you mean something akin to VMWare's disk snapshot, why would you even bother with a ZFS snapshot in addition? We are using VirtualBox as our hypervisor. When it does a save state it generates a memory file. The memory file plus the volume snapshot creates a consistent state. In our platform each student's VM points to a unique backend volume via iscsi using VBox's built-in iscsi initiator. So there is a one-to-one relationship between VM and Volume. Just for clarity, a single VM could have multiple disks attached to it. In that scenario, then a VM would have multiple volumes. end we could have maybe 20-30 VMs getting saved at the same time, which could mean several GB of data would need to get written in a short time frame and would need to get committed to disk. So it seems the best case would be to get those "save state" writes as sync and get them into a ZIL. That I/O pattern is vastly >32kb and so will hit the 'rust' ZIL (which ALWAYS exists) and if you were thinking an SSD would help you, I don't see any/much evidence it will buy you anything. If I set the logbias (b122) to latency, then it will direct all sync IO to the log device, even if it exceeds the zfs_immediate_write_sz threshold. If you combine the hypervisor and storage server and have students connect to the VMs via RDP or VNC or XDM then you will have the performance of local storage and even script VirtualBox to take a snapshot right after a save state. A lot less difficult to configure on the client side, and allows you to deploy thin clients instead of full desktops where you can get away with it. It also allows you to abstract the hypervisor from the client. Need a bigger storage server with lots of memory, CPU and storage though. Later, if need be, you can break out the disks to a storage appliance with an 8GB FC or 10Gbe iSCSI interconnect. -Ross ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
>From: matthew patton [mailto:patto...@yahoo.com] >Sent: Tuesday, April 20, 2010 12:54 PM > >Geoff Nordli wrote: > >> With our particular use case we are going to do a "save >> state" on their >> virtual machines, which is going to write 100-400 MB >> per VM via CIFS or >> NFS, then we take a snapshot of the volume, which >> guarantees we get a >> consistent copy of their VM. > >maybe you left out a detail or two but I can't see how your ZFS snapshot >is going to be consistent UNLESS every VM on that ZFS volume is >prevented from doing any and all I/O from the time it finishes "save >state" and you take your ZFS snapshot. > >If by "save state" you mean something akin to VMWare's disk snapshot, >why would you even bother with a ZFS snapshot in addition? > We are using VirtualBox as our hypervisor. When it does a save state it generates a memory file. The memory file plus the volume snapshot creates a consistent state. In our platform each student's VM points to a unique backend volume via iscsi using VBox's built-in iscsi initiator. So there is a one-to-one relationship between VM and Volume. Just for clarity, a single VM could have multiple disks attached to it. In that scenario, then a VM would have multiple volumes. >> end we could have >> maybe 20-30 VMs getting saved at the same time, which could >> mean several GB >> of data would need to get written in a short time frame and >> would need to >> get committed to disk. >> >> So it seems the best case would be to get those "save >> state" writes as sync >> and get them into a ZIL. > >That I/O pattern is vastly >32kb and so will hit the 'rust' ZIL (which >ALWAYS exists) and if you were thinking an SSD would help you, I don't >see any/much evidence it will buy you anything. > > If I set the logbias (b122) to latency, then it will direct all sync IO to the log device, even if it exceeds the zfs_immediate_write_sz threshold. Have great day! Geoff ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
Geoff Nordli wrote: > With our particular use case we are going to do a "save > state" on their > virtual machines, which is going to write 100-400 MB > per VM via CIFS or > NFS, then we take a snapshot of the volume, which > guarantees we get a > consistent copy of their VM. maybe you left out a detail or two but I can't see how your ZFS snapshot is going to be consistent UNLESS every VM on that ZFS volume is prevented from doing any and all I/O from the time it finishes "save state" and you take your ZFS snapshot. If by "save state" you mean something akin to VMWare's disk snapshot, why would you even bother with a ZFS snapshot in addition? > end we could have > maybe 20-30 VMs getting saved at the same time, which could > mean several GB > of data would need to get written in a short time frame and > would need to > get committed to disk. > > So it seems the best case would be to get those "save > state" writes as sync > and get them into a ZIL. That I/O pattern is vastly >32kb and so will hit the 'rust' ZIL (which ALWAYS exists) and if you were thinking an SSD would help you, I don't see any/much evidence it will buy you anything. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
>From: Richard Elling [mailto:richard.ell...@gmail.com] >Sent: Monday, April 19, 2010 10:17 PM > >Hi Geoff, >The Canucks have already won their last game of the season :-) >more below... Hi Richard, I didn't watch the game last night, but obviously Vancouver better pick up their socks or they will be joining San Jose on the sidelines. With Ottawa, Montreal on the way out too, it could be a tough spring for Canadian hockey fans. > >On Apr 18, 2010, at 11:21 PM, Geoff Nordli wrote: > >> Hi Richard. >> >> Can you explain in a little bit more detail how this process works? >> >> Let's say you are writing from a remote virtual machine via an iscsi >target >> set for async writes and I take a snapshot of that volume. >> >> Are you saying any outstanding writes for that volume will need to be >> written to disk before the snapshot happens? > >Yes. That is interesting, so if your system is under write load and you are doing snapshots it could lead to problems. I was thinking writes wouldn't be an issue because they would be lazily written. With our particular use case we are going to do a "save state" on their virtual machines, which is going to write 100-400 MB per VM via CIFS or NFS, then we take a snapshot of the volume, which guarantees we get a consistent copy of their VM. When a class came to and end we could have maybe 20-30 VMs getting saved at the same time, which could mean several GB of data would need to get written in a short time frame and would need to get committed to disk. So it seems the best case would be to get those "save state" writes as sync and get them into a ZIL. Would you agree with that? > >I'm glad you enjoyed it. I'm looking forward to Vegas next week and >there >are some seats still open. > -- richard I would love to go to Vegas, but I need to work on getting our new product out the door. Enjoy yourself in Vegas next week! Geoff ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
Hi Geoff, The Canucks have already won their last game of the season :-) more below... On Apr 18, 2010, at 11:21 PM, Geoff Nordli wrote: >> On Apr 13, 2010, at 5:22 AM, Tony MacDoodle wrote: >> >>> I was wondering if any data was lost while doing a snapshot on a >> running system? >> >> ZFS will not lose data during a snapshot. >> >>> Does it flush everything to disk or would some stuff be lost? >> >> Yes, all ZFS data will be committed to disk and then the snapshot >> is taken. >> > > Hi Richard. > > Can you explain in a little bit more detail how this process works? > > Let's say you are writing from a remote virtual machine via an iscsi target > set for async writes and I take a snapshot of that volume. > > Are you saying any outstanding writes for that volume will need to be > written to disk before the snapshot happens? Yes. > Setting the target to sync writes and using a ZIL might have better > performance, if you were doing a lot of snapshots. No. In this case we are referring to the sync(2) system call, which has nothing to do with how files are written. Unfortunately, the word "sync" is somewhat overloaded and frequently misapplied. > I know there is a potential to lose data with async set target, but the > virtual machines running on the system are just lab machines using > non-production data. There are use cases where this can be justified. > BTW, great Nexenta / ZFS class in Atlanta. Thanks for getting me on the > right track!! I'm glad you enjoyed it. I'm looking forward to Vegas next week and there are some seats still open. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
>On Apr 13, 2010, at 5:22 AM, Tony MacDoodle wrote: > >> I was wondering if any data was lost while doing a snapshot on a >running system? > >ZFS will not lose data during a snapshot. > >> Does it flush everything to disk or would some stuff be lost? > >Yes, all ZFS data will be committed to disk and then the snapshot >is taken. > Hi Richard. Can you explain in a little bit more detail how this process works? Let's say you are writing from a remote virtual machine via an iscsi target set for async writes and I take a snapshot of that volume. Are you saying any outstanding writes for that volume will need to be written to disk before the snapshot happens? Setting the target to sync writes and using a ZIL might have better performance, if you were doing a lot of snapshots. I know there is a potential to lose data with async set target, but the virtual machines running on the system are just lab machines using non-production data. BTW, great Nexenta / ZFS class in Atlanta. Thanks for getting me on the right track!! Geoff ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
On Apr 16, 2010, at 3:27 AM, Maurilio Longo wrote: > Richard, > >> Applications can take advantage of this and there are services available >> to integrate ZFS snapshots with Oracle databases, Windows clients, etc. > > which services are you referring to? Nexenta offers plugins to integrate Oracle databases, Windows clients, VMWare, Citrix XenServer, Microsoft Hyper-V, storage arrays, and multiple tiers of backup. http://www.nexenta.com/corp/nexentastor-overview/features-nexentastor -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
Richard, > Applications can take advantage of this and there are services available > to integrate ZFS snapshots with Oracle databases, Windows clients, etc. which services are you referring to? best regards. Maurilio. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
On Apr 13, 2010, at 5:22 AM, Tony MacDoodle wrote: > I was wondering if any data was lost while doing a snapshot on a running > system? ZFS will not lose data during a snapshot. > Does it flush everything to disk or would some stuff be lost? Yes, all ZFS data will be committed to disk and then the snapshot is taken. Applications can take advantage of this and there are services available to integrate ZFS snapshots with Oracle databases, Windows clients, etc. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Snapshots and Data Loss
A snapshot is a picture of the storage at a point in time so everything depends on the applications using the storage. If you're running a db with lots of cache it's probably a good idea to stop the service or force a flush to disk before taking the snapshot to ensure the integrity of the data. That said, rolling back to a snapshot would be roughly the same thing as stopping the application brutally and it's up to the application to evaluate the data. Some will handle it better than others. If you're running virtual machines the ideal solution is to take a VM snapshot, followed by the filesystem snapshot, then deleting the VM snashot. ZFS snapshots are very reliable but it's scope is limited to the disks that it manages so if there's unflushed data living at a higher level, ZFS won't be aware of it. Cordialement, Erik Ableson On 13 avr. 2010, at 14:22, Tony MacDoodle wrote: I was wondering if any data was lost while doing a snapshot on a running system? Does it flush everything to disk or would some stuff be lost? Thanks ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Snapshots and Data Loss
I was wondering if any data was lost while doing a snapshot on a running system? Does it flush everything to disk or would some stuff be lost? Thanks ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
