Hardwre for KVM host
Hey all, at the moment I am looking for hardware for a KVM host, but on a few points I realy have problems to get a good hardware setup. My situation is the following: I need a server with minimal energy consumption, the ability of PCI Passtrough (so IOMMU is needed) and good support for Nested Virtualization (KVM, Xen and VMWare) So when I understood everything right I need a CPU and chipset on the board which supports IOMMU, which consumer boards do and which of them works fine with KVM? For the nested virtualization theme I read that EPTs in case of Intel and RVI in case of AMD is needed. Also I read that the AMD RVI would be better than the EPTs of Intel and bring a good speedup, wikipedia mentions a VMWare research whitepaper and Red Hat tests (http://en.wikipedia.org/wiki/Rapid_Virtualization_Indexing). Also it seems that the implementation of Nested Virualization is easier with the AMD RVIs than with the Intel EPTs. So which CPU with which chipset (baord) should i prefer to get this features. For the RAM I thought about 32GB because my experiences show that virtualization needs much RAM, would that be a good value for RAM when the VM host runs a few VMs with simple server services like (HTTP, SMTP etc.)? I want to run VMs with much I/O so I'm thinking about the hardware setup. I want to have a few TB of space so for example 4 3TB hard disks in RAID5, all virtualization solutions (Xen, VMWare) recommand hardware RAID. I also would prefer hardware RAID but would software RAID decrease the I/O performance very much? And would it be a difference to take software RAID with mdadm (for example on ext4 filesystem) or native RAID support like in BTRFS? Which modern hardware RAID controllers would be fully supported by linux and are there recommandations for a special interface (S-ATA, SAS etc.)? I don't have that much experiences with KVM-over-IP managment but I want to start using it. Are there special conditions which must be fulfilled by the board or operation system? And does someone here have experiences with KVM-over-IP managment and can recommand adapters or cards? To say something about the price I targeting: It would be greate to have costs between 800 and 1400€ for the host system without the costs of the KVM-over-IP console. When the price would be higher it would be ok when it don't is greater than 2500€ Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Secure migration of LVM based guests over WAN
Am 16.10.2012 12:10, schrieb Avi Kivity: On 10/16/2012 11:48 AM, Lukas Laukamp wrote: Am 16.10.2012 11:40, schrieb Avi Kivity: On 10/16/2012 11:12 AM, Lukas Laukamp wrote: Hey all, I have a question about a solution for migrate LVM based guests directly over the network. So the situation: Two KVM hosts with libvirt, multiple LVM based guests Want to do: Migrate a LVM based guest directly to the other host over an secure connection I know that migration is possible when the VM disks are stored on an NFS, GFS2 filer/cluster etc. So would it be possible to do an offline migration directly with netcat or something like that? If all you need is offline, you can use scp to copy each volume to the destination volume. Make sure the guests are shut down when you do that. It is also possible to do a live migration, but unless the destination and source are in the same IP subnet, the guests are going to lose connectivity. Hello Avi, so can I simply copy an logical volume to the path of the volume group with scp? Yes. Best to enable compression to avoid sending zero blocks. For the live migration theme, it would be no problem when the guests looses connectivity, how could be done a live migration? See the -b option to the migrate command. I will read a little bit about the live migration theme. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Secure migration of LVM based guests over WAN
Hey all, I have a question about a solution for migrate LVM based guests directly over the network. So the situation: Two KVM hosts with libvirt, multiple LVM based guests Want to do: Migrate a LVM based guest directly to the other host over an secure connection I know that migration is possible when the VM disks are stored on an NFS, GFS2 filer/cluster etc. So would it be possible to do an offline migration directly with netcat or something like that? Would be greate when someone could help me. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Secure migration of LVM based guests over WAN
Am 16.10.2012 11:40, schrieb Avi Kivity: On 10/16/2012 11:12 AM, Lukas Laukamp wrote: Hey all, I have a question about a solution for migrate LVM based guests directly over the network. So the situation: Two KVM hosts with libvirt, multiple LVM based guests Want to do: Migrate a LVM based guest directly to the other host over an secure connection I know that migration is possible when the VM disks are stored on an NFS, GFS2 filer/cluster etc. So would it be possible to do an offline migration directly with netcat or something like that? If all you need is offline, you can use scp to copy each volume to the destination volume. Make sure the guests are shut down when you do that. It is also possible to do a live migration, but unless the destination and source are in the same IP subnet, the guests are going to lose connectivity. Hello Avi, so can I simply copy an logical volume to the path of the volume group with scp? For the live migration theme, it would be no problem when the guests looses connectivity, how could be done a live migration? Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 15.10.2012 05:06, schrieb Javier Guerra Giraldez: On Sat, Oct 13, 2012 at 5:25 PM, Lukas Laukamp lu...@laukamp.me wrote: I have backed up the data within the machine with partimage and fsarchiver. But it would be greate to have a better way than doing this over a live system. make no mistake, the absolutely best way is from within the VM. It's the most consistent, safe and efficient method. Doing it from the outside is attractive, but it's a hack, and in some cases you have to jump through several hoops to make it safe. Because of the fact that this is a problem I bootet the VM with a live cd and made a backup of the important filesystems with the live system which runs inside the VM. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez: On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp lu...@laukamp.me wrote: I think that it must be possible to create an image with a size like the used space + a few hundret MB with metadata or something like that. the 'best' way to do it is 'from within' the VM the typical workaround is to mount/fsck a LVM snapshot (don't skip the fsck, or at least a journal replay) beyond that, there are a few utilities for specific filesystems: PartImage [1], dump/restore [2], and i'm sure some others. I don't know how would these behave with unclean images, which is what you get if you pull the image under the VM's feet. [1] http://www.partimage.org/Main_Page [2] http://dump.sourceforge.net/ So when I boot the VM with for example a live debian system, what would be the best method to backup the complete disk /dev/vda with only the data which is on it over a secure network connection to a destination in the network. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 13.10.2012 12:28, schrieb Lukas Laukamp: Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez: On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp lu...@laukamp.me wrote: I think that it must be possible to create an image with a size like the used space + a few hundret MB with metadata or something like that. the 'best' way to do it is 'from within' the VM the typical workaround is to mount/fsck a LVM snapshot (don't skip the fsck, or at least a journal replay) beyond that, there are a few utilities for specific filesystems: PartImage [1], dump/restore [2], and i'm sure some others. I don't know how would these behave with unclean images, which is what you get if you pull the image under the VM's feet. [1] http://www.partimage.org/Main_Page [2] http://dump.sourceforge.net/ So when I boot the VM with for example a live debian system, what would be the best method to backup the complete disk /dev/vda with only the data which is on it over a secure network connection to a destination in the network. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html I have backed up the data within the machine with partimage and fsarchiver. But it would be greate to have a better way than doing this over a live system. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[User Question] How to create a backup of an LVM based maschine without wasting space
Hey all, I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. Is this supported by qemu_backup or what would be the best way to do this? PS: Sorry for my bed english Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan Ah, thats great so to have a complete task in mind: 1. Create a qcow2 of a snapshot of an running VM 2. Transfer the qcow2 to the backup node 3. For incremental backup sync a daily LVM snapshot with the image on the backup node via rsync Is that right? Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 14:59, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan Ah, thats great so to have a complete task in mind: 1. Create a qcow2 of a snapshot of an running VM 2. Transfer the qcow2 to the backup node 3. For incremental backup sync a daily LVM snapshot with the image on the backup node via rsync Yes. Only make sure not to rsync the raw LVM image onto the qcow2 image - the backup client needs to have a raw image if you're syncing directly against the raw LVM volume. Stefan So I should use a raw file for the syncing and convert it on the backup node to a qcow2 to have a good file format? Best Regards -- To unsubscribe from this list: send the line
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 16:36, schrieb Javier Guerra Giraldez: On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi stefa...@gmail.com wrote: I would leave them raw as long as they are sparse (zero regions do not take up space). If you need to copy them you can either convert to qcow2 or use tools that preserve sparseness (BTW compression tools are good at this). note that free blocks previously used by deleted files won't be sparse, won't be zero and won't be much reduced by compression. i'd say the usual advice stays: A: if you run any non-trivial application on that VM, then use a real network backup tool 'from the inside' of the VM B: if real point-in-time application-cache-storage consistency is not important, then you can: - make a read-write LVM snapshot - mount that and fsck. (it will appear as not-cleanly unmounted) - backup the files. (i like rsync, especially if you have an uncompressed previous backup) - umount and destroy the snapshot - optionally compress the backup but seriously consider option A before. especially important if you run any DB on that VM I already thought about such situations. So DB systems are a real problem. I know that some data can be better backup directly but for example a VM for a Web- or Mailserver I want to backup completely to get this services get to work again very fast if there are problems. So it's a complex problem I think and not every machine can be backuped as another. I think that I must start by the hardware and software configuration of the backup node. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan I tested the qemu-img command now, but it does not do that what I want. I have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When I do the convert command the output is a 5GB qcow2 disk. What do I have to do to get a qcow2 file with only the allocated space/data from the LVM? I also tried the -c option of qemu-img convert but the result was nearly the same. Best Regards -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan I tested the qemu-img command now, but it does not do that what I want. I have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When I do the convert command the output is a 5GB qcow2 disk. What do I have to do to get a qcow2 file with only the allocated space/data from the LVM? I also tried the -c option of qemu-img convert but the result was nearly the same. Please show the exact command-lines you are using and the qemu-img info filename output afterwards. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org
Re: [User Question] How to create a backup of an LVM based maschine without wasting space
Am 12.10.2012 22:43, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp lu...@laukamp.me wrote: Am 12.10.2012 10:58, schrieb Lukas Laukamp: Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. If the VM is not running you can use qemu-img convert: qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefano, thanks for the fast reply. I will test this later. In my case now it would be a offline backup. For the online backup I think about a seperated system which every day makes incremental backups and once a week a full backup. The main problem is, that the systems are in a WAN network and I need encryption between the systems. Would it be possible to do something like this: create the LVM snapshot for the backup, read this LVM snapshot with the remote backup system via ssh tunnel and save the output of this to qcow2 files on the backup system? And in which format could be the incremental backups be stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the last image to rsync against. Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan I tested the qemu-img command now, but it does not do that what I want. I have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When I do the convert command the output is a 5GB qcow2 disk. What do I have to do to get a qcow2 file with only the allocated space/data from the LVM? I also tried the -c option of qemu-img convert but the result was nearly the same. Please show the exact command-lines you are using and the qemu-img info filename output afterwards. Stefan
