Re: [pve-devel] snapshot improvements
Am 29.09.2012 um 08:26 schrieb Dietmar Maurer diet...@proxmox.com: I thought of something like: ssh cloud1-1200.de-nserver.de zfs send JBOD01Pool/vm-105-disk- 1@testsnap | zfs recv tank/abc or ssh cloud1-1200.de-nserver.de zfs send JBOD01Pool/vm-105-disk-1@spriebetest | gzip /mnt/pve/nfstor/vm-105- 2012-09-25.gz Are your aware of the fact that ssh needs to encrypt/decrypt all data. This needs much CPU power and is slow (still no AES support in libs). Oh we could also use netcat instead. Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Are your aware of the fact that ssh needs to encrypt/decrypt all data. This needs much CPU power and is slow (still no AES support in libs). Oh we could also use netcat instead. How exactly? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Ah, good to know. Besides, connecting with ssh looks still strange to me. no api for this... ut I guess we will only backup VM without snapshots, so we do not need that for backup purposes. BTW, is there really no way to delete nexenta snapshots used by another snapshot (snapshot merge)? If not, should we implement that on the qemu side? what do you mean by snapshot used by another snapshot? ex: image-snap1-snap2-you are here you can delete snap1 without any problem, and without need to merge. the only thing with zfs, is that you can't rollback to snap1 without deleting snap2. - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com, Stefan Priebe s.pri...@profihost.ag Envoyé: Vendredi 28 Septembre 2012 06:40:10 Objet: RE: [pve-devel] snapshot improvements Subject: Re: [pve-devel] snapshot improvements do you mean : backup the main image then backup each snasphot increment ? Yes, something like that. zfs send can do incremental backup with zfs send -I http://docs.oracle.com/cd/E19082-01/817-2271/gfwqb/index.html Ah, good to know. Besides, connecting with ssh looks still strange to me. But I guess we will only backup VM without snapshots, so we do not need that for backup purposes. BTW, is there really no way to delete nexenta snapshots used by another snapshot (snapshot merge)? If not, should we implement that on the qemu side? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
what do you mean by snapshot used by another snapshot? ex: image-snap1-snap2-you are here you can delete snap1 without any problem, and without need to merge. Really? - last time I tried I got 'snapshot in use' (or something like that). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
maybe do you have cloned it ? - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com, Stefan Priebe s.pri...@profihost.ag Envoyé: Vendredi 28 Septembre 2012 11:32:01 Objet: RE: [pve-devel] snapshot improvements what do you mean by snapshot used by another snapshot? ex: image-snap1-snap2-you are here you can delete snap1 without any problem, and without need to merge. Really? - last time I tried I got 'snapshot in use' (or something like that). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
maybe do you have cloned it ? no - I need to test that again. - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com, Stefan Priebe s.pri...@profihost.ag Envoyé: Vendredi 28 Septembre 2012 11:32:01 Objet: RE: [pve-devel] snapshot improvements what do you mean by snapshot used by another snapshot? ex: image-snap1-snap2-you are here you can delete snap1 without any problem, and without need to merge. Really? - last time I tried I got 'snapshot in use' (or something like that). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Am 27.09.2012 06:43, schrieb Dietmar Maurer: I also tried todo backups with snapshots but this doesn't seem to work: INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot -- compress lzo --storage backuplocal --node serv121 INFO: Starting Backup of VM 100 (qemu) INFO: status = running ERROR: Backup of VM 100 failed - no such volume 'Stor1:vm-100-disk-1' INFO: Backup job finished with errors TASK ERROR: job errors Yes, I need to implement that first. The idea is that we do not backup any snapshot data. The vzdump would only include the data of from the running instance. I guess that is OK? But isn't the correct way to make a snapshot and then compress the snapshot? How can vzdump verify that the data integrity is fine? Greets, Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
I also don't think that all storage can export datas with snapshots inside. rbd can export an image from a snapshot, nexenta and sheepdog too. How can you access snapshot data with nexenta? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
The idea is that we do not backup any snapshot data. The vzdump would only include the data of from the running instance. I guess that is OK? But isn't the correct way to make a snapshot and then compress the snapshot? How can vzdump verify that the data integrity is fine? Sorry, I do not understand that question? For now, I will simply disable vzdump for VMs with snapshots (no time to work on that now). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
I think I have already respond to that ;) 2 ways : - clone the snapshot and export image through iscsi and backup it - use zfs send through ssh (zfs send image1@snap1 /imagefile) - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com, Stefan Priebe s.pri...@profihost.ag Envoyé: Jeudi 27 Septembre 2012 09:04:31 Objet: RE: [pve-devel] snapshot improvements I also don't think that all storage can export datas with snapshots inside. rbd can export an image from a snapshot, nexenta and sheepdog too. How can you access snapshot data with nexenta? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
I think I have already respond to that ;) 2 ways : - clone the snapshot and export image through iscsi and backup it - use zfs send through ssh (zfs send image1@snap1 /imagefile) Both ways are clumsy. Also, nexenta snapshot support is quite unusable, because you can't delete snapshots. I am not really happy with that plugin. - Dietmar ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
The idea is that we do not backup any snapshot data. The vzdump would only include the data of from the running instance. I guess that is OK? But isn't the correct way to make a snapshot and then compress the snapshot? How can vzdump verify that the data integrity is fine? I guess I found the source of confusion ;-) There are different types of snapshots 1.) vzdump snapshots: vzdump uses LVM to make snapshots of VM data 2.) VM (live) snapshots Those things are currently totally different topics. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Am 27.09.2012 09:26, schrieb Dietmar Maurer: The idea is that we do not backup any snapshot data. The vzdump would only include the data of from the running instance. I guess that is OK? But isn't the correct way to make a snapshot and then compress the snapshot? How can vzdump verify that the data integrity is fine? I guess I found the source of confusion ;-) There are different types of snapshots 1.) vzdump snapshots: vzdump uses LVM to make snapshots of VM data 2.) VM (live) snapshots Those things are currently totally different topics. ah OK but wouldn't it be nice to be able to backup live snapshots? compress and store them on a seperate NFS server? Also Proxmox only allows to schedule beckups not to schedule snapshots ;-) Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
ah OK but wouldn't it be nice to be able to backup live snapshots? compress and store them on a seperate NFS server? Also Proxmox only allows to schedule beckups not to schedule snapshots ;-) Sure, that would be nice. I already have very detailed plans how to do that (but no budget). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
And if one would consider to backup snapshots, I am quite sure he only wants to backup shared data once. A simple data export would duplicate large amounts of data. Subject: Re: [pve-devel] snapshot improvements I think I have already respond to that ;) 2 ways : - clone the snapshot and export image through iscsi and backup it - use zfs send through ssh (zfs send image1@snap1 /imagefile) - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com, Stefan Priebe s.pri...@profihost.ag Envoyé: Jeudi 27 Septembre 2012 09:04:31 Objet: RE: [pve-devel] snapshot improvements I also don't think that all storage can export datas with snapshots inside. rbd can export an image from a snapshot, nexenta and sheepdog too. How can you access snapshot data with nexenta? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Subject: Re: [pve-devel] snapshot improvements do you mean : backup the main image then backup each snasphot increment ? Yes, something like that. zfs send can do incremental backup with zfs send -I http://docs.oracle.com/cd/E19082-01/817-2271/gfwqb/index.html Ah, good to know. Besides, connecting with ssh looks still strange to me. But I guess we will only backup VM without snapshots, so we do not need that for backup purposes. BTW, is there really no way to delete nexenta snapshots used by another snapshot (snapshot merge)? If not, should we implement that on the qemu side? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Am 24.09.2012 10:50, schrieb Dietmar Maurer: I tried to improve snapshot behavior a bit: - take snapshot is now a background task (does not block monitor) - try to save VM state while VM is online So downtime during snapshot should be shorter now, especially if the VM use much RAM. Feel free to test and report bugs ;-) Really great! Works pretty good - tested with todays git. I also tried todo backups with snapshots but this doesn't seem to work: INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot --compress lzo --storage backuplocal --node serv121 INFO: Starting Backup of VM 100 (qemu) INFO: status = running ERROR: Backup of VM 100 failed - no such volume 'Stor1:vm-100-disk-1' INFO: Backup job finished with errors TASK ERROR: job errors Greets, Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
I also tried todo backups with snapshots but this doesn't seem to work: INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot -- compress lzo --storage backuplocal --node serv121 INFO: Starting Backup of VM 100 (qemu) INFO: status = running ERROR: Backup of VM 100 failed - no such volume 'Stor1:vm-100-disk-1' INFO: Backup job finished with errors TASK ERROR: job errors Yes, I need to implement that first. The idea is that we do not backup any snapshot data. The vzdump would only include the data of from the running instance. I guess that is OK? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
do you think image = size of memory if enough ? (if we have some bytes
more with incremental ?)
PVE/QemuServer.pm:
my $driver_state_size = 32; # assume 32MB is enough to safe all driver
state;
my $size = $conf-{memory} + $driver_state_size;
Unfortunately, the qemu code seem to report wrong values for 'remaining'
bytes. So I had to increase allocated space to:
my $size = $conf-{memory}*2 + $driver_state_size;
Note: we truncate the file after save if possible.
- Dietmar
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
Unfortunately, the qemu code seem to report wrong values for 'remaining'
bytes. So I had to increase allocated space to:
my $size = $conf-{memory}*2 + $driver_state_size;
Note: we truncate the file after save if possible.
Ok, I'll rested that today
yesterday tests show no filesystem corruption, running a fio write benchmark
and take snapshot with vmstate,
rollbacking, with fio benchmerk continue to write again, rollback,rollback.
So it's seem stable.
- Mail original -
De: Dietmar Maurer diet...@proxmox.com
À: Dietmar Maurer diet...@proxmox.com, Alexandre DERUMIER
aderum...@odiso.com
Cc: pve-devel@pve.proxmox.com
Envoyé: Mardi 25 Septembre 2012 10:37:42
Objet: RE: [pve-devel] snapshot improvements
do you think image = size of memory if enough ? (if we have some bytes
more with incremental ?)
PVE/QemuServer.pm:
my $driver_state_size = 32; # assume 32MB is enough to safe all driver
state;
my $size = $conf-{memory} + $driver_state_size;
Unfortunately, the qemu code seem to report wrong values for 'remaining'
bytes. So I had to increase allocated space to:
my $size = $conf-{memory}*2 + $driver_state_size;
Note: we truncate the file after save if possible.
- Dietmar
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
How is it possible to save the vmstate if the vm is not paused ? (with a lot a memory write access by example ?) I currently allocate am image equal to the size of the VM memory. Then I simply do an incremental state save (like a vm migration), and keep the VM running until (saved_bytes + remaining_bytes available_space). This is not perfect, but seems to work. But maybe we need to make that feature optional in future? Another improvement would be to save VM state to absolute positions inside the state file. That way disk usage would be limited, and would use minimal space when file-system support sparse files. But this functionality would require much more work inside qemu code. - Dietmar ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
I currently allocate am image equal to the size of the VM memory. Then I simply do an incremental state save (like a vm migration), and keep the VM running until (saved_bytes + remaining_bytes available_space). do you think image = size of memory if enough ? (if we have some bytes more with incremental ?) This is not perfect, but seems to work. But maybe we need to make that feature optional in future? I'll test it. (but it's a huge improvement, I can't pause vm in production) Also, any problem if some datas in vmstate need to be flushed to disk ? I see that you have add a use writeback cache patch, maybe it's for that case ? If yes, does it works with all devices (ide/scsi/virtio) ? (with any guest driver version) - Mail original - De: Dietmar Maurer diet...@proxmox.com À: Alexandre DERUMIER aderum...@odiso.com Cc: pve-devel@pve.proxmox.com Envoyé: Lundi 24 Septembre 2012 11:52:04 Objet: RE: [pve-devel] snapshot improvements How is it possible to save the vmstate if the vm is not paused ? (with a lot a memory write access by example ?) I currently allocate am image equal to the size of the VM memory. Then I simply do an incremental state save (like a vm migration), and keep the VM running until (saved_bytes + remaining_bytes available_space). This is not perfect, but seems to work. But maybe we need to make that feature optional in future? Another improvement would be to save VM state to absolute positions inside the state file. That way disk usage would be limited, and would use minimal space when file-system support sparse files. But this functionality would require much more work inside qemu code. - Dietmar ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] snapshot improvements
do you think image = size of memory if enough ? (if we have some bytes
more with incremental ?)
PVE/QemuServer.pm:
my $driver_state_size = 32; # assume 32MB is enough to safe all driver
state;
my $size = $conf-{memory} + $driver_state_size;
Also, any problem if some datas in vmstate need to be flushed to disk ?
We still stop the VM for a short period, and I think vm_stop() flush all driver
cache.
I see that you have add a use writeback cache patch, maybe it's for that
case ?
no, this is just for the state file itself.
If yes, does it works with all devices (ide/scsi/virtio) ? (with any guest
driver
version)
why not? It is basically the same code as used for live migration.
But yes, we need to carefully test it ;-)
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
