Re: [PVE-User] Online storage migration not working (II)
Although a work-around, maybe putting a sleep 1 could help some users while not being noticeable for the rest? Please can you test with sleep 1 (Seems you currently use sleep 10). ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
I will do so today and will report back. On 30/10/14 10:58, Dietmar Maurer wrote: Although a work-around, maybe putting a sleep 1 could help some users while not being noticeable for the rest? Please can you test with sleep 1 (Seems you currently use sleep 10). -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
root@pmx1:~# kvm --version QEMU emulator version 2.1.2, Copyright (c) 2003-2008 Fabrice Bellard Hi, and to be sure, can you do a #info version in vm monitor through gui ? - Mail original - De: Eneko Lacunza elacu...@binovo.es À: Dietmar Maurer diet...@proxmox.com, pve-user@pve.proxmox.com Envoyé: Jeudi 30 Octobre 2014 11:00:57 Objet: Re: [PVE-User] Online storage migration not working (II) On 30/10/14 10:59, Dietmar Maurer wrote: Also, what kvm version do you use? # kvm --version make sure you have 2.1.2 root@pmx1:~# kvm --version QEMU emulator version 2.1.2, Copyright (c) 2003-2008 Fabrice Bellard -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
Hi Alexandre, No has gone multiple reboots. # info version 2.1.2 BTW, sleep 1, 2, 3 aren't good; I'm trying with sleep 4 right now. Cheers Eneko On 30/10/14 14:19, Alexandre DERUMIER wrote: root@pmx1:~# kvm --version QEMU emulator version 2.1.2, Copyright (c) 2003-2008 Fabrice Bellard Hi, and to be sure, can you do a #info version in vm monitor through gui ? - Mail original - De: Eneko Lacunza elacu...@binovo.es À: Dietmar Maurer diet...@proxmox.com, pve-user@pve.proxmox.com Envoyé: Jeudi 30 Octobre 2014 11:00:57 Objet: Re: [PVE-User] Online storage migration not working (II) On 30/10/14 10:59, Dietmar Maurer wrote: Also, what kvm version do you use? # kvm --version make sure you have 2.1.2 root@pmx1:~# kvm --version QEMU emulator version 2.1.2, Copyright (c) 2003-2008 Fabrice Bellard -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
Hi all,
It's been a while and I have been doing tests in the background.
I found some issues were related to faulty drives, but after removing
them migration issues continued. I had the same issues with PVE 3.2,
nfs, rbd and local storage.
I also noticed that out office production cluster doesn't seem to suffer
this problem (PVE 3.3) .
Today I found this post on the forum:
http://forum.proxmox.com/threads/19733-storage-migration-virtio-failed
Short is that in /usr/share/perl5/PVE/QemuServer.pm
changing:
---
if ($vmiddst == $vmid) {
# switch the disk if source and destination are on the same
guest
vm_mon_cmd($vmid, block-job-complete, device =
drive-$drive);
}
---
with
---
if ($vmiddst == $vmid) {
sleep 10;
# switch the disk if source and destination are on the same
guest
vm_mon_cmd($vmid, block-job-complete, device =
drive-$drive);
}
---
So adding a sleep, seems to have fixed all the migration problems so far
(I continue testing while I write this email).
The problem seems related to performance. The problem-suffering cluster
has 3 nodes each with 1 desktop CPU, 2 GB RAM, 2 relatively old hard
disks (~60MB/s write on NFS). Network is 1x1Gbit on each node, all on
the same HP 1gbit switch.
Although a work-around, maybe putting a sleep 1 could help some users
while not being noticeable for the rest?
Cheers
Eneko
On 13/10/14 15:04, Eneko Lacunza wrote:
Just tried moving a VM WS2008R2 with virtio v74 from local storage to
NFS (2 different servers) and RBD, all failed.
Does anyone with PVE 3.3 have working online storage migration with
Windows guests and virtio?
Thanks
Eneko
On 13/10/14 11:29, Eneko Lacunza wrote:
I have tried with virtio v74, but it also fails. All failures so far
with WS2012R2, Debian8beta2 works ok. I'm trying with WS2008R2 now.
On 10/10/14 20:41, Eneko Lacunza wrote:
I think problem is windows virtio v81. Debian 8beta has worked ok; I
will do further tests on monday.
El 10/10/2014 17:49, Luis G. Coralle luiscora...@fi.uncoma.edu.ar
mailto:luiscora...@fi.uncoma.edu.ar escribió:
Removing snapshots?
2014-10-10 8:36 GMT-03:00 Eneko Lacunza elacu...@binovo.es
mailto:elacu...@binovo.es:
Yes doing it with VM off works, but I can't make it work
with VMs on. It used to work in 3.2 and before, something
must be broken.
On 10/10/14 13:27, Luis G. Coralle wrote:
In some cases the same, migrating a hard drive from a
remote storage to a local one.
I fix this turning off the vm
2014-10-10 7:37 GMT-03:00 Eneko Lacunza elacu...@binovo.es
mailto:elacu...@binovo.es:
Hi all,
I tried local to RBD storage migration with VM running,
and it fails like it does with NFS:
transferred: 53687091200 bytes remaining: 0 bytes
total: 53687091200 bytes progression: 100.00 %
Removing all snapshots: 100% complete...done.
image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
TASK ERROR: storage migration failed: mirroring error:
VM 103 qmp command 'block-job-complete' failed - The
active block job for device 'drive-virtio0' cannot be
completed
I noticed that during migration HTML5 console's key
drop-down menu doesn't work (VM doesn't receive keys).
# pveversion -v
proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve)
pve-manager: 3.3-2 (running version: 3.3-2/995e687e)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-35
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-9
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
Is storage migration working for you in PVE 3.3?
Cheers
Eneko
--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997 tel:943575997
943493611 tel:943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180
Oiartzun (Gipuzkoa)
www.binovo.es
Re: [PVE-User] Online storage migration not working (II)
Just tried moving a VM WS2008R2 with virtio v74 from local storage to NFS (2 different servers) and RBD, all failed. Does anyone with PVE 3.3 have working online storage migration with Windows guests and virtio? Thanks Eneko On 13/10/14 11:29, Eneko Lacunza wrote: I have tried with virtio v74, but it also fails. All failures so far with WS2012R2, Debian8beta2 works ok. I'm trying with WS2008R2 now. On 10/10/14 20:41, Eneko Lacunza wrote: I think problem is windows virtio v81. Debian 8beta has worked ok; I will do further tests on monday. El 10/10/2014 17:49, Luis G. Coralle luiscora...@fi.uncoma.edu.ar mailto:luiscora...@fi.uncoma.edu.ar escribió: Removing snapshots? 2014-10-10 8:36 GMT-03:00 Eneko Lacunza elacu...@binovo.es mailto:elacu...@binovo.es: Yes doing it with VM off works, but I can't make it work with VMs on. It used to work in 3.2 and before, something must be broken. On 10/10/14 13:27, Luis G. Coralle wrote: In some cases the same, migrating a hard drive from a remote storage to a local one. I fix this turning off the vm 2014-10-10 7:37 GMT-03:00 Eneko Lacunza elacu...@binovo.es mailto:elacu...@binovo.es: Hi all, I tried local to RBD storage migration with VM running, and it fails like it does with NFS: transferred: 53687091200 bytes remaining: 0 bytes total: 53687091200 bytes progression: 100.00 % Removing all snapshots: 100% complete...done. image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers TASK ERROR: storage migration failed: mirroring error: VM 103 qmp command 'block-job-complete' failed - The active block job for device 'drive-virtio0' cannot be completed I noticed that during migration HTML5 console's key drop-down menu doesn't work (VM doesn't receive keys). # pveversion -v proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve) pve-manager: 3.3-2 (running version: 3.3-2/995e687e) pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-33-pve: 2.6.32-138 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.7-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.10-1 pve-cluster: 3.0-15 qemu-server: 3.1-35 pve-firmware: 1.1-3 libpve-common-perl: 3.0-19 libpve-access-control: 3.0-15 libpve-storage-perl: 3.0-23 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-8 vzctl: 4.0-1pve6 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 2.1-9 ksm-control-daemon: 1.1-1 glusterfs-client: 3.5.2-1 Is storage migration working for you in PVE 3.3? Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 tel:943575997 943493611 tel:943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es http://www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com mailto:pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf.943575997 tel:943575997 943493611 tel:943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es http://www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com mailto:pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
Yes doing it with VM off works, but I can't make it work with VMs on. It used to work in 3.2 and before, something must be broken. On 10/10/14 13:27, Luis G. Coralle wrote: In some cases the same, migrating a hard drive from a remote storage to a local one. I fix this turning off the vm 2014-10-10 7:37 GMT-03:00 Eneko Lacunza elacu...@binovo.es mailto:elacu...@binovo.es: Hi all, I tried local to RBD storage migration with VM running, and it fails like it does with NFS: transferred: 53687091200 bytes remaining: 0 bytes total: 53687091200 bytes progression: 100.00 % Removing all snapshots: 100% complete...done. image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers TASK ERROR: storage migration failed: mirroring error: VM 103 qmp command 'block-job-complete' failed - The active block job for device 'drive-virtio0' cannot be completed I noticed that during migration HTML5 console's key drop-down menu doesn't work (VM doesn't receive keys). # pveversion -v proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve) pve-manager: 3.3-2 (running version: 3.3-2/995e687e) pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-33-pve: 2.6.32-138 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.7-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.10-1 pve-cluster: 3.0-15 qemu-server: 3.1-35 pve-firmware: 1.1-3 libpve-common-perl: 3.0-19 libpve-access-control: 3.0-15 libpve-storage-perl: 3.0-23 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-8 vzctl: 4.0-1pve6 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 2.1-9 ksm-control-daemon: 1.1-1 glusterfs-client: 3.5.2-1 Is storage migration working for you in PVE 3.3? Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es http://www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com mailto:pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
Removing snapshots? 2014-10-10 8:36 GMT-03:00 Eneko Lacunza elacu...@binovo.es: Yes doing it with VM off works, but I can't make it work with VMs on. It used to work in 3.2 and before, something must be broken. On 10/10/14 13:27, Luis G. Coralle wrote: In some cases the same, migrating a hard drive from a remote storage to a local one. I fix this turning off the vm 2014-10-10 7:37 GMT-03:00 Eneko Lacunza elacu...@binovo.es: Hi all, I tried local to RBD storage migration with VM running, and it fails like it does with NFS: transferred: 53687091200 bytes remaining: 0 bytes total: 53687091200 bytes progression: 100.00 % Removing all snapshots: 100% complete...done. image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers TASK ERROR: storage migration failed: mirroring error: VM 103 qmp command 'block-job-complete' failed - The active block job for device 'drive-virtio0' cannot be completed I noticed that during migration HTML5 console's key drop-down menu doesn't work (VM doesn't receive keys). # pveversion -v proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve) pve-manager: 3.3-2 (running version: 3.3-2/995e687e) pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-33-pve: 2.6.32-138 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.7-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.10-1 pve-cluster: 3.0-15 qemu-server: 3.1-35 pve-firmware: 1.1-3 libpve-common-perl: 3.0-19 libpve-access-control: 3.0-15 libpve-storage-perl: 3.0-23 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-8 vzctl: 4.0-1pve6 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 2.1-9 ksm-control-daemon: 1.1-1 glusterfs-client: 3.5.2-1 Is storage migration working for you in PVE 3.3? Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online storage migration not working (II)
I think problem is windows virtio v81. Debian 8beta has worked ok; I will do further tests on monday. El 10/10/2014 17:49, Luis G. Coralle luiscora...@fi.uncoma.edu.ar escribió: Removing snapshots? 2014-10-10 8:36 GMT-03:00 Eneko Lacunza elacu...@binovo.es: Yes doing it with VM off works, but I can't make it work with VMs on. It used to work in 3.2 and before, something must be broken. On 10/10/14 13:27, Luis G. Coralle wrote: In some cases the same, migrating a hard drive from a remote storage to a local one. I fix this turning off the vm 2014-10-10 7:37 GMT-03:00 Eneko Lacunza elacu...@binovo.es: Hi all, I tried local to RBD storage migration with VM running, and it fails like it does with NFS: transferred: 53687091200 bytes remaining: 0 bytes total: 53687091200 bytes progression: 100.00 % Removing all snapshots: 100% complete...done. image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers TASK ERROR: storage migration failed: mirroring error: VM 103 qmp command 'block-job-complete' failed - The active block job for device 'drive-virtio0' cannot be completed I noticed that during migration HTML5 console's key drop-down menu doesn't work (VM doesn't receive keys). # pveversion -v proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve) pve-manager: 3.3-2 (running version: 3.3-2/995e687e) pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-33-pve: 2.6.32-138 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.7-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.10-1 pve-cluster: 3.0-15 qemu-server: 3.1-35 pve-firmware: 1.1-3 libpve-common-perl: 3.0-19 libpve-access-control: 3.0-15 libpve-storage-perl: 3.0-23 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-8 vzctl: 4.0-1pve6 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 2.1-9 ksm-control-daemon: 1.1-1 glusterfs-client: 3.5.2-1 Is storage migration working for you in PVE 3.3? Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)www.binovo.es ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Luis G. Coralle ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
