Re: KVM live migration i/o error
On Fri, Mar 20, 2015 at 12:34:59PM +0100, Francesc Guasch wrote: On Fri, Mar 20, 2015 at 10:03:20AM +, Stefan Hajnoczi wrote: Hi Stefan, thank you very much for answering me. On Wed, Mar 18, 2015 at 04:53:28PM +0100, Francesc Guasch wrote: I have three Ubuntu Server 14.04 trusty with KVM. Two of them are HP servers and one is Dell. Both brands run fine the KVM virtual servers, and I can do live migration between the HPs. But I get I/O errors in the vda when I migrate to or from the Dell server. I have shared storage with NFS, mounted the same way in all of them: As soon as it starts in the origin console I spot I/O error messages, when it finishes I got them in the console in the destination server. The file system is read only and I have to shut it down hard. end request I/O error, /dev/vda, sector 8790327 origin console == guest's console? Yes, I mean I open two consoles with virt-manager, one in the origin host and another one in the destination I/O errors starting while the guest is still running on the migration source host is strange. I wonder if something happened to the NFS file related to file permissions or SELinux labels? I think I found something checking SELinux. ls -Z and getfattr return nothing. But ps -eZ showed something very different in the Dell server. This is in the HP server: /usr/sbin/libvirtd 1034 ?11:51:44 libvirtd libvirt-09540b5d-82 701 ?05:28:40 qemu-system-x86 unconfined 1?00:01:00 init In the Dell server init is confined in lxc and there are also lxc-start processes. /usr/sbin/libvirtd 1622 ?05:07:07 libvirtd libvirt-8a0f9087-32d... 29926 ? 00:00:01 qemu-system-x86 lxc-container-default 1774 ?00:00:00 init /usr/bin/lxc-start 1763 ?00:00:00 lxc-start There is also LXC installed in that server ! Maybe that is messing with kvm. The qemu processes look fine to me but there is a chance the problem comes from there. I could move the LXC somewhere else or I can keep it there to try to fix this issue. What do you advice I should do now ? I suggest asking on the libvirt mailing list: libvirt-l...@redhat.com pgpk8pbvBvTGx.pgp Description: PGP signature
Re: KVM live migration i/o error
On Wed, Mar 18, 2015 at 04:53:28PM +0100, Francesc Guasch wrote: I have three Ubuntu Server 14.04 trusty with KVM. Two of them are HP servers and one is Dell. Both brands run fine the KVM virtual servers, and I can do live migration between the HPs. But I get I/O errors in the vda when I migrate to or from the Dell server. I have shared storage with NFS, mounted the same way in all of them: nfs.sever:/kvm /var/lib/libvirt/images nfs auto,vers=3 I checked the version of all the packages to make sure are the same. I got: kernel: 3.13.0-43-generic #72-Ubuntu SMP x86_64 libvirt: libvirt: 1.2.2-0ubuntu13.1.9 qemu-utils: 2.0.0+dfsg-2ubuntu1.10 qemu-kvm: 2.0.0+dfsg-2ubuntu1.10 I made sure the Cache in the Storage is set to None. Disk bus: virtio Cache mode: none IO mode: default I run this to do live migration: virsh migrate --live virtual qemu+ssh://dellserver/system As soon as it starts in the origin console I spot I/O error messages, when it finishes I got them in the console in the destination server. The file system is read only and I have to shut it down hard. end request I/O error, /dev/vda, sector 8790327 origin console == guest's console? I/O errors starting while the guest is still running on the migration source host is strange. I wonder if something happened to the NFS file related to file permissions or SELinux labels? Stefan pgpiV3ba25OeW.pgp Description: PGP signature
Re: KVM live migration i/o error
On Fri, Mar 20, 2015 at 10:03:20AM +, Stefan Hajnoczi wrote: Hi Stefan, thank you very much for answering me. On Wed, Mar 18, 2015 at 04:53:28PM +0100, Francesc Guasch wrote: I have three Ubuntu Server 14.04 trusty with KVM. Two of them are HP servers and one is Dell. Both brands run fine the KVM virtual servers, and I can do live migration between the HPs. But I get I/O errors in the vda when I migrate to or from the Dell server. I have shared storage with NFS, mounted the same way in all of them: As soon as it starts in the origin console I spot I/O error messages, when it finishes I got them in the console in the destination server. The file system is read only and I have to shut it down hard. end request I/O error, /dev/vda, sector 8790327 origin console == guest's console? Yes, I mean I open two consoles with virt-manager, one in the origin host and another one in the destination I/O errors starting while the guest is still running on the migration source host is strange. I wonder if something happened to the NFS file related to file permissions or SELinux labels? I think I found something checking SELinux. ls -Z and getfattr return nothing. But ps -eZ showed something very different in the Dell server. This is in the HP server: /usr/sbin/libvirtd 1034 ?11:51:44 libvirtd libvirt-09540b5d-82 701 ?05:28:40 qemu-system-x86 unconfined 1?00:01:00 init In the Dell server init is confined in lxc and there are also lxc-start processes. /usr/sbin/libvirtd 1622 ?05:07:07 libvirtd libvirt-8a0f9087-32d... 29926 ? 00:00:01 qemu-system-x86 lxc-container-default 1774 ?00:00:00 init /usr/bin/lxc-start 1763 ?00:00:00 lxc-start There is also LXC installed in that server ! Maybe that is messing with kvm. The qemu processes look fine to me but there is a chance the problem comes from there. I could move the LXC somewhere else or I can keep it there to try to fix this issue. What do you advice I should do now ? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
KVM live migration i/o error
Hi. I have three Ubuntu Server 14.04 trusty with KVM. Two of them are HP servers and one is Dell. Both brands run fine the KVM virtual servers, and I can do live migration between the HPs. But I get I/O errors in the vda when I migrate to or from the Dell server. I have shared storage with NFS, mounted the same way in all of them: nfs.sever:/kvm /var/lib/libvirt/images nfs auto,vers=3 I checked the version of all the packages to make sure are the same. I got: kernel: 3.13.0-43-generic #72-Ubuntu SMP x86_64 libvirt: libvirt: 1.2.2-0ubuntu13.1.9 qemu-utils: 2.0.0+dfsg-2ubuntu1.10 qemu-kvm: 2.0.0+dfsg-2ubuntu1.10 I made sure the Cache in the Storage is set to None. Disk bus: virtio Cache mode: none IO mode: default I run this to do live migration: virsh migrate --live virtual qemu+ssh://dellserver/system As soon as it starts in the origin console I spot I/O error messages, when it finishes I got them in the console in the destination server. The file system is read only and I have to shut it down hard. end request I/O error, /dev/vda, sector 8790327 When I migrate to the other HP server the process runs fine. I don't know what else to check, I wonder if such different hardware could be a problem. These are the CPU flags in the HP server: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm c onstant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 po pcnt lahf_lm dtherm tpr_shadow vnmi flexpriority ept vpid And those in the Dell server: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant _tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow I tried to check the log files in /var/log/libvirt but I can't see any different message when I migrate from HP to HP than when I do from HP to Dell. What else can I try ? Thank you for your time -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
