Re: [ovirt-users] Disk migration eats all CPU, vms running in SPM become unresponsive
users-boun...@ovirt.org [users-boun...@ovirt.org]quot; im Auftrag von quot;Federico Alberto Sayd [fs...@uncu.edu.ar] Gesendet: Donnerstag, 24. Juli 2014 18:16 An: users@ovirt.org Betreff: [ovirt-users] Disk migration eats all CPU, vms running in SPM become unresponsive Hello: I am experiencing some troubles with ovirt nodes: When a node is selected as SPM and I move a disk between storage domains, it seems that migration process eats all CPU and some VMs (running on the SPM) hang, others lose network connectivity. The events tab at Ovirt Engine reports the CPU exceeding the defined threshold and then reports that VMs in such host (SPM) are not responding. How can I debug this? Why do the VMs become unresponsive or lost network connectivity when the host CPU goes too high? I have attached a screenshot of the ovirt-engine events, and the relevant engine.log My setup: oVirt Engine Version: 3.4.0-1.el6 (Centos 6.5) Nodes: Centos 6.5 vdsm-4.14.6-0.el6 libvirt-0.10.2-29.el6_5.9 KVM: 0.12.1.2 - 2.415.el6_5.10 (jenkins build) Hi, just a quick guess: I would start the analysis by looking if SPM is swapping during the disk move. We had issues during snapshot deletion. More details at https://bugzilla.redhat.com/show_bug.cgi?id=1116558 If you encounter BSODs for windows VMs during that operation have a look at https://bugzilla.redhat.com/show_bug.cgi?id=1110305 Lessons learned so far for us: - run SPM on the smallest host (with the fewest running VMs) to mit mitigate side effects of high CPU/disk utilization. - Clear page cache if nodes run out of free memory. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration Permissions
On 02/17/2014 07:46 PM, Maurice James wrote: What permissions do I need to allow a user to be able to live migrate a disk from one storage domain to another? I have a group with the Power User and Super user permissions on the cluster and they get the following error when attempting to migrate a disk Error while executing action: User is not authorized to perform this action. 3.3.3-2.el6 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users was this resolved? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
- Original Message - From: Nir Soffer nsof...@redhat.com To: d...@redhat.com Cc: Maurice James midnightst...@msn.com, Ofer Blaut obl...@redhat.com, users@ovirt.org, Michal Skrivanek mskri...@redhat.com, Dan Kenigsberg dan...@redhat.com Sent: Thursday, February 27, 2014 9:15:24 AM Subject: Re: [Users] Disk Migration - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. This seems to be the only documentation for the migration network: www.ovirt.org/Features/Migration_Network And it seems that it works as intended. Nir ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
- Original Message - From: Michal Skrivanek mskri...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: d...@redhat.com, Dan Kenigsberg dan...@redhat.com, Maurice James midnightst...@msn.com, Ofer Blaut obl...@redhat.com, Users@ovirt.org Users users@ovirt.org Sent: Thursday, February 27, 2014 10:12:26 AM Subject: Re: [Users] Disk Migration On Feb 27, 2014, at 08:15 , Nir Soffer nsof...@redhat.com wrote: - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. Michal, Dan, can you elaborate on this? with storage offloading it's probably not going to be significant, however today it likely is. Nir, why would not using migration network make it better? Won't we have the same problem as before without migration network at all, i.e. choking the management channel? Gigs of data sent of the same netowrk used for migarations would make migration slower when the network is saturated. Should we maybe consider a dedicated storage network? This can be setup now in 3.4. Sergey, can you explain how this is configured now in 3.4? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
On 28 Feb 2014, at 15:27, Nir Soffer wrote: - Original Message - From: Michal Skrivanek mskri...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: d...@redhat.com, Dan Kenigsberg dan...@redhat.com, Maurice James midnightst...@msn.com, Ofer Blaut obl...@redhat.com, Users@ovirt.org Users users@ovirt.org Sent: Thursday, February 27, 2014 10:12:26 AM Subject: Re: [Users] Disk Migration On Feb 27, 2014, at 08:15 , Nir Soffer nsof...@redhat.com wrote: - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. Michal, Dan, can you elaborate on this? with storage offloading it's probably not going to be significant, however today it likely is. Nir, why would not using migration network make it better? Won't we have the same problem as before without migration network at all, i.e. choking the management channel? Gigs of data sent of the same netowrk used for migarations would make migration slower when the network is saturated. well, you saturate one or the other…still from the functional perspective it's better to not choke the management…so either use the migration network (and the migrations will compete with disk moves) or a separate network. I'd be tempted to say that typically having too many different networks might be an overkill and difficult to set up/maintain Should we maybe consider a dedicated storage network? This can be setup now in 3.4. Sergey, can you explain how this is configured now in 3.4? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
On Feb 27, 2014, at 08:15 , Nir Soffer nsof...@redhat.com wrote: - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. Michal, Dan, can you elaborate on this? with storage offloading it's probably not going to be significant, however today it likely is. Nir, why would not using migration network make it better? Won't we have the same problem as before without migration network at all, i.e. choking the management channel? Should we maybe consider a dedicated storage network? Thanks, michal Nir ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
On Thu, Feb 27, 2014 at 09:12:26AM +0100, Michal Skrivanek wrote: On Feb 27, 2014, at 08:15 , Nir Soffer nsof...@redhat.com wrote: - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. Michal, Dan, can you elaborate on this? with storage offloading it's probably not going to be significant, however today it likely is. Nir, why would not using migration network make it better? Won't we have the same problem as before without migration network at all, i.e. choking the management channel? Should we maybe consider a dedicated storage network? As far as I understand Nir, you are saying the same thing: it's better to separate management, migration, and storage to separate networks, so that one does not choke the other. When we get host network QoS done (hopefully ovirt-3.5) we can do this quite cheaply, over vlans. Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Disk Migration
I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations attachment: Interfaces.PNGattachment: Migrationinterfaces.PNG___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. Dafna On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Dafna Ron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
IIUC, since disk migrations are, as Dafna said, essentially a storage operation, the 'vm migration network' interface is not used for it. However, if the storage is iSCSI, on 3.4 you can set up 'iSCSI Bond' interfaces and define which interface is used to access which target. This would essentially allow you to create a bond that is used to access only the storage, and disk migrations should use that. Gadi Ickowicz - Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. Dafna On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Dafna Ron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk Migration
- Original Message - From: Dafna Ron d...@redhat.com To: Maurice James midnightst...@msn.com Cc: Ofer Blaut obl...@redhat.com, users@ovirt.org Sent: Wednesday, February 26, 2014 7:34:11 PM Subject: Re: [Users] Disk Migration On 02/26/2014 05:24 PM, Maurice James wrote: I have a specific interface set up for migrations. Why do disk migrations not use the interface that I have set for migrations? Is that by design? Shouldnt it use the interfaces that I have set aside for migrations? VM migrations work as they should but not disk migrations I don't think that you can configure interface for disk migration. Disk migration is actually copy of information from the original disk to a new disk created on a new domain + delete of the original disk once that is done. it's not actually a migration and so I am not sure you can actually configure an interface for that. adding ofer - perhpas he has a solution or it's possible and I am not aware of it. I guess that *not* using the migration network for storage operation is the expected behavior, to make migration faster and safer. Michal, Dan, can you elaborate on this? Nir ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Disk Migration Permissions
What permissions do I need to allow a user to be able to live migrate a disk from one storage domain to another? I have a group with the Power User and Super user permissions on the cluster and they get the following error when attempting to migrate a disk Error while executing action: User is not authorized to perform this action. 3.3.3-2.el6 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Disk migration among glusterfs volumes
Hello, I have a vm disk in glusterfs volume X. Now I have a replicated gluster volume Y. I would like to move the vm disk on it but if I go to move menu I cannot choose the target and I see a warning: some disks cannot be moved. How can I move it? Thanks, Mario ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk migration among glusterfs volumes
Mario Giammarco mgiammarco@... writes: How can I move it? Thanks, Mario Sorry forgot to create domain ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Disk migration
Hi, With version 3.2.1, when I move a disk from one storage domain to another allways get the message User admin@internal have failed to move disk WEB1_Disk1 to domain NAS1, but on the Disks options, under Storage Domain(s) I have the information it as moved the disk to domain NAS1. Who is right? If I try to make a snapshot get an error saying that the image does not exist . The really bad thing is if I shutdown the VM, cannot start it again, the error message is Failed to run VM teste (User: admin@internal). Any idea what I can do? Regards Jose -- Jose Ferradeira http://www.logicworks.pt ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Disk migration
Versions: libvirt-0.10.2.4-1.fc18 vdsm-4.10.3-10.fc18 1.2.2 - 6.fc18 - Original Message - From: supo...@logicworks.pt To: Users@ovirt.org Sent: Sexta-feira, 14 de Junho de 2013 11:40:51 Subject: [Users] Disk migration Hi, With version 3.2.1, when I move a disk from one storage domain to another allways get the message User admin@internal have failed to move disk WEB1_Disk1 to domain NAS1, but on the Disks options, under Storage Domain(s) I have the information it as moved the disk to domain NAS1. Who is right? If I try to make a snapshot get an error saying that the image does not exist. The really bad thing is if I shutdown the VM, cannot start it again, the error message is Failed to run VM teste (User: admin@internal). Any idea what I can do? Regards Jose -- Jose Ferradeira http://www.logicworks.pt ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
