Re: [openstack-dev] [OpenStack-dev][Nova] Migration stuck - resize/migrating
Indeed. Ensure you have SSH access between compute nodes (I'm working on some code to remove this requirement, but it may be a while before it gets merged). Also, if you can, could you post logs somewhere with the 'debug' config option enabled? I might be able to spot something quickly, since I've been working on the related code recently. Best Regards, Solly Ross - Original Message - From: Vishvananda Ishaya vishvana...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Tuesday, November 18, 2014 4:07:31 PM Subject: Re: [openstack-dev] [OpenStack-dev][Nova] Migration stuck - resize/migrating Migrate/resize uses scp to copy files back and forth with the libvirt driver. This shouldn’t be necessary with shared storage, but it may still need ssh configured between the user that nova is running as in order to complete the migration. It is also possible that there is a bug in the code path dealing with shared storage, although I would have expected you to see a traceback somewhere. Vish On Nov 11, 2014, at 1:10 AM, Eduard Matei eduard.ma...@cloudfounders.com wrote: Hi, I'm testing our cinder volume driver in the following setup: - 2 nodes, ubuntu, devstack juno (2014.2.1) - shared storage (common backend), our custom software solution + cinder volume on shared storage - 1 instance running on node 1, /instances directory on shared storage - kvm, libvirt (with live migration flags) Live migration of instance between nodes works perfectly. Migrate simply blocks. The instance in in status Resize/Migrate, no errors in n-cpu or n-sch, and it stays like that for over 8 hours (all night). I thought it was copying the disk, but it's a 20GB sparse file with approx. 200 mb of data, and the nodes have 1Gbps link, so it should be a couple of seconds. Any difference between live migration and migration? As i said, we use a shared filesystem-like storage solution so the volume files and the instance files are visible on both nodes, so no data needs copying. I know it's tricky to debug since we use a custom cinder driver, but anyone has any ideas where to start looking? Thanks, Eduard -- Eduard Biceri Matei, Senior Software Developer www.cloudfounders.com | eduard.ma...@cloudfounders.com CloudFounders, The Private Cloud Software Company Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this email in error we request you to notify us by reply e-mail and to delete all electronic files of the message. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the content of this message, and shall have no liability for any loss or damage suffered by the user, which arise as a result of e-mail transmission. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-dev][Nova] Migration stuck - resize/migrating
Migrate/resize uses scp to copy files back and forth with the libvirt driver. This shouldn’t be necessary with shared storage, but it may still need ssh configured between the user that nova is running as in order to complete the migration. It is also possible that there is a bug in the code path dealing with shared storage, although I would have expected you to see a traceback somewhere. Vish On Nov 11, 2014, at 1:10 AM, Eduard Matei eduard.ma...@cloudfounders.com wrote: Hi, I'm testing our cinder volume driver in the following setup: - 2 nodes, ubuntu, devstack juno (2014.2.1) - shared storage (common backend), our custom software solution + cinder volume on shared storage - 1 instance running on node 1, /instances directory on shared storage - kvm, libvirt (with live migration flags) Live migration of instance between nodes works perfectly. Migrate simply blocks. The instance in in status Resize/Migrate, no errors in n-cpu or n-sch, and it stays like that for over 8 hours (all night). I thought it was copying the disk, but it's a 20GB sparse file with approx. 200 mb of data, and the nodes have 1Gbps link, so it should be a couple of seconds. Any difference between live migration and migration? As i said, we use a shared filesystem-like storage solution so the volume files and the instance files are visible on both nodes, so no data needs copying. I know it's tricky to debug since we use a custom cinder driver, but anyone has any ideas where to start looking? Thanks, Eduard -- Eduard Biceri Matei, Senior Software Developer www.cloudfounders.com | eduard.ma...@cloudfounders.com CloudFounders, The Private Cloud Software Company Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this email in error we request you to notify us by reply e-mail and to delete all electronic files of the message. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the content of this message, and shall have no liability for any loss or damage suffered by the user, which arise as a result of e-mail transmission. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [OpenStack-dev][Nova] Migration stuck - resize/migrating
Hi, I'm testing our cinder volume driver in the following setup: - 2 nodes, ubuntu, devstack juno (2014.2.1) - shared storage (common backend), our custom software solution + cinder volume on shared storage - 1 instance running on node 1, /instances directory on shared storage - kvm, libvirt (with live migration flags) Live migration of instance between nodes works perfectly. Migrate simply blocks. The instance in in status Resize/Migrate, no errors in n-cpu or n-sch, and it stays like that for over 8 hours (all night). I thought it was copying the disk, but it's a 20GB sparse file with approx. 200 mb of data, and the nodes have 1Gbps link, so it should be a couple of seconds. Any difference between live migration and migration? As i said, we use a shared filesystem-like storage solution so the volume files and the instance files are visible on both nodes, so no data needs copying. I know it's tricky to debug since we use a custom cinder driver, but anyone has any ideas where to start looking? Thanks, Eduard -- *Eduard Biceri Matei, Senior Software Developer* www.cloudfounders.com | eduard.ma...@cloudfounders.com *CloudFounders, The Private Cloud Software Company* Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this email in error we request you to notify us by reply e-mail and to delete all electronic files of the message. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the content of this message, and shall have no liability for any loss or damage suffered by the user, which arise as a result of e-mail transmission. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
