Re: [openstack-dev] [nova] Help needed in debugging issue - ClientException: Unable to update the attachment. (HTTP 500)
On 25/04/18 9:09 PM, Matt Riedemann wrote: On 4/25/2018 10:34 AM, Sreeram Vancheeswaran wrote: Thank you so much Matt for the detailed steps. We are doing boot from image and are probably running into the issue mentioned in [2] in your email. Hmm, OK, but that doesn't really make sense how you're going down this path [1] in the code because the API doesn't create a volume attachment record when booting from a volume where the source_type='image', so it should be going down the "legacy" attach flow where attachment_update is not called. Do you have some proprietary code in place that might be causing some problems? Otherwise we need to figure out how this is failing because it could be an issue in Queens. [1] https://github.com/openstack/nova/blob/0a642e2eee8d430ddcccf2947aedcca1a0a0b005/nova/virt/block_device.py#L597 Yes, we have some proprietary code to copy image on to volume (as a prototype now), and probably that is causing the issue here; I will debug/trace using the info you previously provided and figure out the root cause. I will get back to you if there is some issue in the "in-tree" code. Again, thank you so much for providing directions on where to continue debugging. -- ------- Sreeram Vancheeswaran System z Firmware - Openstack Development IBM Systems & Technology Lab, Bangalore, India Phone: +91 80 40660826 Mob: +91-9341411511 Email : sree...@linux.vnet.ibm.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Help needed in debugging issue - ClientException: Unable to update the attachment. (HTTP 500)
On 25/04/18 7:40 PM, Matt Riedemann wrote: On 4/25/2018 3:32 AM, Sreeram Vancheeswaran wrote: Hi team! We are currently facing an issue in our out-of-tree driver nova-dpm [1] with nova and cinder on master, where instance launch in devstack is failing due to communication/time-out issues in nova-cinder. We are unable to get to the root cause of the issue and we need your help on getting some hints/directions to debug this issue further. --> From nova-compute service: BuildAbortException: Build of instance aborted: Unable to update the attachment. (HTTP 500) from the nova-compute server (detailed logs here [2]). --> From cinder-volume service: ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment could not be found with filter: attachment_id = 266ef7e1-4735-40f1-b704-509472f565cb. Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server (detailed logs here [3]) Debugging steps done so far:- * Compared the package versions between the current devstack under test with the **last succeeding job in our CI system** (to be exact, it was for the patches https://review.openstack.org/#/c/458514/ and https://review.openstack.org/#/c/458820/); However the package versions for packages such as sqlalchemy, os-brick, oslo* are exactly the same in both the systems. * We used git bisect to revert nova and cinder projects to versions equal to or before the date of our last succeeding CI run; but still we were able to reproduce the same error. * Our guess is that the db "Save" operation during the update of volume attachment is failing. But we are unable to trace/debug to that point in the rpc call; Any suggestions on how to debug this sceario would be really helpful. * We are running devstack master on Ubuntu 16.04.04 References [1] https://github.com/openstack/nova-dpm [2] Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.volume.cinder [None req-751d4586-cd97-4a8f-9423-f2bc4b1f1269 service nova] Update attachment failed for attachment 266ef7e1-4735-40f1-b704-509472f565cb. Error: Unable to update the attachment. (HTTP 500) (Request-ID: req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce) Code: 500: ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [None req-751d4586-cd97-4a8f-9423-f2bc4b1f1269 service nova] [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] Instance failed block device setup: ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] Traceback (most recent call last): Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/opt/stack/nova/nova/compute/manager.py", line 1577, in _prep_block_device Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] wait_func=self._await_block_device_map_created) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/opt/stack/nova/nova/virt/block_device.py", line 828, in attach_block_devices Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] _log_and_attach(device) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/opt/stack/nova/nova/virt/block_device.py", line 825, in _log_and_attach Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] bdm.attach(*attach_args, **attach_kwargs) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] ret_val = method(obj, context, *args, **kwargs) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/opt/stack/nova/nova/virt/block_device.py", line 618, in attach Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] virt_driver, do_driver_attach) Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR nova.compute.manager [instance: d761da60-7bb1-415e-b5b9-eaaed124d6d2] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner Apr 25 06:41:57 zos057 nova-compute[6190]:
[openstack-dev] [nova] Help needed in debugging issue - ClientException: Unable to update the attachment. (HTTP 500)
opt/stack/cinder/cinder/volume/manager.py", line 4349, in _connection_create Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server self.db.volume_attachment_update(ctxt, attachment.id, values) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/api.py", line 365, in volume_attachment_update Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server return IMPL.volume_attachment_update(context, attachment_id, values) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 182, in wrapper Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 2674, in volume_attachment_update Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server filter='attachment_id = ' + attachment_id) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment could not be found with filter: attachment_id = 266ef7e1-4735-40f1-b704-509472f565cb. Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server -- --- Sreeram Vancheeswaran System z Firmware - Openstack Development IBM Systems & Technology Lab, Bangalore, India Phone: +91 80 40660826 Mob: +91-9341411511 Email : sree...@linux.vnet.ibm.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Help needed in debugging issue - ClientException: Unable to update the attachment. (HTTP 500)
"/opt/stack/cinder/cinder/volume/manager.py", line 4349, in _connection_create Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server self.db.volume_attachment_update(ctxt, attachment.id, values) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/api.py", line 365, in volume_attachment_update Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server return IMPL.volume_attachment_update(context, attachment_id, values) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 182, in wrapper Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 2674, in volume_attachment_update Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server filter='attachment_id = ' + attachment_id) Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment could not be found with filter: attachment_id = 266ef7e1-4735-40f1-b704-509472f565cb. Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR oslo_messaging.rpc.server -- --- Sreeram Vancheeswaran System z Firmware - Openstack Development IBM Systems & Technology Lab, Bangalore, India Phone: +91 80 40660826 Mob: +91-9341411511 Email : sree...@linux.vnet.ibm.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
