There is a lock being held in c-vol that is taking awhile, 16 seconds in one case, I'm not sure if that's a lot or not:
http://logs.openstack.org/10/230510/2/gate/gate-grenade- dsvm/e796b39/logs/new/screen-c-vol.txt.gz#_2015-10-06_05_40_15_936 2015-10-06 05:40:15.936 DEBUG oslo_concurrency.lockutils [req- 6b402bc1-3c24-47f8-be65-609bfb98c8fd tempest- TestVolumeBootPatternV2-836916434] Lock "35fbbb48-90a3-45e2-9275 -d66c9b8e9b2b-delete_snapshot" released by "cinder.volume.manager._run_flow_locked" :: held 16.103s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265 ** Also affects: cinder Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1503290 Title: grenade failures on mitaka side with failure prepping block device due to RPC timeout Status in Cinder: New Status in OpenStack Compute (nova): Confirmed Bug description: http://logs.openstack.org/10/230510/2/gate/gate-grenade- dsvm/e796b39/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2015-10-06_05_39_36_682 2015-10-06 05:39:36.682 ERROR nova.compute.manager [req-03e13a69-3da6-4bc6-bcab-3b2a3fe4d236 tempest-ServersTestManualDisk-1771971010 tempest-ServersTestManualDisk-1239478904] [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device. 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Traceback (most recent call last): 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/opt/stack/new/nova/nova/compute/manager.py", line 1906, in _do_build_and_run_instance 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] filter_properties) 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/opt/stack/new/nova/nova/compute/manager.py", line 2034, in _build_and_run_instance 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] 'create.error', fault=e) 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] six.reraise(self.type_, self.value, self.tb) 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/opt/stack/new/nova/nova/compute/manager.py", line 1999, in _build_and_run_instance 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] block_device_mapping) as resources: 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] return self.gen.next() 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] File "/opt/stack/new/nova/nova/compute/manager.py", line 2165, in _build_resources 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] reason=msg) 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] BuildAbortException: Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device. 2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2RvX2J1aWxkX2FuZF9ydW5faW5zdGFuY2VcIiBBTkQgbWVzc2FnZTpcImFib3J0ZWQ6IEZhaWx1cmUgcHJlcHBpbmcgYmxvY2sgZGV2aWNlXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDQxMzg1OTE3NjV9 7 hits in 7 days, check and gate, all failures, master branch only. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1503290/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
