[Yahoo-eng-team] [Bug 1850514] Re: ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails with "Cannot 'migrate' instance while it is in vm_state building"
Reviewed: https://review.opendev.org/691941
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=56a391aafc1d75e33197423b0f1bb540c6dd2673
Submitter: Zuul
Branch:master
commit 56a391aafc1d75e33197423b0f1bb540c6dd2673
Author: Matt Riedemann
Date: Tue Oct 29 14:29:52 2019 -0400
Fix race in test_vcpu_to_pcpu_reshape
This test uses the ServersTestBase._wait_for_state_change method
which waits for the status to change *from* what is provided, so
when creating a server and waiting for the status to change from
ACTIVE makes _wait_for_state_change return immediately since the
status starts as BUILD. This can lead to a failure when the test
tries to migrate a server that is in BUILD status rather than
ACTIVE status.
This fixes the test by using this version of _wait_for_state_change
correctly, not to be confused with the same method in
InstanceHelperMixin which is more accurate (it waits for the
terminal status of the server operation).
Change-Id: I56ff050194d0eb465b8c41795fdea2a8b0d764d6
Closes-Bug: #1850514
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1850514
Title:
ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails
with "Cannot 'migrate' instance while it is in vm_state building"
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) train series:
Confirmed
Bug description:
Seen here:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f05/691390/6/check
/nova-tox-functional/f059dc0/testr_results.html.gz
ft1.1:
nova.tests.functional.libvirt.test_numa_servers.ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshapetesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-10-29 17:33:28,372 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2019-10-29 17:33:28,423 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-10-29 17:33:28,478 INFO [nova.service] Starting scheduler node (version
20.1.0)
2019-10-29 17:33:29,023 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-10-29 17:33:29,026 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-10-29 17:33:29,028 INFO [nova.service] Starting compute node (version
20.1.0)
2019-10-29 17:33:29,030 WARNING [nova.virt.libvirt.driver] The 'vcpu_pin_set'
config option has been deprecated and will be removed in a future release. When
defined, 'vcpu_pin_set' will be used to calculate 'VCPU' inventory and schedule
instances that have 'VCPU' allocations. If you wish to define specific host
CPUs to be used for 'VCPU' or 'PCPU' inventory, you must migrate the
'vcpu_pin_set' config option value to '[compute] cpu_shared_set' and '[compute]
cpu_dedicated_set', respectively, and undefine 'vcpu_pin_set'.
2019-10-29 17:33:29,038 WARNING [nova.virt.libvirt.driver] my_ip address
(38.108.68.36) was not found on any of the interfaces:
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a libvirt version less than 4.0.0 is deprecated. The required minimum version
of libvirt will be raised to 4.0.0 in the next release.
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a QEMU version less than 2.11.0 is deprecated. The required minimum version of
QEMU will be raised to 2.11.0 in the next release.
2019-10-29 17:33:29,059 WARNING [nova.compute.manager] Compute node
test_compute0 not found in the database. If this is the first time this service
is starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,066 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2019-10-29 17:33:29,081 WARNING [nova.compute.manager] No compute node record
found for host test_compute0. If this is the first time this service is
starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,094 WARNING [nova.compute.resource_tracker] No compute
node record for test_compute0:test_compute0
2019-10-29 17:33:29,101 INFO [nova.compute.resource_tracker] Compute node
record created for test_compute0:test_compute0 with uuid:
cada8f90-3f3d-4f22-8312-770a0a818828
2019-10-29 17:33:29,179 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=cada8f90-3f3d-4f22-8312-770a0a818828"
status: 200 len: 26 microversion: 1.14
2019-10-29 17:33:29,192 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 836 microversion: 1.20
2019-10-29 17:33:29,193 INFO [nova.scheduler.client.report]
[req-3fabe3ee-b8f4-4d5e-9c62-185c0ae18c74]
[Yahoo-eng-team] [Bug 1850514] Re: ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails with "Cannot 'migrate' instance while it is in vm_state building"
** Also affects: nova/train
Importance: Undecided
Status: New
** Changed in: nova/train
Status: New => Confirmed
** Changed in: nova/train
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1850514
Title:
ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails
with "Cannot 'migrate' instance while it is in vm_state building"
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) train series:
Confirmed
Bug description:
Seen here:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f05/691390/6/check
/nova-tox-functional/f059dc0/testr_results.html.gz
ft1.1:
nova.tests.functional.libvirt.test_numa_servers.ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshapetesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-10-29 17:33:28,372 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2019-10-29 17:33:28,423 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-10-29 17:33:28,478 INFO [nova.service] Starting scheduler node (version
20.1.0)
2019-10-29 17:33:29,023 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-10-29 17:33:29,026 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-10-29 17:33:29,028 INFO [nova.service] Starting compute node (version
20.1.0)
2019-10-29 17:33:29,030 WARNING [nova.virt.libvirt.driver] The 'vcpu_pin_set'
config option has been deprecated and will be removed in a future release. When
defined, 'vcpu_pin_set' will be used to calculate 'VCPU' inventory and schedule
instances that have 'VCPU' allocations. If you wish to define specific host
CPUs to be used for 'VCPU' or 'PCPU' inventory, you must migrate the
'vcpu_pin_set' config option value to '[compute] cpu_shared_set' and '[compute]
cpu_dedicated_set', respectively, and undefine 'vcpu_pin_set'.
2019-10-29 17:33:29,038 WARNING [nova.virt.libvirt.driver] my_ip address
(38.108.68.36) was not found on any of the interfaces:
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a libvirt version less than 4.0.0 is deprecated. The required minimum version
of libvirt will be raised to 4.0.0 in the next release.
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a QEMU version less than 2.11.0 is deprecated. The required minimum version of
QEMU will be raised to 2.11.0 in the next release.
2019-10-29 17:33:29,059 WARNING [nova.compute.manager] Compute node
test_compute0 not found in the database. If this is the first time this service
is starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,066 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2019-10-29 17:33:29,081 WARNING [nova.compute.manager] No compute node record
found for host test_compute0. If this is the first time this service is
starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,094 WARNING [nova.compute.resource_tracker] No compute
node record for test_compute0:test_compute0
2019-10-29 17:33:29,101 INFO [nova.compute.resource_tracker] Compute node
record created for test_compute0:test_compute0 with uuid:
cada8f90-3f3d-4f22-8312-770a0a818828
2019-10-29 17:33:29,179 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=cada8f90-3f3d-4f22-8312-770a0a818828"
status: 200 len: 26 microversion: 1.14
2019-10-29 17:33:29,192 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 836 microversion: 1.20
2019-10-29 17:33:29,193 INFO [nova.scheduler.client.report]
[req-3fabe3ee-b8f4-4d5e-9c62-185c0ae18c74] Created resource provider record via
placement API for resource provider with UUID
cada8f90-3f3d-4f22-8312-770a0a818828 and name test_compute0.
2019-10-29 17:33:29,195 INFO [nova.virt.libvirt.host] kernel doesn't support
AMD SEV
2019-10-29 17:33:29,227 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/cada8f90-3f3d-4f22-8312-770a0a818828/inventories"
status: 200 len: 405 microversion: 1.26
2019-10-29 17:33:29,241 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/traits?name=in:COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_AMI,COMPUTE_IMAGE_TYPE_RAW,HW_CPU_X86_VMX,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_VOLUME_MULTI_ATTACH,HW_CPU_X86_AESNI,HW_CPU_HYPERTHREADING,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_TRUSTED_CERTS,COMPUTE_DEVICE_TAGGING"
status: 200 len: 447 microversion: 1.6
2019-10-29 17:33:29,266 INFO
