Hi all, I'll disable the shelve/unshelve failing tests in Intel NFV CI for the time being till we have clarity on the problem. Stephen Finucane will be working on that in next week or so.
>-----Original Message----- >From: Chris Friesen [mailto:[email protected]] >Sent: Wednesday, May 25, 2016 5:40 PM >To: [email protected] >Subject: Re: [openstack-dev] [nova] Intel NFV CI failing all shelve/unshelve >tests > >On 05/22/2016 05:41 PM, Jay Pipes wrote: >> Hello Novaites, >> >> I've noticed that the Intel NFV CI has been failing all test runs for >> quite some time (at least a few days), always failing the same tests >> around shelve/unshelve operations. > ><snip> > >> I looked through the conductor and compute logs to see if I could find >> any possible reasons for the errors and found a number of the >> following errors in the compute logs: >> >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] Traceback (most recent call last): >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] File >> "/opt/stack/new/nova/nova/compute/manager.py", line 4230, in >> _unshelve_instance >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] with rt.instance_claim(context, >> instance, limits): > ><snip> > >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] >newcell.unpin_cpus(pinned_cpus) >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] File >> "/opt/stack/new/nova/nova/objects/numa.py", line 94, in unpin_cpus >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] pinned=list(self.pinned_cpus)) >> 2016-05-22 22:18:59.403 8145 ERROR nova.compute.manager [instance: >> cae6fd47-0968-4922-a03e-3f2872e4eb52] CPUPinningInvalid: Cannot >> pin/unpin cpus [6] from the following pinned set [0, 2, 4] >> >> on or around the time of the failures in Tempest. >> >> Perhaps tomorrow morning we can look into handling the above exception >> properly from the compute manager, since clearly we shouldn't be >> allowing CPUPinningInvalid to be raised in the resource tracker's >_update_usage() call.... > >First, it seems wrong to me that an _unshelve_instance() call would result in >unpinning any CPUs. If the instance was using pinned CPUs then I would >expect the CPUs to be unpinned when doing the "shelve" operation. When >we do an instance claim as part of the "unshelve" operation we should be >pinning CPUs, not unpinning them. > >Second, the reason why CPUPinningInvalid gets raised in _update_usage() is >that it has discovered an inconsistency in its view of resources. In this >case, >it's trying to unpin CPU 6 from a set of pinned cpus that doesn't include CPU >6. I think this is a valid concern and should result in an error log. >Whether it >should cause the unshelve operation to fail is a separate question, but it's >definitely a symptom that something is wrong with resource tracking on this >compute node. > >Chris > > >__________________________________________________________ >________________ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: OpenStack-dev- >[email protected]?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
