Re: [Openstack] Create VMs with Power architecture(ppc64) on OpenStack running on x86_64 nodes??

[Chris Friesen]

On 11/19/2018 10:25 AM, Yedhu Sastri wrote:

Hello All,

I have some use-cases which I want to test in PowerPC 
architecture(ppc64). As I dont have any Power machines I would like to 
try it with ppc64 VM's. Is it possible to run these kind of VM's on my 
OpenStack cluster(Queens) which runs on X86_64 architecture nodes(OS 
RHEL 7)??


I set the image property architecture=ppc64 to the ppc64 image I 
uploaded to glance but no success in launching VM with those images. I 
am using KVM as hypervisor(qemu 2.10.0) in my compute nodes and I think 
it is not built to support power architecture. For testing without 
OpenStack I manually built qemu on a x86_64 host with ppc64 
support(qemu-ppc64) and then I am able to host the ppc64 VM. But I dont 
know how to do this on my OpenStack cluster. Whether I need to manually 
build qemu on compute nodes with ppc64 support or I need to add some 
lines in my nova.conf to do this?? Any help to solve this issue would be 
much appreciated.


I think that within an OpenStack cluster you'd have to dedicate a whole 
compute node to running ppc64 and have it advertise the architecture as 
ppc64.  Then when you ask for "architecture=ppc64" it should land on 
that node.


If this is for "development, testing or migration of applications to 
Power" have you checked out these people?  They provide free Power VMs.


http://openpower.ic.unicamp.br/minicloud/

Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [Openstack-operators] [nova] about resize the instance

[Chris Friesen]

On 11/8/2018 5:30 AM, Rambo wrote:


  When I resize the instance, the compute node report that 
"libvirtError: internal error: qemu unexpectedly closed the monitor: 
2018-11-08T09:42:04.695681Z qemu-kvm: cannot set up guest memory 
'pc.ram': Cannot allocate memory".Has anyone seen this situation?And 
the ram_allocation_ratio is set 3 in nova.conf.The total memory is 
125G.When I use the "nova hypervisor-show server" command to show the 
compute node's free_ram_mb is -45G.If it is the result of excessive use 
of memory?

Can you give me some suggestions about this?Thank you very much.


I suspect that you simply don't have any available memory on that system.

What is your kernel overcommit setting on the host?  If 
/proc/sys/vm/overcommit_memory is set to 2, then try either changing the 
overcommit ratio or setting it to 1 to see if that makes a difference.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Chris Friesen]

On 10/25/2018 12:00 PM, Jay Pipes wrote:

On 10/25/2018 01:38 PM, Chris Friesen wrote:

On 10/24/2018 9:10 AM, Jay Pipes wrote:
Nova's API has the ability to create "quota classes", which are 
basically limits for a set of resource types. There is something 
called the "default quota class" which corresponds to the limits in 
the CONF.quota section. Quota classes are basically templates of 
limits to be applied if the calling project doesn't have any stored 
project-specific limits.


Has anyone ever created a quota class that is different from "default"?


The Compute API specifically says:

"Only ‘default’ quota class is valid and used to set the default 
quotas, all other quota class would not be used anywhere."


What this API does provide is the ability to set new default quotas 
for *all* projects at once rather than individually specifying new 
defaults for each project.


It's a "defaults template", yes.


Chris, are you advocating for *keeping* the os-quota-classes API?


Nope.  I had two points:

1) It's kind of irrelevant whether anyone has created a quota class 
other than "default" because nova wouldn't use it anyways.


2) The main benefit (as I see it) of the quota class API is to allow 
dynamic adjustment of the default quotas without restarting services.


I totally agree that keystone limits should replace it.  I just didn't 
want the discussion to be focused on the non-default class portion 
because it doesn't matter.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Chris Friesen]

On 10/24/2018 9:10 AM, Jay Pipes wrote:
Nova's API has the ability to create "quota classes", which are 
basically limits for a set of resource types. There is something called 
the "default quota class" which corresponds to the limits in the 
CONF.quota section. Quota classes are basically templates of limits to 
be applied if the calling project doesn't have any stored 
project-specific limits.


Has anyone ever created a quota class that is different from "default"?


The Compute API specifically says:

"Only ‘default’ quota class is valid and used to set the default quotas, 
all other quota class would not be used anywhere."


What this API does provide is the ability to set new default quotas for 
*all* projects at once rather than individually specifying new defaults 
for each project.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Supporting force live-migrate and force evacuate with nested allocations

[Chris Friesen]

On 10/9/2018 1:20 PM, Jay Pipes wrote:

On 10/09/2018 11:04 AM, Balázs Gibizer wrote:

If you do the force flag removal in a nw microversion that also means
(at least to me) that you should not change the behavior of the force
flag in the old microversions.


Agreed.

Keep the old, buggy and unsafe behaviour for the old microversion and in 
a new microversion remove the --force flag entirely and always call GET 
/a_c, followed by a claim_resources() on the destination host.


Agreed.  Once you start looking at more complicated resource topologies, 
you pretty much need to handle allocations properly.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] agreement on how to specify options that impact scheduling and configuration

[Chris Friesen]

While discussing the "Add HPET timer support for x86 guests" 
blueprint[1] one of the items that came up was how to represent what are 
essentially flags that impact both scheduling and configuration.  Eric 
Fried posted a spec to start a discussion[2], and a number of nova 
developers met on a hangout to hash it out.  This is the result.


In this specific scenario the goal was to allow the user to specify that 
their image required a virtual HPET.  For efficient scheduling we wanted 
this to map to a placement trait, and the virt driver also needed to 
enable the feature when booting the instance.  (This can be generalized 
to other similar problems, including how to specify scheduling and 
configuration information for Ironic.)


We discussed two primary approaches:

The first approach was to specify an arbitrary "key=val" in flavor 
extra-specs or image properties, which nova would automatically 
translate into the appropriate placement trait before passing it to 
placement.  Once scheduled to a compute node, the virt driver would look 
for "key=val" in the flavor/image to determine how to proceed.


The second approach was to directly specify the placement trait in the 
flavor extra-specs or image properties.  Once scheduled to a compute 
node, the virt driver would look for the placement trait in the 
flavor/image to determine how to proceed.


Ultimately, the decision was made to go with the second approach.  The 
result is that it is officially acceptable for virt drivers to key off 
placement traits specified in the image/flavor in order to turn on/off 
configuration options for the instance.  If we do get down to the virt 
driver and the trait is set, and the driver for whatever reason 
determines it's not capable of flipping the switch, it should fail.


It should be noted that it only makes sense to use placement traits for 
things that affect scheduling.  If it doesn't affect scheduling, then it 
can be stored in the flavor extra-specs or image properties separate 
from the placement traits.  Also, this approach only makes sense for 
simple booleans.  Anything requiring more complex configuration will 
likely need additional extra-spec and/or config and/or unicorn dust.


Chris

[1] https://blueprints.launchpad.net/nova/+spec/support-hpet-on-guest
[2] 
https://review.openstack.org/#/c/607989/1/specs/stein/approved/support-hpet-on-guest.rst


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [helm] multiple nova compute nodes

[Chris Friesen]

On 10/2/2018 4:15 PM, Giridhar Jayavelu wrote:

Hi,
Currently, all nova components are packaged in same helm chart "nova". Are 
there any plans to separate nova-compute from rest of the services ?
What should be the approach for deploying multiple nova computes nodes using 
OpenStack helm charts?


The nova-compute pods are part of a daemonset which will automatically 
create a nova-compute pod on each node that has the 
"openstack-compute-node=enabled" label.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [storyboard] why use different "bug" tags per project?

[Chris Friesen]

Hi,

At the PTG, it was suggested that each project should tag their bugs 
with "-bug" to avoid tags being "leaked" across projects, or 
something like that.


Could someone elaborate on why this was recommended?  It seems to me 
that it'd be better for all projects to just use the "bug" tag for 
consistency.


If you want to get all bugs in a specific project it would be pretty 
easy to search for stories with a tag of "bug" and a project of "X".


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3] mixed versions?

[Chris Friesen]

On 9/12/2018 12:04 PM, Doug Hellmann wrote:


This came up in a Vancouver summit session (the python3 one I think). General 
consensus there seemed to be that we should have grenade jobs that run python2 
on the old side and python3 on the new side and test the update from one to 
another through a release that way. Additionally there was thought that the 
nova partial job (and similar grenade jobs) could hold the non upgraded node on 
python2 and that would talk to a python3 control plane.

I haven't seen or heard of anyone working on this yet though.

Clark



IIRC, we also talked about not supporting multiple versions of
python on a given node, so all of the services on a node would need
to be upgraded together.


As I understand it, the various services talk to each other using 
over-the-wire protocols.  Assuming this is correct, why would we need to 
ensure they are using the same python version?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [openstack-dev] [all] Bringing the community together (combine the lists!)

[Chris Friesen]

On 08/30/2018 11:03 AM, Jeremy Stanley wrote:


The proposal is simple: create a new openstack-discuss mailing list
to cover all the above sorts of discussion and stop using the other
four.


Do we want to merge usage and development onto one list?  That could be a busy 
list for someone who's just asking a simple usage question.


Alternately, if we are going to merge everything then why not just use the 
"openstack" mailing list since it already exists and there are references to it 
on the web.


(Or do you want to force people to move to something new to make them recognize 
that something has changed?)


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [openstack-dev] [all] Bringing the community together (combine the lists!)

[Chris Friesen]

On 08/30/2018 11:03 AM, Jeremy Stanley wrote:


The proposal is simple: create a new openstack-discuss mailing list
to cover all the above sorts of discussion and stop using the other
four.


Do we want to merge usage and development onto one list?  That could be a busy 
list for someone who's just asking a simple usage question.


Alternately, if we are going to merge everything then why not just use the 
"openstack" mailing list since it already exists and there are references to it 
on the web.


(Or do you want to force people to move to something new to make them recognize 
that something has changed?)


Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [all] Bringing the community together (combine the lists!)

[Chris Friesen]

On 08/30/2018 11:03 AM, Jeremy Stanley wrote:


The proposal is simple: create a new openstack-discuss mailing list
to cover all the above sorts of discussion and stop using the other
four.


Do we want to merge usage and development onto one list?  That could be a busy 
list for someone who's just asking a simple usage question.


Alternately, if we are going to merge everything then why not just use the 
"openstack" mailing list since it already exists and there are references to it 
on the web.


(Or do you want to force people to move to something new to make them recognize 
that something has changed?)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [nova] Nova-scheduler: when are filters applied?

[Chris Friesen]

On 08/30/2018 08:54 AM, Eugen Block wrote:

Hi Jay,


You need to set your ram_allocation_ratio nova.CONF option to 1.0 if you're
running into OOM issues. This will prevent overcommit of memory on your
compute nodes.


I understand that, the overcommitment works quite well most of the time.

It just has been an issue twice when I booted an instance that had been shutdown
a while ago. In the meantime there were new instances created on that
hypervisor, and this old instance caused the OOM.

I would expect that with a ratio of 1.0 I would experience the same issue,
wouldn't I? As far as I understand the scheduler only checks at instance
creation, not when booting existing instances. Is that a correct assumption?


The system keeps track of how much memory is available and how much has been 
assigned to instances on each compute node.  With a ratio of 1.0 it shouldn't 
let you consume more RAM than is available even if the instances have been shut 
down.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Chris Friesen]

On 08/29/2018 10:02 AM, Jay Pipes wrote:


Also, I'd love to hear from anyone in the real world who has successfully
migrated (live or otherwise) an instance that "owns" expensive hardware
(accelerators, SR-IOV PFs, GPUs or otherwise).


I thought cold migration of instances with such devices was supported upstream?

Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Chris Friesen]

On 08/21/2018 04:33 PM, melanie witt wrote:


If we separate into two different groups, all of the items I discussed in my
previous reply will become cross-project efforts. To me, this means that the
placement group will have their own priorities and goal setting process and if
their priorities and goals happen to align with ours on certain items, we can
agree to work on those in collaboration. But I won't make assumptions about how
much alignment we will have. The placement group, as a hypothetical example,
won't necessarily find helping us fix issues with compute functionality like
vGPUs as important as we do, if we need additional work in placement to support 
it.


I guess what I'm saying is that even with placement under nova governance, if 
the placement developers don't want to implement what the nova cores want them 
to implement there really isn't much that the nova cores can do to force them to 
implement it.


And if the placement developers/cores are on board with what nova wants, I don't 
see how it makes a difference if placement is a separate project, especially if 
all existing nova cores are also placement cores to start.


(Note that this is in the context of scratch-your-own-itch developers.  It would 
be very different if the PTL could order developers to work on something.)


Chris



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Chris Friesen]

On 08/21/2018 01:53 PM, melanie witt wrote:


Given all of that, I'm not seeing how *now* is a good time to separate the
placement project under separate governance with separate goals and priorities.
If operators need things for compute, that are well-known and that placement was
created to solve, how will placement have a shared interest in solving compute
problems, if it is not part of the compute project?


As someone who is not involved in the governance of nova, this seems like kind 
of an odd statement for an open-source project.


From the outside, it seems like there is a fairly small pool of active 
placement developers.  And either the placement developers are willing to 
implement the capabilities desired by compute or else they're not.  And if 
they're not, I don't see how being under compute governance would resolve that 
since the only official hard leverage the compute governance has is refusing to 
review/merge placement patches (which wouldn't really help implement compute's 
desires anyways).


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Chris Friesen]

On 08/20/2018 11:44 AM, Zane Bitter wrote:


If you want my personal opinion then I'm a big believer in incremental change.
So, despite recognising that it is born of long experience of which I have been
blissfully mostly unaware, I have to disagree with Chris's position that if
anybody lets you change something then you should try to change as much as
possible in case they don't let you try again. (In fact I'd go so far as to
suggest that those kinds of speculative changes are a contributing factor in
making people reluctant to allow anything to happen at all.) So I'd suggest
splitting the repo, trying things out for a while within Nova's governance, and
then re-evaluating. If there are that point specific problems that separate
governance would appear to address, then it's only a trivial governance patch
and a PTL election away. It should also be much easier to get consensus at that
point than it is at this distance where we're only speculating what things will
be like after the extraction.

I'd like to point out for the record that Mel already said this and said it
better and is AFAICT pretty much never wrong :)


In order to address the "velocity of change in placement" issues, how about 
making the main placement folks members of nova-core with the understanding that 
those powers would only be used in the new placement repo?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/04/2018 05:18 PM, Matt Riedemann wrote:

On 8/3/2018 9:14 AM, Chris Friesen wrote:

I'm of two minds here.

On the one hand, you have the case where the end user has accidentally
requested some combination of things that isn't normally available, and they
need to be able to ask the provider what they did wrong.  I agree that this
case is not really an exception, those resources were never available in the
first place.

On the other hand, suppose the customer issues a valid request and it works,
and then issues the same request again and it fails, leading to a violation of
that customers SLA.  In this case I would suggest that it could be considered
an exception since the system is not delivering the service that it was
intended to deliver.


As I'm sure you're aware Chris, it looks like StarlingX has a kind of
post-mortem query utility to try and figure out where requested resources didn't
end up yielding a resource provider (for a compute node):

https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-94f87e728df6465becce5241f3da53c8R330


But as you noted way earlier in this thread, it might not be the actual reasons
at the time of the failure and in a busy cloud could quickly change.


Just noticed this email, sorry for the delay.

The bit you point out isn't a post-mortem query but rather a way of printing out 
the rejection reasons that were stored (via calls to filter_reject()) at the 
time the request was processed by each filter.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [openstack-dev] [puppet] migrating to storyboard

[Chris Friesen]

On 08/14/2018 10:33 AM, Tobias Urdin wrote:


My goal is that we will be able to swap to Storyboard during the Stein cycle but
considering that we have a low activity on
bugs my opinion is that we could do this swap very easily anything soon as long
as everybody is in favor of it.

Please let me know what you think about moving to Storyboard?


Not a puppet dev, but am currently using Storyboard.

One of the things we've run into is that there is no way to attach log files for 
bug reports to a story.  There's an open story on this[1] but it's not assigned 
to anyone.


Chris


[1] https://storyboard.openstack.org/#!/story/2003071

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [puppet] migrating to storyboard

[Chris Friesen]

On 08/14/2018 10:33 AM, Tobias Urdin wrote:


My goal is that we will be able to swap to Storyboard during the Stein cycle but
considering that we have a low activity on
bugs my opinion is that we could do this swap very easily anything soon as long
as everybody is in favor of it.

Please let me know what you think about moving to Storyboard?


Not a puppet dev, but am currently using Storyboard.

One of the things we've run into is that there is no way to attach log files for 
bug reports to a story.  There's an open story on this[1] but it's not assigned 
to anyone.


Chris


[1] https://storyboard.openstack.org/#!/story/2003071

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova] StarlingX diff analysis

[Chris Friesen]

On 08/07/2018 07:29 AM, Matt Riedemann wrote:

On 8/7/2018 1:10 AM, Flint WALRUS wrote:

I didn’t had time to check StarlingX code quality, how did you feel it while
you were doing your analysis?


I didn't dig into the test diffs themselves, but it was my impression that from
what I was poking around in the local git repo, there were several changes which
didn't have any test coverage.


Full disclosure, I'm on the StarlingX team.

Certainly some changes didn't have unit/functional test coverage, generally due 
to the perceived cost of writing useful tests.  (And when you don't have a lot 
of experience writing tests this becomes a self-fulfilling prophecy.)  On the 
other hand, we had fairly robust periodic integration testing including 
multi-node testing with physical hardware.



For the really big full stack changes (L3 CAT, CPU scaling and shared/pinned
CPUs on same host), toward the end I just started glossing over a lot of that
because it's so much code in so many places, so I can't really speak very well
to how it was written or how well it is tested (maybe WindRiver had a more
robust CI system running integration tests, I don't know).


We didn't have a per-commit CI system, though that's starting to change.  We do 
have a QA team running regression and targeted tests.



There were also some things which would have been caught in code review
upstream. For example, they ignore the "force" parameter for live migration so
that live migration requests always go through the scheduler. However, the
"force" parameter is only on newer microversions. Before that, if you specified
a host at all it would bypass the scheduler, but the change didn't take that
into account, so they still have gaps in some of the things they were trying to
essentially disable in the API.


Agreed, that's not up to upstream quality.  In this case we made some 
simplifying assumptions because our customers were expected to use the matching 
modified clients and to use the "current" microversion rather than explicitly 
specifying older microversions.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we still want to lowercase metadata keys?

[Chris Friesen]

On 08/13/2018 08:26 AM, Jay Pipes wrote:

On 08/13/2018 10:10 AM, Matthew Booth wrote:



I suspect I've misunderstood, but I was arguing this is an anti-goal.
There's no reason to do this if the db is working correctly, and it
would violate the principal of least surprise in dbs with legacy
datasets (being all current dbs). These values have always been mixed
case, lets just leave them be and fix the db.


Do you want case-insensitive keys or do you not want case-insensitive keys?

It seems to me that people complain that MySQL is case-insensitive by default
but actually *like* the concept that a metadata key of "abc" should be "equal
to" a metadata key of "ABC".


How do we behave on PostgreSQL?  (I realize it's unsupported, but it still has 
users.)  It's case-sensitive by default, do we override that?


Personally, I've worked on case-sensitive systems long enough that I'd actually 
be surprised if "abc" matched "ABC". :)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] about live-resize down the instance

[Chris Friesen]

On 08/13/2018 02:07 AM, Rambo wrote:

Hi,all

   I find it is important that live-resize the instance in production
environment,especially live downsize the disk.And we have talked it many
years.But I don't know why the bp[1] didn't approved.Can you tell me more about
this ?Thank you very much.

[1]https://review.openstack.org/#/c/141219/



It's been reviewed a number of times...I thought it was going to get approved 
for Rocky, but I think it didn't quite make it in...you'd have to ask the nova 
cores why not.


It should be noted though that the above live-resize spec explicitly did not 
cover resizing smaller, only larger.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/02/2018 06:27 PM, Jay Pipes wrote:

On 08/02/2018 06:18 PM, Michael Glasgow wrote:



More generally, any time a service fails to deliver a resource which it is
primarily designed to deliver, it seems to me at this stage that should
probably be taken a bit more seriously than just "check the log file, maybe
there's something in there?"  From the user's perspective, if nova fails to
produce an instance, or cinder fails to produce a volume, or neutron fails to
build a subnet, that's kind of a big deal, right?

In such cases, would it be possible to generate a detailed exception object
which contains all the necessary info to ascertain why that specific failure
occurred?


It's not an exception. It's normal course of events. NoValidHosts means there
were no compute nodes that met the requested resource amounts.


I'm of two minds here.

On the one hand, you have the case where the end user has accidentally requested 
some combination of things that isn't normally available, and they need to be 
able to ask the provider what they did wrong.  I agree that this case is not 
really an exception, those resources were never available in the first place.


On the other hand, suppose the customer issues a valid request and it works, and 
then issues the same request again and it fails, leading to a violation of that 
customers SLA.  In this case I would suggest that it could be considered an 
exception since the system is not delivering the service that it was intended to 
deliver.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/02/2018 01:04 PM, melanie witt wrote:


The problem is an infamous one, which is, your users are trying to boot
instances and they get "No Valid Host" and an instance in ERROR state. They
contact support, and now support is trying to determine why NoValidHost
happened. In the past, they would turn on DEBUG log level on the nova-scheduler,
try another request, and take a look at the scheduler logs.


At a previous Summit[1] there were some operators that said they just always ran 
nova-scheduler with debug logging enabled in order to deal with this issue, but 
that it was a pain to isolate the useful logs from the not-useful ones.


Chris


[1] in a discussion related to 
https://blueprints.launchpad.net/nova/+spec/improve-sched-logging


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/02/2018 04:10 AM, Chris Dent wrote:


When people ask for something like what Chris mentioned:

 hosts with enough CPU: 
 hosts that also have enough disk: 
 hosts that also have enough memory: 
 hosts that also meet extra spec host aggregate keys: 
 hosts that also meet image properties host aggregate keys: 
 hosts that also have requested PCI devices: 

What are the operational questions that people are trying to answer
with those results? Is the idea to be able to have some insight into
the resource usage and reporting on and from the various hosts and
discover that things are being used differently than thought? Is
placement a resource monitoring tool, or is it more simple and
focused than that? Or is it that we might have flavors or other
resource requesting constraints that have bad logic and we want to
see at what stage the failure is?  I don't know and I haven't really
seen it stated explicitly here, and knowing it would help.

Do people want info like this for requests as they happen, or to be
able to go back later and try the same request again with some flag
on that says: "diagnose what happened"?

Or to put it another way: Before we design something that provides
the information above, which is a solution to an undescribed
problem, can we describe the problem more completely first to make
sure that what solution we get is the right one. The thing above,
that set of information, is context free.


The reason my organization added additional failure-case logging to the 
pre-placement scheduler was that we were enabling complex features (cpu pinning, 
hugepages, PCI, SRIOV, CPU model requests, NUMA topology, etc.) and we were 
running into scheduling failures, and people were asking the question "why did 
this scheduler request fail to find a valid host?".


There are a few reasons we might want to ask this question.  Some of them 
include:

1) double-checking the scheduler is working properly when first using additional 
features

2) weeding out images/flavors with excessive or mutually-contradictory 
constraints
3) determining whether the cluster needs to be reconfigured to meet user 
requirements


I suspect that something like "do the same request again with a debug flag" 
would cover many scenarios.  I suspect its main weakness would be dealing with 
contention between short-lived entities.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/01/2018 11:34 PM, Joshua Harlow wrote:


And I would be able to say request the explanation for a given request id
(historical even) so that analysis could be done post-change and pre-change (say
I update the algorithm for selection) so that the effects of alternations to
said decisions could be determined.


This would require storing a snapshot of all resources prior to processing every 
request...seems like that could add overhead and increase storage consumption.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/01/2018 11:32 AM, melanie witt wrote:


I think it's definitely a significant issue that troubleshooting "No allocation
candidates returned" from placement is so difficult. However, it's not
straightforward to log detail in placement when the request for allocation
candidates is essentially "SELECT * FROM nodes WHERE cpu usage < needed and disk
usage < needed and memory usage < needed" and the result is returned from the 
API.


I think the only way to get useful info on a failure would be to break down the 
huge SQL statement into subclauses and store the results of the intermediate 
queries.  So then if it failed placement could log something like:


hosts with enough CPU: 
hosts that also have enough disk: 
hosts that also have enough memory: 
hosts that also meet extra spec host aggregate keys: 
hosts that also meet image properties host aggregate keys: 
hosts that also have requested PCI devices: 

And maybe we could optimize the above by only emitting logs where the list has a 
length less than X (to avoid flooding the logs with hostnames in large clusters).


This would let you zero in on the things that finally caused the list to be 
whittled down to nothing.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/01/2018 11:17 AM, Ben Nemec wrote:



On 08/01/2018 11:23 AM, Chris Friesen wrote:



The fact that there is no real way to get the equivalent of the old detailed
scheduler logs is a known shortcoming in placement, and will become more of a
problem if/when we move more complicated things like CPU pinning, hugepages,
and NUMA-awareness into placement.

The problem is that getting useful logs out of placement would require
significant development work.


Yeah, in my case I only had one compute node so it was obvious what the problem
was, but if I had a scheduling failure on a busy cloud with hundreds of nodes I
don't see how you would ever track it down.  Maybe we need to have a discussion
with operators about how often they do post-mortem debugging of this sort of 
thing?


For Wind River's Titanium Cloud it was enough of an issue that we customized the 
scheduler to emit detailed logs on scheduler failure.


We started upstreaming it[1] but the effort stalled out when the upstream folks 
requested major implementation changes.


Chris


[1] https://blueprints.launchpad.net/nova/+spec/improve-sched-logging

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Chris Friesen]

On 08/01/2018 09:58 AM, Andrey Volkov wrote:

Hi,

It seems you need first to check what placement knows about resources of your 
cloud.
This can be done either with REST API [1] or with osc-placement [2].
For osc-placement you could use:

pip install osc-placement
openstack allocation candidate list --resource DISK_GB=20 --resource
MEMORY_MB=2048 --resource VCPU=1 --os-placement-api-version 1.10

And you can explore placement state with other commands like openstack resource
provider list, resource provider inventory list, resource provider usage show.



Unfortunately this doesn't help figure out what the missing resources were *at 
the time of the failure*.


The fact that there is no real way to get the equivalent of the old detailed 
scheduler logs is a known shortcoming in placement, and will become more of a 
problem if/when we move more complicated things like CPU pinning, hugepages, and 
NUMA-awareness into placement.


The problem is that getting useful logs out of placement would require 
significant development work.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] keypair quota usage info for user

[Chris Friesen]

On 07/25/2018 06:22 PM, Alex Xu wrote:



2018-07-26 1:43 GMT+08:00 Chris Friesen mailto:chris.frie...@windriver.com>>:



Keypairs are weird in that they're owned by users, not projects.  This is
arguably wrong, since it can cause problems if a user boots an instance with
their keypair and then gets removed from a project.

Nova microversion 2.54 added support for modifying the keypair associated
with an instance when doing a rebuild.  Before that there was no clean way
to do it.


I don't understand this, we didn't count the keypair usage with the instance
together, we just count the keypair usage for specific user.



I was giving an example of why it's strange that keypairs are owned by users 
rather than projects.  (When instances are owned by projects, and keypairs are 
used to access instances.)


Chris



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] keypair quota usage info for user

[Chris Friesen]

On 07/25/2018 06:21 PM, Alex Xu wrote:



2018-07-26 0:29 GMT+08:00 William M Edmonds mailto:edmon...@us.ibm.com>>:


Ghanshyam Mann mailto:gm...@ghanshyammann.com>>
wrote on 07/25/2018 05:44:46 AM:
... snip ...
> 1. is it ok to show the keypair used info via API ? any original
> rational not to do so or it was just like that from starting.

keypairs aren't tied to a tenant/project, so how could nova track/report a
quota for them on a given tenant/project? Which is how the API is
constructed... note the "tenant_id" in GET /os-quota-sets/{tenant_id}/detail


Keypairs usage is only value for the API 'GET
/os-quota-sets/{tenant_id}/detail?user_id={user_id}'


The objection is that keypairs are tied to the user, not the tenant, so it 
doesn't make sense to specify a tenant_id in the above query.


And for Pike at least I think the above command does not actually show how many 
keypairs have been created by that user...it still shows zero.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] keypair quota usage info for user

[Chris Friesen]

On 07/25/2018 10:29 AM, William M Edmonds wrote:


Ghanshyam Mann  wrote on 07/25/2018 05:44:46 AM:
... snip ...
 > 1. is it ok to show the keypair used info via API ? any original
 > rational not to do so or it was just like that from starting.

keypairs aren't tied to a tenant/project, so how could nova track/report a quota
for them on a given tenant/project? Which is how the API is constructed... note
the "tenant_id" in GET /os-quota-sets/{tenant_id}/detail

 > 2. Because this change will show the keypair used quota information
 > in API's existing filed 'in_use', it is API behaviour change (not
 > interface signature change in backward incompatible way) which can
 > cause interop issue. Should we bump microversion for this change?

If we find a meaningful way to return in_use data for keypairs, then yes, I
would expect a microversion bump so that callers can distinguish between a)
talking to an older installation where in_use is always 0 vs. b) talking to a
newer installation where in_use is 0 because there are really none in use. Or if
we remove keypairs from the response, which at a glance seems to make more
sense, that should also have a microversion bump so that someone who expects the
old response format will still get it.


Keypairs are weird in that they're owned by users, not projects.  This is 
arguably wrong, since it can cause problems if a user boots an instance with 
their keypair and then gets removed from a project.


Nova microversion 2.54 added support for modifying the keypair associated with 
an instance when doing a rebuild.  Before that there was no clean way to do it.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][nova] Running NFV tests in CI

[Chris Friesen]

On 07/24/2018 12:47 PM, Clark Boylan wrote:


Can you get by with qemu or is nested virt required?


Pretty sure that nested virt is needed in order to test CPU pinning.


As for hugepages, I've done a quick survey of cpuinfo across our clouds and all 
seem to have pse available but not all have pdpe1gb available. Are you using 
1GB hugepages?


If we want to test nova's handling of 1G hugepages then I think we'd need 
pdpe1gb.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [StoryBoard] issues found while using storyboard

[Chris Friesen]

Hi,

I'm on a team that is starting to use StoryBoard, and I just thought I'd raise 
some issues I've recently run into.  It may be that I'm making assumptions based 
on previous tools that I've used (Launchpad and Atlassian's Jira) and perhaps 
StoryBoard is intended to be used differently, so if that's the case please let 
me know.


1) There doesn't seem to be a formal way to search for newly-created stories 
that have not yet been triaged.


2) There doesn't seem to be a way to find stories/tasks using arbitrary boolean 
logic, for example something of the form "(A OR (B AND C)) AND NOT D". 
Automatic worklists will only let you do "(A AND B) OR (C AND D) OR (E AND F)" 
and story queries won't even let you do that.


3) I don't see a structured way to specify that a bug has been confirmed by 
someone other than the reporter, or how many people have been impacted by it.


4) I can't find a way to add attachments to a story.  (Like a big log file, or a 
proposed patch, or a screenshot.)


5) I don't see a way to search for stories that have not been assigned to 
someone.

6) This is more a convenience thing, but when looking at someone else's public 
automatic worklist, there's no way to see what the query terms were that 
generated the worklist.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Bug 1781710 killing the check queue

[Chris Friesen]

On 07/18/2018 03:43 PM, melanie witt wrote:

On Wed, 18 Jul 2018 15:14:55 -0500, Matt Riedemann wrote:

On 7/18/2018 1:13 PM, melanie witt wrote:

Can we get rid of multi-create?  It keeps causing complications, and
it already
has weird behaviour if you ask for min_count=X and max_count=Y and only X
instances can be scheduled.  (Currently it fails with NoValidHost, but
it should
arguably start up X instances.)

We've discussed that before but I think users do use it and appreciate
the ability to boot instances in batches (one request). The behavior you
describe could be changed with a microversion, though I'm not sure if
that would mean we have to preserve old behavior with the previous
microversion.

Correct, we can't just remove it since that's a backward incompatible
microversion change. Plus, NFV people*love*  it.


Sorry, I think I might have caused confusion with my question about a
microversion. I was saying that to change the min_count=X and max_count=Y
behavior of raising NoValidHost if X can be satisfied but Y can't, I thought we
could change that in a microversion. And I wasn't sure if that would also mean
we would have to keep the old behavior for previous microversions (and thus
maintain both behaviors).


I understood you. :)

For the case where we could satisfy min_count but not max_count I think we 
*would* need to keep the existing kill-them-all behaviour for existing 
microversions since that's definitely an end-user-visible behaviour.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Bug 1781710 killing the check queue

[Chris Friesen]

On 07/18/2018 10:14 AM, Matt Riedemann wrote:

As can be seen from logstash [1] this bug is hurting us pretty bad in the check
queue.

I thought I originally had this fixed with [2] but that turned out to only be
part of the issue.

I think I've identified the problem but I have failed to write a recreate
regression test [3] because (I think) it's due to random ordering of which
request spec we select to send to the scheduler during a multi-create request
(and I tried making that predictable by sorting the instances by uuid in both
conductor and the scheduler but that didn't make a difference in my test).


Can we get rid of multi-create?  It keeps causing complications, and it already 
has weird behaviour if you ask for min_count=X and max_count=Y and only X 
instances can be scheduled.  (Currently it fails with NoValidHost, but it should 
arguably start up X instances.)



After talking with Sean Mooney, we have another fix which is self-contained to
the scheduler [5] so we wouldn't need to make any changes to the RequestSpec
handling in conductor. It's admittedly a bit hairy, so I'm asking for some eyes
on it since either way we go, we should get going soon before we hit the FF and
RC1 rush which *always* kills the gate.


One of your options mentioned using RequestSpec.num_instances to decide if it's 
in a multi-create.  Is there any reason to persist RequestSpec.num_instances? 
It seems like it's only applicable to the initial request, since after that each 
instance is managed individually.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] creating instance

[Chris Friesen]

On 07/10/2018 03:04 AM, jayshankar nair wrote:

Hi,

I  am trying to create an instance of cirros os(Project/Compute/Instances). I am
getting the following error.

Error: Failed to perform requested operation on instance "cirros1", the instance
has an error status: Please try again later [Error: Build of instance
5de65e6d-fca6-4e78-a688-ead942e8ed2a aborted: The server has either erred or is
incapable of performing the requested operation. (HTTP 500) (Request-ID:
req-91535564-4caf-4975-8eff-7bca515d414e)].

How to debug the error.


You'll want to look at the logs for the individual service.  Since you were 
trying to create a server instance, you probably want to start with the logs for 
the "nova-api" service to see if there are any failure messages.  You can then 
check the logs for "nova-scheduler", "nova-conductor", and "nova-compute". 
There should be something useful in one of those.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] making volume available without stopping VM

[Chris Friesen]

On 06/23/2018 08:38 AM, Volodymyr Litovka wrote:

Dear friends,

I did some tests with making volume available without stopping VM. I'm using
CEPH and these steps produce the following results:

1) openstack volume set --state available [UUID]
- nothing changed inside both VM (volume is still connected) and CEPH
2) openstack volume set --size [new size] --state in-use [UUID]
- nothing changed inside VM (volume is still connected and has an old size)
- size of CEPH volume changed to the new value
3) during these operations I was copying a lot of data from external source and
all md5 sums are the same on both VM and source
4) changes on VM happens upon any kind of power-cycle (e.g. reboot (either soft
or hard): openstack server reboot [--hard] [VM uuid] )
- note: NOT after 'reboot' from inside VM

It seems, that all these manipilations with cinder just update internal
parameters of cinder/CEPH subsystems, without immediate effect for VMs. Is it
safe to use this mechanism in this particular environent (e.g. CEPH as backend)?


There are a different set of instructions[1] which imply that the change should 
be done via the hypervisor, and that the guest will then see the changes 
immediately.


Also, If you resize the backend in a way that bypasses nova, I think it will 
result in the placement data being wrong.  (At least temporarily.)


Chris


[1] 
https://wiki.skytech.dk/index.php/Ceph_-_howto,_rbd,_lvm,_cluster#Online_resizing_of_KVM_images_.28rbd.29



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA-aware live migration: easy but incomplete vs complete but hard

[Chris Friesen]

On 06/21/2018 07:04 AM, Artom Lifshitz wrote:

As I understand it, Artom is proposing to have a larger race window,
essentially
from when the scheduler selects a node until the resource audit runs on that
node.


Exactly. When writing the spec I thought we could just call the resource tracker
to claim the resources when the migration was done. However, when I started
looking at the code in reaction to Sahid's feedback, I noticed that there's no
way to do it without the MoveClaim context (right?)


In the previous patch, the MoveClaim is the thing that calculates the dest NUMA 
topology given the flavor/image, then calls hardware.numa_fit_instance_to_host() 
to figure out what specific host resources to consume.  That claim is then 
associated with the migration object and the instance.migration_context, and 
then we call _update_usage_from_migration() to actually consume the resources on 
the destination.  This all happens within check_can_live_migrate_destination().


As an improvement over what you've got, I think you could just kick off an early 
call of update_available_resource() once the migration is done.  It'd be 
potentially computationally expensive, but it'd reduce the race window.



Keep in mind, we're not making any race windows worse - I'm proposing keeping
the status quo and fixing it later with NUMA in placement (or the resource
tracker if we can swing it).


Well, right now live migration is totally broken so nobody's doing it.  You're 
going to make it kind of work but with racy resource tracking, which could lead 
to people doing it then getting in trouble.  We'll want to make sure there's a 
suitable release note for this.



The resource tracker stuff is just so... opaque. For instance, the original
patch [1] uses a mutated_migration_context around the pre_live_migration call to
the libvirt driver. Would I still need to do that? Why or why not?


The mutated context applies the "new" numa_topology and PCI stuff.

The reason for the mutated context for pre_live_migration() is so that the 
plug_vifs(instance) call will make use of the new macvtap device information. 
See Moshe's comment from Dec 8 2016 at https://review.openstack.org/#/c/244489/46.


I think the mutated context around the call to self.driver.live_migration() is 
so that the new XML represents the newly-claimed pinned CPUs on the destination.



At this point we need to commit to something and roll with it, so I'm sticking
to the "easy way". If it gets shut down in code review, at least we'll have
certainty on how to approach this next cycle.


Yep, I'm cool with incremental improvement.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA-aware live migration: easy but incomplete vs complete but hard

[Chris Friesen]

On 06/21/2018 07:50 AM, Mooney, Sean K wrote:

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]



Side question... does either approach touch PCI device management
during live migration?

I ask because the only workloads I've ever seen that pin guest vCPU
threads to specific host processors -- or make use of huge pages
consumed from a specific host NUMA node -- have also made use of SR-IOV
and/or PCI passthrough. [1]

If workloads that use PCI passthrough or SR-IOV VFs cannot be live
migrated (due to existing complications in the lower-level virt layers)
I don't see much of a point spending lots of developer resources trying
to "fix" this situation when in the real world, only a mythical
workload that uses CPU pinning or huge pages but *doesn't* use PCI
passthrough or SR-IOV VFs would be helped by it.



[Mooney, Sean K]  I would generally agree but with the extention of include 
dpdk based vswitch like ovs-dpdk or vpp.
Cpu pinned or hugepage backed guests generally also have some kind of high 
performance networking solution or use a hardware
Acclaortor like a gpu to justify the performance assertion that pinning of 
cores or ram is required.
Dpdk networking stack would however not require the pci remaping to be 
addressed though I belive that is planned to be added in stine.


Jay, you make a good point but I'll second what Sean says...for the last few 
years my organization has been using a DPDK-accelerated vswitch which performs 
well enough for many high-performance purposes.


In the general case, I think live migration while using physical devices would 
require coordinating the migration with the guest software.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA-aware live migration: easy but incomplete vs complete but hard

[Chris Friesen]

On 06/20/2018 10:00 AM, Sylvain Bauza wrote:


When we reviewed the spec, we agreed as a community to say that we should still
get race conditions once the series is implemented, but at least it helps 
operators.
Quoting :
"It would also be possible for another instance to steal NUMA resources from a
live migrated instance before the latter’s destination compute host has a chance
to claim them. Until NUMA resource providers are implemented [3]
 and allow for an essentially atomic
schedule+claim operation, scheduling and claiming will keep being done at
different times on different nodes. Thus, the potential for races will continue
to exist."
https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html#proposed-change


My understanding of that quote was that we were acknowledging the fact that when 
using the ResourceTracker there was an unavoidable race window between the time 
when the scheduler selected a compute node and when the resources were claimed 
on that compute node in check_can_live_migrate_destination().  And in this model 
no resources are actually *used* until they are claimed.


As I understand it, Artom is proposing to have a larger race window, essentially 
from when the scheduler selects a node until the resource audit runs on that node.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA-aware live migration: easy but incomplete vs complete but hard

[Chris Friesen]

On 06/19/2018 01:59 PM, Artom Lifshitz wrote:

Adding
claims support later on wouldn't change any on-the-wire messaging, it would
just make things work more robustly.


I'm not even sure about that. Assuming [1] has at least the right
idea, it looks like it's an either-or kind of thing: either we use
resource tracker claims and get the new instance NUMA topology that
way, or do what was in the spec and have the dest send it to the
source.


One way or another you need to calculate the new topology in 
ComputeManager.check_can_live_migrate_destination() and communicate that 
information back to the source so that it can be used in 
ComputeManager._do_live_migration().  The previous patches communicated the new 
topoology as part of instance.



That being said, I still think I'm still in favor of choosing the
"easy" way out. For instance, [2] should fail because we can't access
the api db from the compute node.


I think you could use objects.ImageMeta.from_instance(instance) instead of 
request_spec.image.  The limits might be an issue.



So unless there's a simpler way,
using RT claims would involve changing the RPC to add parameters to
check_can_live_migration_destination, which, while not necessarily
bad, seems like useless complexity for a thing we know will get ripped
out.


I agree that it makes sense to get the "simple" option working first.  If we 
later choose to make it work "properly" I don't think it would require undoing 
too much.


Something to maybe factor in to what you're doing--I think there is currently a 
bug when migrating an instance with no numa_topology to a host with a different 
set of host CPUs usable for floating instances--I think it will assume it can 
still float over the same host CPUs as before.  Once we have the ability to 
re-write the instance XML prior to the live-migration it would be good to fix 
this.  I think this would require passing the set of available CPUs on the 
destination back to the host for use when recalculating the XML for the guest. 
(See the "if not guest_cpu_numa_config" case in 
LibvirtDriver._get_guest_numa_config() where "allowed_cpus" is specified, and 
LibvirtDriver._get_guest_config() where guest.cpuset is written.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA-aware live migration: easy but incomplete vs complete but hard

[Chris Friesen]

On 06/18/2018 08:16 AM, Artom Lifshitz wrote:

Hey all,

For Rocky I'm trying to get live migration to work properly for
instances that have a NUMA topology [1].

A question that came up on one of patches [2] is how to handle
resources claims on the destination, or indeed whether to handle that
at all.


I think getting the live migration to work at all is better than having it stay 
broken, so even without resource claiming on the destination it's an improvement 
over the status quo and I think it'd be a desirable change.


However, *not* doing resource claiming means that until the migration is 
complete and the regular resource audit runs on the destination (which could be 
a minute later by default) you could end up having other instances try to use 
the same resources, causing various operations to fail.  I think we'd want to 
have a very clear notice in the release notes about the limitations if we go 
this route.


I'm a little bit worried that waiting for support in placement will result in 
"fully-functional" live migration with dedicated resources being punted out 
indefinitely.  One of the reasons why the spec[1] called for using the existing 
resource tracker was that we don't expect placement to be functional for all 
NUMA-related stuff for a while yet.


For what it's worth, I think the previous patch languished for a number of 
reasons other than the complexity of the code...the original author left, the 
coding style was a bit odd, there was an attempt to make it work even if the 
source was an earlier version, etc.  I think a fresh implementation would be 
less complicated to review.


Given the above, my personal preference would be to merge it even without 
claims, but then try to get the claims support merged as well.  (Adding claims 
support later on wouldn't change any on-the-wire messaging, it would just make 
things work more robustly.)


Chris

[1] 
https://github.com/openstack/nova-specs/blob/master/specs/rocky/approved/numa-aware-live-migration.rst


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] large high-performance ephemeral storage

[Chris Friesen]

On 06/13/2018 07:58 AM, Blair Bethwaite wrote:


Is the collective wisdom to use LVM based instances for these use-cases? Putting
a host filesystem with qcow2 based disk images on it can't help
performance-wise... Though we have not used LVM based instance storage before,
are there any significant gotchas? And furthermore, is it possible to use set IO
QoS limits on these?


LVM has the drawback that deleting instances results in significant disk traffic 
while the volume is scrubbed with zeros.  If you don't care about security you 
can set a config option to turn this off.  Also, while this is happening I think 
your disk resource tracking will be wrong because nova assumes the space is 
available. (At least it used to be this way, I haven't checked that code recently.)


Also, migration and resize are not supported for LVM-backed instances.  I 
proposed a patch to support them (https://review.openstack.org/#/c/337334/) but 
hit issues and never got around to fixing them up.


Chris



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26

[Chris Friesen]

On 06/07/2018 12:07 PM, Matt Riedemann wrote:

On 6/7/2018 12:56 PM, melanie witt wrote:



C) Create a configurable API limit for maximum number of volumes to attach to
a single instance that is either a quota or similar to a quota. Pros: lets
operators opt-in to a maximum that works in their environment. Cons: it's yet
another quota?


This seems the most reasonable to me if we're going to do this, but I'm probably
in the minority. Yes more quota limits sucks, but it's (1) discoverable by API
users and therefore (2) interoperable.


Quota seems like kind of a blunt instrument, since it might not make sense for a 
little single-vCPU guest to get the same number of connections as a massive 
guest with many dedicated vCPUs.  (Since you can fit many more of the former on 
a given compute node.)


If what we care about is the number of connections per compute node it almost 
feels like a resource that should be tracked...but you wouldn't want to have one 
instance consume all of the connections on the node so you're back to needing a 
per-instance limit of some sort.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [nova] isolate hypervisor to project

[Chris Friesen]

On 06/04/2018 05:43 AM, Tobias Urdin wrote:

Hello,

I have received a question about a more specialized use case where we need to
isolate several hypervisors to a specific project. My first thinking was
using nova flavors for only that project and add extra specs properties to
use a specific host aggregate but this means I need to assign values to all
other flavors to not use those which seems weird.

How could I go about solving this the easies/best way or from the
history of the mailing lists, the most supported way since there is a
lot of changes to scheduler/placement part right now?


There was a "Strict isolation of group of hosts for images" spec that was 
proposed for a number of releases but never got accepted:


https://review.openstack.org/#/c/381912/

The idea was to have special metadata on a host aggregate and a new scheduler 
filter such that only instances with images having a property matching the 
metadata would be allowed to land on that host aggregate.


In the end the spec was abandoned (see the final comment in the review) because 
it was expected that a combination of other accepted features would enable the 
desired behaviour.


It might be worth checking out the links in the final comment.

Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova][glance] Deprecation of nova.image.download.modules extension point

[Chris Friesen]

On 05/31/2018 04:14 PM, Moore, Curt wrote:


The challenge is that transferring the Glance image transfer is _glacially slow_
when using the Glance HTTP API (~30 min for a 50GB Windows image (It’s Windows,
it’s huge with all of the necessary tools installed)).  If libvirt can instead
perform an RBD export on the image using the image download functionality, it is
able to download the same image in ~30 sec.


This seems oddly slow.  I just downloaded a 1.6 GB image from glance in slightly 
under 10 seconds.  That would map to about 5 minutes for a 50GB image.




We could look at attaching an additional ephemeral disk to the instance and have
cloudbase-init use it as the pagefile but it appears that if libvirt is using
rbd for its images_type, _all_ disks must then come from Ceph, there is no way
at present to allow the VM image to run from Ceph and have an ephemeral disk
mapped in from node-local storage.  Even still, this would have the effect of
"wasting" Ceph IOPS for the VM disk itself which could be better used for other
purposes.

Based on what I have explained about our use case, is there a better/different
way to accomplish the same goal without using the deprecated image download
functionality?  If not, can we work to "un-deprecate" the download extension
point? Should I work to get the code for this RBD download into the upstream
repository?


Have you considered using compute nodes configured for local storage but then 
use boot-from-volume with cinder and glance both using ceph?  I *think* there's 
an optimization there such that the volume creation is fast.


Assuming the volume creation is indeed fast, in this scenario you could then 
have a local ephemeral/swap disk for your pagefile.  You'd still have your VM 
root disks on ceph though.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] "Resource doesn't have field name."

[Chris Friesen]

I think it'd be worth filing a bug against the "openstack" client...most of the 
clients try to be compatible with any server version.


Probably best to include the details from the run with the --debug option for 
both the new and old version of the client.


Chris

On 05/29/2018 10:36 AM, Ken D'Ambrosio wrote:

On 2018-05-25 13:52, r...@italy1.com wrote:


Use the —debug option to see what calls are going on and which one fails.

Thanks!  That did the trick.  Turned out the image that was causing failure was
one that's been stuck queueing since July, and has no associated name.  The lack
of a name is causing the "openstack image list" to fail.
GET call to None for
http://10.20.139.20:9292/v2/images?marker=2fd99d59-01de-4bde-a432-0e5274f45536
used request id req-6c1a9c23-1edd-4e6f-b970-4bd1ea5a7324
Resource doesn't have field name
Note that the (incredibly, insanely ancient) 1.0.1 release of the "openstack"
CLI command works fine.  This is against Juno, so maybe that's just the way it 
is?
Should that be expected behavior, or a bug?
-Ken


Il giorno 25 mag 2018, alle ore 10:08, Amit Uniyal mailto:auniya...@gmail.com>> ha scritto:


You can use -v (for verbose), directly check logs on openstackclient run.

On Fri, May 25, 2018 at 10:11 PM, Ken D'Ambrosio mailto:k...@jots.org>> wrote:

Hey, all.  I've got a new job, and I tried my first Openstack command on
it -- a Juno cloud -- with Openstack CLI 3.14.0, and it failed.
Specifically:

kdambrosio@mintyfresh:~/oscreds newton(QA/PCI)$ openstack image list
Resource doesn't have field name

glance image-list does fine.

Is this a case of, "Don't do that!"?  Or is there something I should be
digging into?

Thanks!

-Ken

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org

Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org

Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack





___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack




___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [cyborg] [nova] Cyborg quotas

[Chris Friesen]

On 05/19/2018 05:58 PM, Blair Bethwaite wrote:

G'day Jay,

On 20 May 2018 at 08:37, Jay Pipes  wrote:

If it's not the VM or baremetal machine that is using the accelerator, what
is?


It will be a VM or BM, but I don't think accelerators should be tied
to the life of a single instance if that isn't technically necessary
(i.e., they are hot-pluggable devices). I can see plenty of scope for
use-cases where Cyborg is managing devices that are accessible to
compute infrastructure via network/fabric (e.g. rCUDA or dedicated
PCIe fabric). And even in the simple pci passthrough case (vfio or
mdev) it isn't hard to imagine use-cases for workloads that only need
an accelerator sometimes.


Currently nova only supports attach/detach of volumes and network interfaces. 
Is Cyborg looking to implement new Compute API operations to support hot 
attach/detach of various types of accelerators?


Chris



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [OpenStack-Operators][OpenStack] Regarding production grade OpenStack deployment

[Chris Friesen]

Are you talking about downtime of instances (and the dataplane), or of the 
OpenStack API and control plane?


And when you say "zero downtime" are you really talking about "five nines" or 
similar?  Because nothing is truly zero downtime.


If you care about HA then you'll need additional components outside of OpenStack 
proper.  You'll need health checks on your physical nodes, health checks on your 
network links, possibly end-to-end health checks up into the applications 
running in your guests, redundant network paths, redundant controller nodes, HA 
storage, etc.  You'll have to think about how to ensure your database and 
messaging service are HA.  You may want to look at ensuring that your OpenStack 
services do not interfere with the VMs running on that node and vice versa.


We ended up rolling our own install mechanisms because we weren't satisfied with 
any of the existing projects.  That was a while ago now so I don't know how far 
they've come.


Chris

On 05/18/2018 02:07 PM, Fox, Kevin M wrote:

I don't think openstack itself can meet full zero downtime requirements. But if
it can, then I also think none of the deployment tools try and support that use
case either.

Thanks,
Kevin

*From:* Amit Kumar [ebiib...@gmail.com]
*Sent:* Friday, May 18, 2018 3:46 AM
*To:* OpenStack Operators; Openstack
*Subject:* [Openstack-operators] [OpenStack-Operators][OpenStack] Regarding
production grade OpenStack deployment

Hi All,

We want to deploy our private cloud using OpenStack as highly available (zero
downtime (ZDT) - in normal course of action and during upgrades as well)
production grade environment. We came across following tools.

  * We thought of using /Kolla-Kubernetes/ as deployment tool, but we got
feedback from Kolla IRC channel that this project is being retired.
Moreover, we couldn't find latest documents having multi-node deployment
steps and, High Availability support was also not mentioned at all anywhere
in the documentation.
  * Another option to have Kubernetes based deployment is to use OpenStack-Helm,
but it seems the OSH community has not made OSH 1.0 officially available 
yet.
  * Last option, is to use /Kolla-Ansible/, although it is not a Kubernetes
deployment, but seems to have good community support around it. Also, its
documentation talks a little about production grade deployment, probably it
is being used in production grade environments.


If you folks have used any of these tools for deploying OpenStack to fulfill
these requirements: HA and ZDT, then please provide your inputs specifically
about HA and ZDT support of the deployment tool, based on your experience. And
please share if you have any reference links that you have used for achieving HA
and ZDT for the respective tools.

Lastly, if you think we should think that we have missed another more viable and
stable options of deployment tools which can serve our requirement: HA and ZDT,
then please do suggest the same.

Regards,
Amit




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] openstack-dev] [nova] Cannot live migrattion, because error:libvirtError: the CPU is incompatible with host CPU: Host CPU does not provide required features: cmt, mbm_total, mbm_lo

[Chris Friesen]

On 05/13/2018 09:23 PM, 何健乐 wrote:

Hi, all
When I did live-miration , I met the following error: |result 
||=||proxy_call(||self||._autowrap, f, ||*||args, ||*||*||kwargs)|


|May ||14| |10||:||33||:||11| |nova||-||compute[||981335||]: ||File| 
|"/usr/lib64/python2.7/site-packages/libvirt.py"||, line ||1939||, ||in| 
|migrateToURI3|
|May ||14| |10||:||33||:||11| |nova||-||compute[||981335||]: ||if| |ret 
||=||=| |-||1||: ||raise| |libvirtError (||'virDomainMigrateToURI3() 
failed'||, dom||=||self||)|
|May ||14| |10||:||33||:||11| |nova||-||compute[||981335||]: libvirtError: the 
CPU ||is| |incompatible with host CPU: Host CPU does ||not| |provide required 
features: cmt, mbm_total, mbm_local|



Is there any one that has solution for this problem.
Thanks



Can you run "virsh capabilities" and provide the "cpu" section for both the 
source and dest compute nodes?  Can you also provide the "cpu_mode", 
"cpu_model", and "cpu_model_extra_flags" options from the "libvirt" section of 
/etc/nova/nova.conf on both compute nodes?


Chris
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] Windows images into OpenStack

[Chris Friesen]

On 05/11/2018 10:30 AM, Remo Mattei wrote:

Hello guys, I have a need now to get a Windows VM into the OpenStack 
deployment. Can anyone suggest the best way to do this. I have done mostly 
Linux. I could use the ISO and build one within OpenStack not sure I want to go 
that route. I have some Windows that are coming from VMWare.


Here are the instructions if you choose to go the ISO route:

https://docs.openstack.org/image-guide/windows-image.html

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova] reboot a rescued instance?

[Chris Friesen]

On 05/04/2018 07:50 AM, Matt Riedemann wrote:

For full details on this, see the IRC conversation [1].

tl;dr: the nova compute manager and xen virt driver assume that you can reboot a
rescued instance [2] but the API does not allow that [3] and as far as I can
tell, it never has.

I can only assume that Rackspace had an out of tree change to the API to allow
rebooting a rescued instance. I don't know why that wouldn't have been
upstreamed, but the upstream API doesn't allow it. I'm also not aware of
anything internal to nova that reboots an instance in a rescued state.

So the question now is, should we add rescue to the possible states to reboot an
instance in the API? Or just rollback this essentially dead code in the compute
manager and xen virt driver? I don't know if any other virt drivers will support
rebooting a rescued instance.


Not sure where the more recent equivalent is, but the mitaka user guide[1] has 
this:

"Pause, suspend, and stop operations are not allowed when an instance is running 
in rescue mode, as triggering these actions causes the loss of the original 
instance state, and makes it impossible to unrescue the instance."


Would the same logic apply to reboot since it's basically stop/start?

Chris



[1] https://docs.openstack.org/mitaka/user-guide/cli_reboot_an_instance.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] Glance image definition using V2 API

[Chris Friesen]

On 04/22/2018 12:57 AM, Fabian Zimmermann wrote:

Hi,

just create an empty image (Without file or location param), then use
add-location to set your locations.


I was under the impression that the V2 API didn't let you update the location 
unless the "show_multiple_locations" config option was set to "True", which is 
explicitly stated in the help text to be a Bad Idea.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] which SDK to use?

[Chris Friesen]

On 04/20/2018 01:48 AM, Adrian Turjak wrote:

What version of the SDK are you using?


Originally I just used what was installed in my devstack VM, which seems to be 
0.9.17. Upgrading to 0.12.0 allowed it to work.


Thanks,
Chris


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] which SDK to use?

[Chris Friesen]

On 04/19/2018 09:19 PM, Adrian Turjak wrote:

On 20/04/18 01:46, Chris Friesen wrote:

On 04/19/2018 07:01 AM, Jeremy Stanley wrote:



Or, for that matter, leverage OpenStackSDK's ability to pass
arbitrary calls to individual service APIs when you need something
not exposed by the porcelain layer.


Is that documented somewhere?  I spent some time looking at
https://docs.openstack.org/openstacksdk/latest/ and didn't see
anything that looked like that.


Not that I believe, but basically it amounts to that on any service
proxy object you can call .get .post etc. So if the SDK doesn't yet
support a given feature, you can still use the feature yourself, but you
need to do some raw requests work, which honestly isn't that bad.

servers = list(conn.compute.servers())
vs
servers_resp = conn.compute.get("/servers")


I think the second statement above is not quite right.

>>> from openstack import connection
>>> conn = connection.Connection(auth_url=)
>>> [flavor.name for flavor in conn.compute.flavors()]
[u'small', u'medium']
>>> conn.compute.get("/servers")
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'Proxy' object has no attribute 'get'

Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova] Heads up for out-of-tree drivers: supports_recreate -> supports_evacuate

[Chris Friesen]

On 04/19/2018 08:33 AM, Jay Pipes wrote:

On 04/19/2018 09:15 AM, Matthew Booth wrote:

We've had inconsistent naming of recreate/evacuate in Nova for a long
time, and it will persist in a couple of places for a while more.
However, I've proposed the following to rename 'recreate' to
'evacuate' everywhere with no rpc/api impact here:

https://review.openstack.org/560900

One of the things which is renamed is the driver 'supports_recreate'
capability, which I've renamed to 'supports_evacuate'. The above
change updates this for in-tree drivers, but as noted in review this
would impact out-of-tree drivers. If this might affect you, please
follow the above in case it merges.


I have to admit, Matt, I'm a bit confused by this. I was under the impression
that we were trying to *remove* uses of the term "evacuate" as much as possible
because that term is not adequately descriptive of the operation and terms like
"recreate" were more descriptive?


This is a good point.

Personally I'd prefer to see it go the other way and convert everything to the 
"recreate" terminology, including the external API.


From the CLI perspective, it makes no sense that "nova evacuate" operates after 
a host is already down, but "nova evacuate-live" operates on a running host.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] which SDK to use?

[Chris Friesen]

On 04/19/2018 07:01 AM, Jeremy Stanley wrote:

On 2018-04-19 12:24:48 +1000 (+1000), Joshua Hesketh wrote:

There is also nothing stopping you from using both. For example,
you could use the OpenStack SDK for most things but if you hit an
edge case where you need something specific you can then import
the particular client lib.

[...]

Or, for that matter, leverage OpenStackSDK's ability to pass
arbitrary calls to individual service APIs when you need something
not exposed by the porcelain layer.


Is that documented somewhere?  I spent some time looking at
https://docs.openstack.org/openstacksdk/latest/ and didn't see anything that 
looked like that.


Thanks,
Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova] Concern about trusted certificates API change

[Chris Friesen]

On 04/18/2018 10:57 AM, Jay Pipes wrote:

On 04/18/2018 12:41 PM, Matt Riedemann wrote:

There is a compute REST API change proposed [1] which will allow users to pass
trusted certificate IDs to be used with validation of images when creating or
rebuilding a server. The trusted cert IDs are based on certificates stored in
some key manager, e.g. Barbican.

The full nova spec is here [2].

The main concern I have is that trusted certs will not be supported for
volume-backed instances, and some clouds only support volume-backed instances.


Yes. And some clouds only support VMWare vCenter virt driver. And some only
support Hyper-V. I don't believe we should delay adding good functionality to
(large percentage of) clouds because it doesn't yet work with one virt driver or
one piece of (badly-designed) functionality.

 > The way the patch is written is that if the user attempts to

boot from volume with trusted certs, it will fail.


And... I think that's perfectly fine.


If this happens, is it clear to the end-user that the reason the boot failed is 
that the cloud doesn't support trusted cert IDs for boot-from-vol?  If so, then 
I think that's totally fine.


If the error message is unclear, then maybe we should just improve it.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Default scheduler filters survey

[Chris Friesen]

On 04/18/2018 09:17 AM, Artom Lifshitz wrote:


To that end, we'd like to know what filters operators are enabling in
their deployment. If you can, please reply to this email with your
[filter_scheduler]/enabled_filters (or
[DEFAULT]/scheduler_default_filters if you're using an older version)
option from nova.conf. Any other comments are welcome as well :)


RetryFilter
ComputeFilter
AvailabilityZoneFilter
AggregateInstanceExtraSpecsFilter
ComputeCapabilitiesFilter
ImagePropertiesFilter
NUMATopologyFilter
ServerGroupAffinityFilter
ServerGroupAntiAffinityFilter
PciPassthroughFilter


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [placement][nova] Decision time on granular request groups for like resources

[Chris Friesen]

On 04/18/2018 09:58 AM, Matt Riedemann wrote:

On 4/18/2018 9:06 AM, Jay Pipes wrote:

"By default, should resources/traits submitted in different numbered request
groups be supplied by separate resource providers?"


Without knowing all of the hairy use cases, I'm trying to channel my inner
sdague and some of the similar types of discussions we've had to changes in the
compute API, and a lot of the time we've agreed that we shouldn't assume a
default in certain cases.

So for this case, if I'm requesting numbered request groups, why doesn't the API
just require that I pass a query parameter telling it how I'd like those
requests to be handled, either via affinity or anti-affinity.


The request might get unwieldy if we have to specify affinity/anti-affinity for 
each resource.  Maybe you could specify the default for the request and then 
optionally override it for each resource?


I'm not current on the placement implementation details, but would this level of 
flexibility cause complexity problems in the code?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [placement][nova] Decision time on granular request groups for like resources

[Chris Friesen]

On 04/18/2018 08:06 AM, Jay Pipes wrote:

Stackers,

Eric Fried and I are currently at an impasse regarding a decision that will have
far-reaching (and end-user facing) impacts to the placement API and how nova
interacts with the placement service from the nova scheduler.

We need to make a decision regarding the following question:

"By default, should resources/traits submitted in different numbered request
groups be supplied by separate resource providers?"


I'm a bit conflicted.  On the one hand if we're talking about virtual resources 
like "vCPUs" then there's really no reason why they couldn't be sourced from the 
same resource provider.


On the other hand, once we're talking about *physical* resources it seems like 
it might be more common to want them to be coming from different resource 
providers.  We may want memory spread across multiple NUMA nodes for higher 
aggregate bandwidth, we may want VFs from separate PFs for high availability.


I'm half tempted to side with mriedem and say that there is no default and it 
must be explicit, but I'm concerned that this would make the requests a lot 
larger if you have to specify it for every resource.  (Will follow up in a reply 
to mriedem's post.)



Both proposals include ways to specify whether certain resources or whole
request groups can be forced to be sources from either a single provider or from
different providers.

In Viewpoint A, the proposal is to have a can_split=RESOURCE1,RESOURCE2 query
parameter that would indicate which resource classes in the unnumbered request
group that may be split across multiple providers (remember that viewpoint A
considers different request groups to explicitly mean different providers, so it
doesn't make sense to have a can_split query parameter for numbered request
groups).



In Viewpoint B, the proposal is to have a separate_providers=1,2 query parameter
that would indicate that the identified request groups should be sourced from
separate providers. Request groups that are not listed in the separate_providers
query parameter are not guaranteed to be sourced from different providers.


In either viewpoint, is there a way to represent "I want two resource groups, 
with resource X in each group coming from different resource providers 
(anti-affinity) and resource Y from the same resource provider (affinity)?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] which SDK to use?

[Chris Friesen]

On 04/17/2018 07:13 AM, Jeremy Stanley wrote:


The various "client libraries" (e.g. python-novaclient,
python-cinderclient, et cetera) can also be used to that end, but
are mostly for service-to-service communication these days, aren't
extremely consistent with each other, and tend to eventually drop
support for older OpenStack APIs so if you're going to be
interacting with a variety of different OpenStack deployments built
on different releases you may need multiple versions of the client
libraries (depending on what it is you're trying to do).


The above is all good information.

I'd like to add that if you need bleeding-edge functionality in nova it will 
often be implemented first in python-novaclient.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [novaclient] invoking methods on the same client object in different theads caused malformed requests

[Chris Friesen]

On 04/03/2018 04:25 AM, Xiong, Huan wrote:

Hi,

I'm using a cloud benchmarking tool [1], which creates a *single* nova
client object in main thread and invoke methods on that object in different
worker threads. I find it generated malformed requests at random (my
system has python-novaclient 10.1.0 installed). The root cause was because
some methods in novaclient (e.g., those in images.py and networks.py)
changed client object's service_type. Since all threads shared a single
client object, the change caused other threads generated malformed requests
and hence the failure.

I wonder if this is a known issue for novaclient, or the above approach is
not supported?


In general, unless something says it is thread-safe you should assume it is not.

Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Hard fail if you try to rename an AZ with instances in it?

[Chris Friesen]

On 03/27/2018 10:42 AM, Matt Riedemann wrote:

On 3/27/2018 10:37 AM, Jay Pipes wrote:

If we want to actually fix the issue once and for all, we need to make
availability zones a real thing that has a permanent identifier (UUID) and
store that permanent identifier in the instance (not the instance metadata).


Aggregates have a UUID now, exposed in microversion 2.41 (you added it). Is that
what you mean by AZs having a UUID, since AZs are modeled as host aggregates?

One of the alternatives in the spec is not relying on name as a unique
identifier and just make sure everything is held together via the aggregate
UUID, which is now possible.


If we allow non-unique availability zone names, we'd need to display the 
availability zone UUID in horizon when selecting an availability zone.


I think it'd make sense to still require the availability zone names to be 
unique, but internally store the availability zone UUID in the instance instead 
of the name.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] Clock Drift

[Chris Friesen]

On 03/21/2018 08:17 PM, Tyler Bishop wrote:

We've been fighting a constant clock skew issue lately on 4 of our clusters.
  They all use NTP but seem to go into WARN every 12 hours or so.

Anyone else experiencing this?


What clock are you using in the guest?

Chris


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [keystone] batch processing with unified limits

[Chris Friesen]

On 03/07/2018 06:10 PM, Lance Bragstad wrote:

The keystone team is parsing the unified limits discussions from last
week. One of the things we went over as a group was the usability of the
current API [0].

Currently, the create and update APIs support batch processing. So
specifying a list of limits is valid for both. This was a part of the
original proposal as a way to make it easier for operators to set all
their registered limits with a single API call. The API also has unique
IDs for each limit reference. The consensus was that this felt a bit
weird with a resource that contains a unique set of attributes that can
make up a constraints (service, resource type, and optionally a region).
We're discussing ways to make this API more consistent with how the rest
of keystone works while maintaining usability for operators. Does anyone
see issues with supporting batch creation for limits and individual
updates? In other words, removing the ability to update a set of limits
in a single API call, but keeping the ability to create them in batches?


I suspect this would cover the typical usecases we have for standing up new 
clouds or a new service within a cloud.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [openstack-dev] [Openstack-sigs] [keystone] [oslo] new unified limit library

[Chris Friesen]

On 03/07/2018 10:44 AM, Tim Bell wrote:

I think nested quotas would give the same thing, i.e. you have a parent project
for the group and child projects for the users. This would not need user/group
quotas but continue with the ‘project owns resources’ approach.


Agreed, I think that if we support nested quotas with a suitable depth of 
nesting it could be used to handle the existing nova user/project quotas.


Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [Openstack-sigs] [keystone] [oslo] new unified limit library

[Chris Friesen]

On 03/07/2018 10:44 AM, Tim Bell wrote:

I think nested quotas would give the same thing, i.e. you have a parent project
for the group and child projects for the users. This would not need user/group
quotas but continue with the ‘project owns resources’ approach.


Agreed, I think that if we support nested quotas with a suitable depth of 
nesting it could be used to handle the existing nova user/project quotas.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [oslo] new unified limit library

[Chris Friesen]

On 03/07/2018 09:49 AM, Lance Bragstad wrote:



On 03/07/2018 09:31 AM, Chris Friesen wrote:

On 03/07/2018 08:58 AM, Lance Bragstad wrote:

Hi all,

Per the identity-integration track at the PTG [0], I proposed a new oslo
library for services to use for hierarchical quota enforcement [1]. Let
me know if you have any questions or concerns about the library. If the
oslo team would like, I can add an agenda item for next weeks oslo
meeting to discuss.

Thanks,

Lance

[0] https://etherpad.openstack.org/p/unified-limits-rocky-ptg


Looks interesting.

Some complications related to quotas:

1) Nova currently supports quotas for a user/group tuple that can be
stricter than the overall quotas for that group.  As far as I know no
other project supports this.

By group, do you mean keystone group? Or are you talking about the quota
associated to a project?


Sorry, typo.  I meant  quotas for a user/project tuple, which can be stricter 
than the overall quotas for that project.



2) Nova and cinder also support the ability to set the "default" quota
class (which applies to any group that hasn't overridden their
quota).  Currently once it's set there is no way to revert back to the
original defaults.

This sounds like a registered limit [0], but again, I'm not exactly sure
what "group" means in this context. It sounds like group is supposed to
be a limit for a specific project?

[0]
https://docs.openstack.org/keystone/latest/admin/identity-unified-limits.html#registered-limits


Again, should be project instead of group.  And registered limits seem 
essentially analogous.




3) Neutron allows you to list quotas for projects with non-default
quota values.  This is useful, and I'd like to see it extended to
optionally just display the non-default quota values rather than all
quota values for that project.  If we were to support user/group
quotas this would be the only way to efficiently query which
user/group tuples have non-default quotas.

This might be something we can work into the keystone implementation
since it's still marked as experimental [1]. We have two APIs, one
returns the default limits, also known as a registered limit, for a
resource and one that returns the project-specific overrides. It sounds
like you're interested in the second one?

[1]
https://developer.openstack.org/api-ref/identity/v3/index.html#unified-limits


Again, should be user/project tuples.  Yes, in this case I'm talking about the 
project-specific ones.  (It's actually worse if you support user/project limits 
since with the current nova API you can potentially get combinatorial explosion 
if many users are part of many projects.)


I think it would be useful to be able to constrain this query to report limits 
for a specific project, (and a specific user if that will be supported.)


I also think it would be useful to be able to constrain it to report only the 
limits that have been explicitly set (rather than inheriting the default from 
the project or the registered limit).  Maybe it's already intended to work this 
way--if so that should be explicitly documented.



4) In nova, keypairs belong to the user rather than the project.
(This is a bit messed up, but is the current behaviour.)  The quota
for these should really be outside of any group, or else we should
modify nova to make them belong to the project.

I think the initial implementation of a unified limit pattern is
targeting limits and quotas for things associated to projects. In the
future, we can probably expand on the limit information in keystone to
include user-specific limits, which would be great if nova wants to move
away from handling that kind of stuff.


The quota handling for keypairs is a bit messed up in nova right now, but it's 
legacy behaviour at this point.  It'd be nice to be able to get it right if 
we're switching to new quota management mechanisms.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [oslo] new unified limit library

[Chris Friesen]

On 03/07/2018 10:33 AM, Tim Bell wrote:

Sorry, I remember more detail now... it was using the 'owner' of the VM as part 
of the policy rather than quota.

Is there a per-user/per-group quota in Nova?


Nova supports setting quotas for individual users within a project (as long as 
they are smaller than the project quota for that resource).  I'm not sure how 
much it's actually used, or if they want to get rid of it.  (Maybe melwitt can 
chime in.)  But it's there now.


As you can see at 
"https://developer.openstack.org/api-ref/compute/#update-quotas;, there's an 
optional "user_id" field in the request.  Same thing for the "delete" and 
"detailed get" operations.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [oslo] new unified limit library

[Chris Friesen]

On 03/07/2018 08:58 AM, Lance Bragstad wrote:

Hi all,

Per the identity-integration track at the PTG [0], I proposed a new oslo
library for services to use for hierarchical quota enforcement [1]. Let
me know if you have any questions or concerns about the library. If the
oslo team would like, I can add an agenda item for next weeks oslo
meeting to discuss.

Thanks,

Lance

[0] https://etherpad.openstack.org/p/unified-limits-rocky-ptg


Looks interesting.

Some complications related to quotas:

1) Nova currently supports quotas for a user/group tuple that can be stricter 
than the overall quotas for that group.  As far as I know no other project 
supports this.


2) Nova and cinder also support the ability to set the "default" quota class 
(which applies to any group that hasn't overridden their quota).  Currently once 
it's set there is no way to revert back to the original defaults.


3) Neutron allows you to list quotas for projects with non-default quota values. 
 This is useful, and I'd like to see it extended to optionally just display the 
non-default quota values rather than all quota values for that project.  If we 
were to support user/group quotas this would be the only way to efficiently 
query which user/group tuples have non-default quotas.


4) In nova, keypairs belong to the user rather than the project.  (This is a bit 
messed up, but is the current behaviour.)  The quota for these should really be 
outside of any group, or else we should modify nova to make them belong to the 
project.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [libvrit] Can QEMU or LIBVIRT know VM is powering-off

[Chris Friesen]

On 02/21/2018 03:19 PM, Kwan, Louie wrote:

When turning off a VM by doing nova stop,  the Status and Task State is there
for Nova. But can Libvirt / qemu programmatically figure out the ‘Task State’
that the VM is trying to powering-off ?.

For libvirt, it seems only know the “Power State”? Anyway to read the
“powering-off” info?


The fact that you have asked nova to power off the instance means nothing to 
libvirt/qemu.


In the "nova stop" case nova will do some housekeeping stuff, optionally tell 
libvirt to shutdown the domain cleanly, then tell libvirt to destroy the domain, 
then do more housekeeping stuff.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] is there a way to set the number of queues with the virtio-scsi driver ?

[Chris Friesen]

On 02/13/2018 09:32 AM, Vincent Godin wrote:

When creating a image, in metadata "libvirt Driver Options", it's just
possible set the "hw_scsi_model" to "virtio-scsi" but there is no way
to set the number of queues. As this is a big factor of io
improvement, why this option is still not available in openstack ?
Does someone made a patch for this ?


As far as I know this is not currently supported.

There was an old spec proposal to add more virtio-scsi tunables at 
https://review.openstack.org/#/c/103797/ but it seems it was abandoned due to 
objections.  I didn't see anything newer.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [all][Kingbird]Multi-Region Orchestrator

[Chris Friesen]

On 02/05/2018 06:33 PM, Jay Pipes wrote:


It does seem to me, however, that if the intention is *not* to get into the
multi-cloud orchestration game, that a simpler solution to this multi-region
OpenStack deployment use case would be to simply have a global Glance and
Keystone infrastructure that can seamlessly scale to multiple regions.

That way, there'd be no need for replicating anything.


One use-case I've seen for this sort of thing is someone that has multiple 
geographically-separate clouds, and maybe they want to run the same heat stack 
in all of them.


So they can use global glance/keystone, but they need to ensure that they have 
the right flavor(s) available in all the clouds.  This needs to be done by the 
admin user, so it can't be done as part of the normal user's heat stack.


Something like "create a keypair in each of the clouds with the same public key 
and same name" could be done by the end user with some coding, but it's 
convenient to have a tool to do it for you.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][osc] How to deal with add/remove fixed/floating CLIs after novaclient 10.0.0?

[Chris Friesen]

On 01/30/2018 09:15 AM, Matt Riedemann wrote:

The 10.0.0 release of python-novaclient dropped some deprecated CLIs and python
API bindings for the server actions to add/remove fixed and floating IPs:

https://docs.openstack.org/releasenotes/python-novaclient/queens.html#id2

python-openstackclient was using some of those python API bindings from
novaclient which now no longer work:

https://bugs.launchpad.net/python-openstackclient/+bug/1745795




Is there a plan going forward to ensure that python-novaclient and OSC are on 
the same page as far as deprecating CLIs and API bindings?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [openstack-dev] [nova] heads up to users of Aggregate[Core|Ram|Disk]Filter: behavior change in >= Ocata

[Chris Friesen]

On 01/29/2018 07:47 AM, Jay Pipes wrote:


What I believe we can do is change the behaviour so that if a 0.0 value is found
in the nova.conf file on the nova-compute worker, then instead of defaulting to
16.0, the resource tracker would first look to see if the compute node was
associated with a host aggregate that had the "cpu_allocation_ratio" a metadata
item. If one was found, then the host aggregate's cpu_allocation_ratio would be
used. If not, then the 16.0 default would be used.


Presumably you'd need to handle the case where the host is in multiple host 
aggregates that have "cpu_allocation_ratio" as a metadata item.  I think the 
AggregateCoreFilter uses the smallest ratio in this case.


Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] heads up to users of Aggregate[Core|Ram|Disk]Filter: behavior change in >= Ocata

[Chris Friesen]

On 01/18/2018 02:54 PM, Mathieu Gagné wrote:


We use this feature to segregate capacity/hosts based on CPU
allocation ratio using aggregates.
This is because we have different offers/flavors based on those
allocation ratios. This is part of our business model.
A flavor extra_specs is use to schedule instances on appropriate hosts
using AggregateInstanceExtraSpecsFilter.

Our setup has a configuration management system and we use aggregates
exclusively when it comes to allocation ratio.
We do not rely on cpu_allocation_ratio config in nova-scheduler or nova-compute.
One of the reasons is we do not wish to have to
update/package/redeploy our configuration management system just to
add one or multiple compute nodes to an aggregate/capacity pool.
This means anyone (likely an operator or other provisioning
technician) can perform this action without having to touch or even
know about our configuration management system.
We can also transfer capacity from one aggregate to another if there
is a need, again, using aggregate memberships. (we do "evacuate" the
node if there are instances on it)
Our capacity monitoring is based on aggregate memberships and this
offer an easy overview of the current capacity. Note that a host can
be in one and only one aggregate in our setup.


The existing mechanisms to control aggregate membership will still work, so the 
remaining issue is how to control the allocation ratios.


What about implementing a new HTTP API call (as a local private patch) to set 
the allocation ratios for a given host?  This would only be valid for your 
scenario where a given host is only present in a single aggregate, but it would 
allow your techs to modify the ratios.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Booting IPA from cinder: Was: Summary of ironic sessions from Sydney

[Chris Friesen]

On 11/24/2017 10:23 AM, Julia Kreger wrote:

Greetings Michael,

I believe It would need to involve multiple machines at the same time.

I guess there are two different approaches that I think _could_ be
taken to facilitate this:

1) Provide a facility to use a specific volume as the "golden volume"
to boot up for IPA, and then initiate copies of that volume. The
downside that I see is the act of either copying the volume, or
presenting a snapshot of it that will be deleted a little later. I
think that is really going to depend on the back-end, and if the
backend can handle it or not. :\


Don't most reasonable backends support copy-on-write for volumes?  If they do, 
then creating a mostly-read copy of the volume should be low-overhead.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 02:10 PM, Doug Hellmann wrote:

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris



Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?



For what it's worth, we often lag more than 6 months behind master and so some 
of the things we backport wouldn't be allowed by the existing stable branch 
support phases.  (ie aren't "critical" or security patches.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully 
experienced) people doing the downstream work that already happens within the 
various distros.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 10:25 AM, Doug Hellmann wrote:

Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?


One possible reason is to test the stable version of OpenStack against a stable 
version of the underlying OS distro.   (Where that distro may not meet the 
package version requirements for running master.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] Openstack release compatibility

[Chris Friesen]

On 10/31/2017 01:13 AM, haad wrote:

Hi,

We have an OSA installation with 10-12 compute nodes running Mitaka on Ubuntu
16.04. As initially we have not prepared any long term update strategy we would
like to create one now. Plan would be to upgrade it to new OSA
release(Ocata/Pike/Queens) in near future.

Our original plan was to update management/networking/backend at once by using
rolling updates to newer release and then upgrade compute nodes one by one to
new release.. I think that [2] provides a general upgrade manual. Is there any
document describing how are different OSA releases compatible ? Is there any
policy in place about backward compatibility ?


As a general rule, OpenStack only supports an online upgrade of one version at a 
time.  That is, controller nodes running version N can talk to compute nodes 
running version N-1.


If you can tolerate downtime of the API layer, there has been some discussion 
around "skip-level" upgrades.


Chris


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] Compute Node shutdown how to prevent instance suspend

[Chris Friesen]

On 11/02/2017 01:03 AM, Chris wrote:

Hello,

When we shut down a compute node the instances running on it get suspended. This
generates some difficulties with some applications like RabbitMQ dont like to be
suspended. Is there a way to change this behavior so that the running instances
gets killed or shutdown instead?


This may be done by the libvirtd shutdown scripts rather than anything in nova.

As others have said, you should probably either shut down the VMs in an orderly 
fashion or else cold/live migrate the instances off the compute node before 
rebooting it.


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] Live migration failures

[Chris Friesen]

On 11/02/2017 08:48 AM, Mike Lowe wrote:

After moving from CentOS 7.3 to 7.4, I’ve had trouble getting live migration to 
work when a volume is attached.  As it turns out when a live migration takes 
place the libvirt driver rewrites portions of the xml definition for the 
destination hypervisor and gets it wrong.  Here is an example.


Did you change versions of OpenStack as well?

Chris


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [ironic][nova][libvirt] Adding ironic to already-existing kvm deployment

[Chris Friesen]

On 10/18/2017 11:37 AM, Chris Apsey wrote:

All,

I'm working to add baremetal provisioning to an already-existing libvirt (kvm)
deployment.  I was under the impression that our currently-existing endpoints
that already run nova-conductor/nova-scheduler/etc. can be modified to support
both kvm and ironic, but after looking at the ironic installation guide
(https://docs.openstack.org/ironic/latest/install/configure-compute.html), this
doesn't appear to be the case.  Changes are made in the [default] section that
you obviously wouldn't want to apply to your virtual instances.

Given that information, it would appear that ironic requires that you create an
additional host to run nova-compute separately from your already-existing
compute nodes purely for the purpose of managing the ironic-nova integration,
which makes sense.


I think you could run nova-compute with a config file specified as part of the 
commandline.  From what I understand if you run it on the same host as the 
libvirt nova-compute you'd need to use a separate hostname for running the 
ironic nova-compute since nova uses the binary/hostname tuple to uniquely 
identify services in the DB.


Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] Interesting bug when unshelving an instance in an AZ and the AZ is gone

[Chris Friesen]

On 10/16/2017 09:22 AM, Matt Riedemann wrote:


2. Should we null out the instance.availability_zone when it's shelved offloaded
like we do for the instance.host and instance.node attributes? Similarly, we
would not take into account the RequestSpec.availability_zone when scheduling
during unshelve. I tend to prefer this option because once you unshelve offload
an instance, it's no longer associated with a host and therefore no longer
associated with an AZ.


This statement isn't true in the case where the user specifically requested a 
non-default AZ at boot time.



However, is it reasonable to assume that the user doesn't
care that the instance, once unshelved, is no longer in the originally requested
AZ? Probably not a safe assumption.


If they didn't request a non-default AZ then I think we could remove it.


3. When a user unshelves, they can't propose a new AZ (and I don't think we want
to add that capability to the unshelve API). So if the original AZ is gone,
should we automatically remove the RequestSpec.availability_zone when
scheduling? I tend to not like this as it's very implicit and the user could see
the AZ on their instance change before and after unshelve and be confused.


I think allowing the user to specify an AZ on unshelve might be a reasonable 
option.  Or maybe just allow modifying the AZ of a shelved instance without 
unshelving it via a PUT on /servers/{server_id}.



4. We could simply do nothing about this specific bug and assert the behavior is
correct. The user requested an instance in a specific AZ, shelved that instance
and when they wanted to unshelve it, it's no longer available so it fails. The
user would have to delete the instance and create a new instance from the shelve
snapshot image in a new AZ.


I'm inclined to feel that this is operator error.  If they want to delete an AZ 
that has shelved instances then they should talk with their customers and update 
the stored AZ in the DB to a new "valid" one.  (Though currently this would 
require manual DB operations.)


If we implemented Sylvain's spec in #1 above, maybe

we don't have this problem going forward since you couldn't remove/delete an AZ
when there are even shelved offloaded instances still tied to it.


I kind of think it would be okay to disallow deleting AZs with shelved instances 
in them.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should we make rebuild + new image on a volume-backed instance fail fast?

[Chris Friesen]

On 10/06/2017 11:32 AM, Mathieu Gagné wrote:

Why not supporting this use case?


I don't think anyone is suggesting we support do it, but nobody has stepped up 
to actually merge a change that implements it.


I think what Matt is suggesting is that we make it fail fast *now*, and if 
someone else implements it then they can remove the fast failure at the same time.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] What is the goal of AggregateImagePropertiesIsolation filter?

[Chris Friesen]

On 10/05/2017 03:47 AM, Kekane, Abhishek wrote:


So the question here is, what is the exact goal of
AggregateImagePropertiesIsolation' scheduler filter: - Is it one of the 
following:-

1. Matching all metadata of host aggregate with image properties.

2. Matching image properties with host aggregate metadata.

If we decide that actual goal of 'AggregateImagePropertiesIsolation' filter is
as  pointed in #1, then a small correction is required to return False if image
property is not present from the host aggregate metadata.


The name of the filter includes "Isolation", so I think the intent was to 
accomplish #1.  However, as you point out it only fails the filter if both the 
aggregate and the image have the same key but different values and so the 
isolation is imperfect.


At the same time we have the AggregateInstanceExtraSpecsFilter (which ensures 
that any keys in the flavor extra-specs must be present in the aggregate).


Since keys can be specified in either the flavor or the image, it could be 
confusing that the behaviour is different between these two filters.  At the 
same time we don't want to break existing users by modifying the behaviour of 
the existing filters.  Given this it might make sense to create a new filter 
which unifies the checks and behaves the same whether the key is specified in 
the image or the flavor, with some way to toggle whether we want strict 
isolation or not (so that we can ensure only "special" flavors/images are 
allowed to use aggregates with specific limited resources).


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [nova] Should we allow passing new user_data during rebuild?

[Chris Friesen]

On 10/03/2017 11:12 AM, Clint Byrum wrote:


My personal opinion is that rebuild is an anti-pattern for cloud, and
should be frozen and deprecated. It does nothing but complicate Nova
and present challenges for scaling.

That said, if it must stay as a feature, I don't think updating the
user_data should be a priority. At that point, you've basically created an
entirely new server, and you can already do that by creating an entirely
new server.


If you've got a whole heat stack with multiple resources, and you realize that 
you messed up one thing in the template and one of your servers has the wrong 
personality/user_data, it can be useful to be able to rebuild that one server 
without affecting anything else in the stack.  That's just a convenience though.


Chris


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Chris Friesen]

On 09/28/2017 05:29 AM, Sahid Orentino Ferdjaoui wrote:


Only the memory mapped for the guest is striclty allocated from the
NUMA node selected. The QEMU overhead should float on the host NUMA
nodes. So it seems that the "reserved_host_memory_mb" is enough.


What I see in the code/docs doesn't match that, but it's entirely possible I'm 
missing something.


nova uses LibvirtConfigGuestNUMATuneMemory with a mode of "strict" and a nodeset 
of "the host NUMA nodes used by a guest".


For a guest with single NUMA node, I think this would map to libvirt XML of 
something like


  

  

The docs at https://libvirt.org/formatdomain.html#elementsNUMATuning say, "The 
optional memory element specifies how to allocate memory for the domain process 
on a NUMA host."


That seems to me that the qemu overhead would be NUMA-affined, no?  (If you had 
a multi-NUMA-node guest, then the qemu overhead would float across all the NUMA 
nodes used by the guest.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Chris Friesen]

On 09/27/2017 04:55 PM, Blair Bethwaite wrote:

Hi Prema

On 28 September 2017 at 07:10, Premysl Kouril  wrote:

Hi, I work with Jakub (the op of this thread) and here is my two
cents: I think what is critical to realize is that KVM virtual
machines can have substantial memory overhead of up to 25% of memory,
allocated to KVM virtual machine itself. This overhead memory is not


I'm curious what sort of VM configuration causes such high overheads,
is this when using highly tuned virt devices with very large buffers?


For what it's worth we ran into issues a couple years back with I/O to 
RDB-backed disks in writethrough/writeback.  There was a bug that allowed a very 
large number of in-flight operations if the ceph server couldn't keep up with 
the aggregate load.  We hacked a local solution, I'm not sure if it's been dealt 
with upstream.


I think virtio networking has also caused issues, though not as bad.  (But 
noticeable when running close to the line.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Chris Friesen]

On 09/27/2017 03:10 PM, Premysl Kouril wrote:

Lastly, qemu has overhead that varies depending on what you're doing in the
guest.  In particular, there are various IO queues that can consume
significant amounts of memory.  The company that I work for put in a good
bit of effort engineering things so that they work more reliably, and part
of that was determining how much memory to reserve for the host.

Chris


Hi, I work with Jakub (the op of this thread) and here is my two
cents: I think what is critical to realize is that KVM virtual
machines can have substantial memory overhead of up to 25% of memory,
allocated to KVM virtual machine itself. This overhead memory is not
considered in nova code when calculating if the instance being
provisioned actually fits into host's available resources (only the
memory, configured in instance's flavor is considered). And this is
especially being a problem when CPU pinning is used as the memory
allocation is bounded by limits of specific NUMA node (due to the
strict memory allocation mode). This renders the global reservation
parameter reserved_host_memory_mb useless as it doesn't take NUMA into
account.

This KVM virtual machine overhead is what is causing the OOMs in our
infrastructure and that's what we need to fix.


Feel free to report a bug against nova...maybe reserved_host_memory_mb should be 
a list of per-numa-node values.


It's a bit of a hack, but if you use hugepages for all the guests you can 
control the amount of per-numa-node memory reserved for host overhead.


Since the kvm overhead memory is allocated from 4K pages (in my experience) you 
can just choose to leave some memory on each host NUMA node as 4K pages instead 
of allocating them as hugepages.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Chris Friesen]

On 09/27/2017 08:01 AM, Blair Bethwaite wrote:

On 27 September 2017 at 23:19, Jakub Jursa  wrote:

'hw:cpu_policy=dedicated' (while NOT setting 'hw:numa_nodes') results in
libvirt pinning CPU in 'strict' memory mode

(from libvirt xml for given instance)
...
   
 
 
   
...

So yeah, the instance is not able to allocate memory from another NUMA node.


I can't recall what the docs say on this but I wouldn't be surprised
if that was a bug. Though I do think most users would want CPU & NUMA
pinning together (you haven't shared your use case but perhaps you do
too?).


Not a bug.  Once you enable CPU pinning we assume you care about performance, 
and for max performance you need NUMA affinity as well.  (And hugepages are 
beneficial too.)



I'm not quite sure what do you mean by 'memory will be locked for the
guest'. Also, aren't huge pages enabled in kernel by default?


I think that suggestion was probably referring to static hugepages,
which can be reserved (per NUMA node) at boot and then (assuming your
host is configured correctly) QEMU will be able to back guest RAM with
them.


One nice thing about static hugepages is that you pre-allocate them at startup, 
so you can decide on a per-NUMA-node basis how much 4K memory you want to leave 
for incidental host stuff and qemu overhead.  This lets you specify different 
amounts of "host-reserved" memory on different NUMA nodes.


In order to use static hugepages for the guest you need to explicitly ask for a 
page size of 2MB.  (1GB is possible as well but in most cases doesn't buy you 
much compared to 2MB.)


Lastly, qemu has overhead that varies depending on what you're doing in the 
guest.  In particular, there are various IO queues that can consume significant 
amounts of memory.  The company that I work for put in a good bit of effort 
engineering things so that they work more reliably, and part of that was 
determining how much memory to reserve for the host.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Chris Friesen]

On 09/27/2017 03:12 AM, Jakub Jursa wrote:



On 27.09.2017 10:40, Blair Bethwaite wrote:

On 27 September 2017 at 18:14, Stephen Finucane  wrote:

What you're probably looking for is the 'reserved_host_memory_mb' option. This
defaults to 512 (at least in the latest master) so if you up this to 4192 or
similar you should resolve the issue.


I don't see how this would help given the problem description -
reserved_host_memory_mb would only help avoid causing OOM when
launching the last guest that would otherwise fit on a host based on
Nova's simplified notion of memory capacity. It sounds like both CPU
and NUMA pinning are in play here, otherwise the host would have no
problem allocating RAM on a different NUMA node and OOM would be
avoided.


I'm not quite sure if/how OpenStack handles NUMA pinning (why is VM
being killed by OOM rather than having memory allocated on different
NUMA node). Anyway, good point, thank you, I should have a look at exact
parameters passed to QEMU when using CPU pinning.


OpenStack uses strict memory pinning when using CPU pinning and/or memory 
hugepages, so all allocations are supposed to be local.  When it can't allocate 
locally, it triggers OOM.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Is there any reason to exclude originally failed build hosts during live migration?

[Chris Friesen]

On 09/20/2017 12:47 PM, Matt Riedemann wrote:


I wanted to bring it up here in case anyone had a good reason why we should not
continue to exclude originally failed hosts during live migration, even if the
admin is specifying one of those hosts for the live migration destination.

Presumably there was a good reason why the instance failed to build on a host
originally, but that could be for any number of reasons: resource claim failed
during a race, configuration issues, etc. Since we don't really know what
originally happened, it seems reasonable to not exclude originally attempted
build targets since the scheduler filters should still validate them during live
migration (this is all assuming you're not using the 'force' flag with live
migration - and if you are, all bets are off).


As you say, a failure on a host during the original instance creation (which 
could have been a long time ago) is not a reason to bypass that host during 
subsequent operations.


In other words, I think the list of hosts to ignore should be scoped to a single 
"operation" that requires scheduling (which would include any necessary 
rescheduling for that "operation").


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] Pike NOVA Disable and Live Migrate all instances.

[Chris Friesen]

On 09/20/2017 08:59 AM, Steven D. Searles wrote:

Done, thanks for the assistance Chris and everyone.

https://bugs.launchpad.net/nova/+bug/1718455


I pinged the nova devs and mriedem suggested a fix you might want to try.  In 
nova/scheduler/filter_scheduler.py, function select_destinations(), around line 
81 there is a line that reads:


num_instances = spec_obj.num_instances


The suggestion is to change that to the following:

num_instances = len(instance_uuids)


Could you try that and see if it fixes the original problem?


Thanks,
Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Pike NOVA Disable and Live Migrate all instances.

[Chris Friesen]

On 09/19/2017 05:21 PM, Steven D. Searles wrote:

Hello everyone and thanks in advance.  I have Openstack Pike (KVM,FC-SAN/Cinder)
installed in our lab for testing before upgrade and am seeing a possible issue
with disabling a host and live migrating the instances off via the horizon
interface. I can migrate the instances individually via the Openstack client
without issue. It looks like I might be missing something relating to concurrent
jobs in my nova config? Interestingly enough when a migrate host is attempted
via horizon they all fail.  Migrating a single instance through the horizon
interface does function.   Below is what I am seeing in my scheduler log on the
controller when trying to live migrate all instances from a disabled host. I
believe the last line to be the obvious issue but I can not find a nova variable
that seems to relate to this.  Can anyone point me in the right direction?


There's a default limit of 1 outgoing live migration per compute node.  I don't 
think that's the whole issue here though.



*2017-09-19 19:02:30.588 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] There are 1 hosts available
but 10 instances requested to build. select_destinations
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:101*


It's unclear to me why it's trying to schedule 10 instances all at once.  Did 
you originally create all the instances as part of a single boot request?


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] Can Mitaka RamFilter support free hugepages?

[Chris Friesen]

On 09/07/2017 02:27 PM, Sahid Orentino Ferdjaoui wrote:

On Wed, Sep 06, 2017 at 11:57:25PM -0400, Jay Pipes wrote:

Sahid, Stephen, what are your thoughts on this?

On 09/06/2017 10:17 PM, Yaguang Tang wrote:

I think the fact that RamFilter can't deal with huge pages is a bug ,
duo to this limit, we have to set a balance  between normal memory and
huge pages to use RamFilter and NUMATopologyFilter. what do you think
Jay?


Huge Pages has been built on top of the NUMA Topology
implementation. You have to consider isolate all hosts which are going
to handle NUMA instances to a specific host aggregate.

We don't want RAMFilter to handle any NUMA related feature (in Nova
world: hugepages, pinning, realtime...) but we don't want it blocking
scheduling and that should not be the case.

I'm surprised to see this bug I think libvirt is reporting the total
amount of physical RAM available on the host and that is not depending
of the size of the pages.

So If that is true libvirt is reporting only the amount of small pages
memory available we will probably have to fix that point instead of
the RAMFilter.


From what I can see, in Mitaka the RAMFilter uses "free_ram_mb", which is 
calculated from "self.compute_node.memory_mb" and 
"self.compute_node.memory_mb_used".


memory_mb is calculated for libvirt as self._host.get_memory_mb_total(), which 
just calls libvirtmod.virNodeGetInfo() to get the memory available.  As far as I 
can tell, this is the total amount of memory available on the system, without 
paying any attention to hugepages.  (I ran "virsh nodeinfo" to check.)


memory_mb_used() is kind of messed up, it's calculated by running 
get_memory_mb_total() and subtracting the memory showed as available in 
/proc/meminfo.


So from what I can see, RAMFilter should be working with all the memory and 
ignoring hugepages.


(There is some additional complexity involving limits and hypervisor overhead 
that I've skipped over, that might possibly affect what's going on in the 
original problem scenario.)


Chris

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack