Re: [Openstack] unexpected distribution of compute instances in queens

[Jay Pipes]

On 11/30/2018 05:52 PM, Mike Carden wrote:


Have you set the placement_randomize_allocation_candidates CONF option
and are still seeing the packing behaviour?


No I haven't. Where would be the place to do that? In a nova.conf 
somewhere that the nova-scheduler containers on the controller hosts 
could pick it up?


Just about to deploy for realz with about forty x86 compute nodes, so it 
would be really nice to sort this first. :)


Presuming you are deploying Rocky or Queens,

It goes in the nova.conf file under the [placement] section:

randomize_allocation_candidates = true

The nova.conf file should be the one used by nova-scheduler.

Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] unexpected distribution of compute instances in queens

[Jay Pipes]

On 11/30/2018 02:53 AM, Mike Carden wrote:

I'm seeing a similar issue in Queens deployed via tripleo.

Two x86 compute nodes and one ppc64le node and host aggregates for 
virtual instances and baremetal (x86) instances. Baremetal on x86 is 
working fine.


All VMs get deployed to compute-0. I can live migrate VMs to compute-1 
and all is well, but I tire of being the 'meatspace scheduler'.


LOL, I love that term and will have to remember to use it in the future.

I've looked at the nova.conf in the various nova-xxx containers on the 
controllers, but I have failed to discern the root of this issue.


Have you set the placement_randomize_allocation_candidates CONF option 
and are still seeing the packing behaviour?


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] unexpected distribution of compute instances in queens

[Jay Pipes]

On 11/28/2018 02:50 AM, Zufar Dhiyaulhaq wrote:

Hi,

Thank you. I am able to fix this issue by adding this configuration into 
nova configuration file in controller node.


driver=filter_scheduler


That's the default:

https://docs.openstack.org/ocata/config-reference/compute/config-options.html

So that was definitely not the solution to your problem.

My guess is that Sean's suggestion to randomize the allocation 
candidates fixed your issue.


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[openstack-dev] [nova] Can we deprecate the server backup API please?

[Jay Pipes]

The server backup API was added 8 years ago. It has Nova basically 
implementing a poor-man's cron for some unknown reason (probably because 
the original RAX Cloud Servers API had some similar or identical 
functionality, who knows...).


Can we deprecate this functionality please? It's confusing for end users 
to have an `openstack server image create` and `openstack server backup 
create` command where the latter does virtually the same thing as the 
former only sets up some whacky cron-like thing and deletes images after 
some number of rotations.


If a cloud provider wants to offer some backup thing as a service, they 
could implement this functionality separately IMHO, store the user's 
requested cronjob state in their own system (or in glance which is kind 
of how the existing Nova createBackup functionality works), and run a 
simple cronjob executor that ran `openstack server image create` and 
`openstack image delete` as needed.


This is a perfect example of an API that should never have been added to 
the Compute API, in my opinion, and removing it would be a step in the 
right direction if we're going to get serious about cleaning the Compute 
API up.


Thoughts?
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] VMs cannot fetch metadata

[Jay Pipes]

https://bugs.launchpad.net/neutron/+bug/1777640

Best,
-jay

On 11/06/2018 08:21 AM, Terry Lundin wrote:

Hi all,

I've been struggling with instances suddenly not being able to fetch 
metadata from Openstack Queens (this has worked fine earlier).


Newly created VMs fail to connect to the magic ip, eg. 
http://169.254.169.254/, and won't initialize properly. Subsequently ssh 
login will fail since no key is uploaded.


The symptom is failed requests in the log

*Cirros:*
Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending select for 10.0.0.18...
Lease of 10.0.0.18 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.0.0.1"
cirros-ds 'net' up at 0.94
checkinghttp://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 0.94. request failed
failed 2/20: up 3.01. request failed
failed 3/20: up 5.03. request failed
failed 4/20: up 7.04. request failed

*..and on Centos6:*
ci-info: | Route |   Destination   | Gateway  | Genmask | Interface | 
Flags |
ci-info: 
+---+-+--+-+---+---+
ci-info: |   0   | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 |eth0   |  
UGH  |
ci-info: |   1   | 10.0.0.0| 0.0.0.0  |  255.255.255.0  |eth0   |   
U   |
ci-info: |   2   | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 |eth0   |   
UG  |
ci-info: 
+---+-+--+-+---+---+
2018-11-06 08:10:07,892 - url_helper.py[WARNING]: Calling 
'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: 
unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:08,906 - url_helper.py[WARNING]: Calling 
'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: 
unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:09,925 - url_helper.py[WARNING]: Calling 
'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: 
unexpected error ['NoneType' object has no attribute
...

Using Curl manually, eg. '/curl http://169.254.169.254/openstack/' one 
gets:


/curl: (52) Empty reply from server/

*At the same time this error is showing up in the syslog on the controller:*

Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 460, 
in fire_timers

Nov  6 12:51:01 controller neutron-metadata-agent[3094]: timer()
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 
59, in __call__

Nov  6 12:51:01 controller neutron-metadata-agent[3094]: cb(*args, **kw)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 
219, in main
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: result = 
function(*args, **kwargs)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]:   File 
"/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 793, in 
process_request
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: 
proto.__init__(conn_state, self)
Nov  6 12:51:01 controller neutron-metadata-agent[3094]: TypeError: 
__init__() takes exactly 4 arguments (3 given)


*Neither rebooting the controller, reinstalling neutron, or restarting 
the services will do anything top fix this.*


Has anyone else seen this? We are using Queens with a single controller.

Kind Regards

Terje Lundin






___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova][placement] Placement requests and caching in the resource tracker

[Jay Pipes]

On 11/02/2018 03:22 PM, Eric Fried wrote:

All-

Based on a (long) discussion yesterday [1] I have put up a patch [2]
whereby you can set [compute]resource_provider_association_refresh to
zero and the resource tracker will never* refresh the report client's
provider cache. Philosophically, we're removing the "healing" aspect of
the resource tracker's periodic and trusting that placement won't
diverge from whatever's in our cache. (If it does, it's because the op
hit the CLI, in which case they should SIGHUP - see below.)

*except:
- When we initially create the compute node record and bootstrap its
resource provider.
- When the virt driver's update_provider_tree makes a change,
update_from_provider_tree reflects them in the cache as well as pushing
them back to placement.
- If update_from_provider_tree fails, the cache is cleared and gets
rebuilt on the next periodic.
- If you send SIGHUP to the compute process, the cache is cleared.

This should dramatically reduce the number of calls to placement from
the compute service. Like, to nearly zero, unless something is actually
changing.

Can I get some initial feedback as to whether this is worth polishing up
into something real? (It will probably need a bp/spec if so.)

[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-11-01.log.html#t2018-11-01T17:32:03
[2] https://review.openstack.org/#/c/614886/

==
Background
==
In the Queens release, our friends at CERN noticed a serious spike in
the number of requests to placement from compute nodes, even in a
stable-state cloud. Given that we were in the process of adding a ton of
infrastructure to support sharing and nested providers, this was not
unexpected. Roughly, what was previously:

  @periodic_task:
  GET /resource_providers/$compute_uuid
  GET /resource_providers/$compute_uuid/inventories

became more like:

  @periodic_task:
  # In Queens/Rocky, this would still just return the compute RP
  GET /resource_providers?in_tree=$compute_uuid
  # In Queens/Rocky, this would return nothing
  GET /resource_providers?member_of=...=MISC_SHARES...
  for each provider returned above:  # i.e. just one in Q/R
  GET /resource_providers/$compute_uuid/inventories
  GET /resource_providers/$compute_uuid/traits
  GET /resource_providers/$compute_uuid/aggregates

In a cloud the size of CERN's, the load wasn't acceptable. But at the
time, CERN worked around the problem by disabling refreshing entirely.
(The fact that this seems to have worked for them is an encouraging sign
for the proposed code change.)

We're not actually making use of most of that information, but it sets
the stage for things that we're working on in Stein and beyond, like
multiple VGPU types, bandwidth resource providers, accelerators, NUMA,
etc., so removing/reducing the amount of information we look at isn't
really an option strategically.


I support your idea of getting rid of the periodic refresh of the cache 
in the scheduler report client. Much of that was added in order to 
emulate the original way the resource tracker worked.


Most of the behaviour in the original resource tracker (and some of the 
code still in there for dealing with (surprise!) PCI passthrough devices 
and NUMA topology) was due to doing allocations on the compute node (the 
whole claims stuff). We needed to always be syncing the state of the 
compute_nodes and pci_devices table in the cell database with whatever 
usage information was being created/modified on the compute nodes [0].


All of the "healing" code that's in the resource tracker was basically 
to deal with "soft delete", migrations that didn't complete or work 
properly, and, again, to handle allocations becoming out-of-sync because 
the compute nodes were responsible for allocating (as opposed to the 
current situation we have where the placement service -- via the 
scheduler's call to claim_resources() -- is responsible for allocating 
resources [1]).


Now that we have generation markers protecting both providers and 
consumers, we can rely on those generations to signal to the scheduler 
report client that it needs to pull fresh information about a provider 
or consumer. So, there's really no need to automatically and blindly 
refresh any more.


Best,
-jay

[0] We always need to be syncing those tables because those tables, 
unlike the placement database's data modeling, couple both inventory AND 
usage in the same table structure...


[1] again, except for PCI devices and NUMA topology, because of the 
tight coupling in place with the different resource trackers those types 
of resources use...



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] We're combining the lists!

[Jay Pipes]

I'm not willing to subscribe with a password over a non-TLS connection...

-jay

On 10/29/2018 12:53 PM, Jeremy Stanley wrote:

REMINDER: The openstack, openstack-dev, openstack-sigs and
openstack-operators mailing lists (to which this is being sent) will
be replaced by a new openstack-disc...@lists.openstack.org mailing
list. The new list is open for subscriptions[0] now, but is not yet
accepting posts until Monday November 19 and it's strongly
recommended to subscribe before that date so as not to miss any
messages posted there. The old lists will be configured to no longer
accept posts starting on Monday December 3, but in the interim posts
to the old lists will also get copied to the new list so it's safe
to unsubscribe from them any time after the 19th and not miss any
messages. See my previous notice[1] for details.

For those wondering, we have 127 subscribers so far on
openstack-discuss with 3 weeks to go before it will be put into use
(and 5 weeks now before the old lists are closed down for good).

[0] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
[1] 
http://lists.openstack.org/pipermail/openstack-dev/2018-September/134911.html



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Jay Pipes]

On 10/25/2018 02:44 PM, melanie witt wrote:

On Thu, 25 Oct 2018 14:00:08 -0400, Jay Pipes wrote:

On 10/25/2018 01:38 PM, Chris Friesen wrote:

On 10/24/2018 9:10 AM, Jay Pipes wrote:

Nova's API has the ability to create "quota classes", which are
basically limits for a set of resource types. There is something
called the "default quota class" which corresponds to the limits in
the CONF.quota section. Quota classes are basically templates of
limits to be applied if the calling project doesn't have any stored
project-specific limits.

Has anyone ever created a quota class that is different from "default"?


The Compute API specifically says:

"Only ‘default’ quota class is valid and used to set the default quotas,
all other quota class would not be used anywhere."

What this API does provide is the ability to set new default quotas for
*all* projects at once rather than individually specifying new defaults
for each project.


It's a "defaults template", yes.

The alternative is, you know, to just set the default values in
CONF.quota, which is what I said above. Or, if you want project X to
have different quota limits from those CONF-driven defaults, then set
the quotas for the project to some different values via the
os-quota-sets API (or better yet, just use Keystone's /limits API when
we write the "limits driver" into Nova). The issue is that the
os-quota-classes API is currently blocking *me* writing that "limits
driver" in Nova because I don't want to port nova-specific functionality
(like quota classes) to a limits driver when the Keystone /limits
endpoint doesn't have that functionality and nobody I know of has ever
used it.


When you say it's blocking you from writing the "limits driver" in nova, 
are you saying you're picking up John's unified limits spec [1]? It's 
been in -W mode and hasn't been updated in 4 weeks. In the spec, 
migration from quota classes => registered limits and deprecation of the 
existing quota API and quota classes is described.


Cheers,
-melanie

[1] https://review.openstack.org/602201


Actually, I wasn't familiar with John's spec. I'll review it today.

I was referring to my own attempts to clean up the quota system and 
remove all the limits-related methods from the QuotaDriver class...


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Jay Pipes]

On 10/25/2018 01:38 PM, Chris Friesen wrote:

On 10/24/2018 9:10 AM, Jay Pipes wrote:
Nova's API has the ability to create "quota classes", which are 
basically limits for a set of resource types. There is something 
called the "default quota class" which corresponds to the limits in 
the CONF.quota section. Quota classes are basically templates of 
limits to be applied if the calling project doesn't have any stored 
project-specific limits.


Has anyone ever created a quota class that is different from "default"?


The Compute API specifically says:

"Only ‘default’ quota class is valid and used to set the default quotas, 
all other quota class would not be used anywhere."


What this API does provide is the ability to set new default quotas for 
*all* projects at once rather than individually specifying new defaults 
for each project.


It's a "defaults template", yes.

The alternative is, you know, to just set the default values in 
CONF.quota, which is what I said above. Or, if you want project X to 
have different quota limits from those CONF-driven defaults, then set 
the quotas for the project to some different values via the 
os-quota-sets API (or better yet, just use Keystone's /limits API when 
we write the "limits driver" into Nova). The issue is that the 
os-quota-classes API is currently blocking *me* writing that "limits 
driver" in Nova because I don't want to port nova-specific functionality 
(like quota classes) to a limits driver when the Keystone /limits 
endpoint doesn't have that functionality and nobody I know of has ever 
used it.


Chris, are you advocating for *keeping* the os-quota-classes API?

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Jay Pipes]

On 10/24/2018 02:57 PM, Matt Riedemann wrote:

On 10/24/2018 10:10 AM, Jay Pipes wrote:
I'd like to propose deprecating this API and getting rid of this 
functionality since it conflicts with the new Keystone /limits 
endpoint, is highly coupled with RAX's turnstile middleware and I 
can't seem to find anyone who has ever used it. Deprecating this API 
and functionality would make the transition to a saner quota 
management system much easier and straightforward.


I was trying to do this before it was cool:

https://review.openstack.org/#/c/411035/

I think it was the Pike PTG in ATL where people said, "meh, let's just 
wait for unified limits from keystone and let this rot on the vine".


I'd be happy to restore and update that spec.


++

I think partly things have stalled out because maybe each side (keystone 
+ nova) think the other is working on something but isn't?


I'm currently working on cleaning up the quota system and would be happy 
to deprecate the os-quota-classes API along with the patch series that 
does that cleanup.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][limits] Does ANYONE at all use the quota class functionality in Nova?

[Jay Pipes]

Nova's API has the ability to create "quota classes", which are 
basically limits for a set of resource types. There is something called 
the "default quota class" which corresponds to the limits in the 
CONF.quota section. Quota classes are basically templates of limits to 
be applied if the calling project doesn't have any stored 
project-specific limits.


Has anyone ever created a quota class that is different from "default"?

I'd like to propose deprecating this API and getting rid of this 
functionality since it conflicts with the new Keystone /limits endpoint, 
is highly coupled with RAX's turnstile middleware and I can't seem to 
find anyone who has ever used it. Deprecating this API and functionality 
would make the transition to a saner quota management system much easier 
and straightforward.


Also, I'm apparently blocked now from the operators ML so could someone 
please forward this there?


Thanks,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] Fleio - OpenStack billing - ver. 1.1 released

[Jay Pipes]

Please do not use these mailing lists to advertise 
closed-source/proprietary software solutions.


Thank you,
-jay

On 10/19/2018 05:42 AM, Adrian Andreias wrote:

Hello,

We've just released Fleio version 1.1.

Fleio is a billing solution and control panel for OpenStack public 
clouds and traditional web hosters.


Fleio software automates the entire process for cloud users. New 
customers can use Fleio to sign up for an account, pay invoices, add 
credit to their account, as well as create and manage cloud resources 
such as virtual machines, storage and networking.


Full feature list:
https://fleio.com#features

You can see an online demo:
https://fleio.com/demo

And sign-up for a free trial:
https://fleio.com/signup



Cheers!

- Adrian Andreias
https://fleio.com



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [FEMDC] [Edge] [tripleo] On the use of terms "Edge" and "Far Edge"

[Jay Pipes]

On 10/18/2018 10:23 AM, Dmitry Tantsur wrote:

Hi all,

Sorry for chiming in really late in this topic, but I think $subj is 
worth discussing until we settle harder on the potentially confusing 
terminology.


I think the difference between "Edge" and "Far Edge" is too vague to use 
these terms in practice. Think about the "edge" metaphor itself: 
something rarely has several layers of edges. A knife has an edge, there 
are no far edges. I imagine zooming in and seeing more edges at the 
edge, and then it's quite cool indeed, but is it really a useful 
metaphor for those who never used a strong microscope? :)


I think in the trivial sense "Far Edge" is a tautology, and should be 
avoided. As a weak proof of my words, I already see a lot of smart 
people confusing these two and actually use Central/Edge where they mean 
Edge/Far Edge. I suggest we adopt a different terminology, even if it 
less consistent with typical marketing term around the "Edge" movement.


Now, I don't have really great suggestions. Something that came up in 
TripleO discussions [1] is Core/Hub/Edge, which I think reflects the 
idea better.


I'd be very interested to hear your ideas.


"The Edge" and "Lunatic Fringe".

There, problem solved.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] nova_api resource_providers table issues on ocata

[Jay Pipes]

On 10/17/2018 01:41 AM, Ignazio Cassano wrote:
Hello Jay,  when I add a New compute node I run nova-manage cell_v2 
discover host .

IS it possible this command update the old host uuid in resource table?


No, not unless you already had a nova-compute installed on a host with 
the exact same hostname... which, from looking at the output of your 
SELECT from compute_nodes table, doesn't seem to be the case.


In short, I think both Sylvain and I are stumped as to how your 
placement resource_providers table ended up with these phantom records :(


-jay

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] nova_api resource_providers table issues on ocata

[Jay Pipes]

On 10/16/2018 10:11 AM, Sylvain Bauza wrote:
On Tue, Oct 16, 2018 at 3:28 PM Ignazio Cassano 
mailto:ignaziocass...@gmail.com>> wrote:


Hi everybody,
when on my ocata installation based on centos7 I update (only update
not  changing openstack version) some kvm compute nodes, I
diescovered uuid in resource_providers nova_api db table are
different from uuid in compute_nodes nova db table.
This causes several errors in nova-compute service, because it not
able to receive instances anymore.
Aligning uuid from compute_nodes solves this problem.
Could anyone tel me if it is a bug ?


What do you mean by "updating some compute nodes" ? In Nova, we consider 
uniqueness of compute nodes by a tuple (host, hypervisor_hostname) where 
host is your nova-compute service name for this compute host, and 
hypervisor_hostname is in the case of libvirt the 'hostname' reported by 
the libvirt API [1]


If somehow one of the two values change, then the Nova Resource Tracker 
will consider this new record as a separate compute node, hereby 
creating a new compute_nodes table record, and then a new UUID.
Could you please check your compute_nodes table and see whether some 
entries were recently created ?


The compute_nodes table has no unique constraint on the 
hypervisor_hostname field unfortunately, even though it should. It's not 
like you can have two compute nodes with the same hostname. But, alas, 
this is one of those vestigial tails in nova due to poor initial table 
design and coupling between the concept of a nova-compute service worker 
and the hypervisor resource node itself.


Ignazio, I was tempted to say you may have run into this:

https://bugs.launchpad.net/nova/+bug/1714248

But then I see you're not using Ironic... I'm not entirely sure how you 
ended up with duplicate hypervisor_hostname records for the same compute 
node, but some of those duplicate records must have had the deleted 
field set to a non-zero value, given the constraint we currently have on 
(host, hypervisor_hostname, deleted).


This means that your deployment script or some external scripts must 
have been deleting compute node records somehow, though I'm not entirely 
sure how...


Best,
-jay



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] shall we do a spec review day next tuesday oct 23?

[Jay Pipes]

Works for me too, thanks Melanie.

On Mon, Oct 15, 2018, 10:07 AM melanie witt  wrote:

> Hey all,
>
> Milestone s-1 is coming up next week on Thursday Oct 25 [1] and I was
> thinking it would be a good idea to have a spec review day next week on
> Tuesday Oct 23 to spend some focus on spec reviews together.
>
> Spec freeze is s-2 Jan 10, so the review day isn't related to any
> deadlines, but would just be a way to organize and make sure we have
> initial review on the specs that have been proposed so far.
>
> How does Tuesday Oct 23 work for everyone? Let me know if another day
> works better.
>
> So far, efried and mriedem are on board when I asked in the
> #openstack-nova channel. I'm sending this mail to gather more responses
> asynchronously.
>
> Cheers,
> -melanie
>
> [1] https://wiki.openstack.org/wiki/Nova/Stein_Release_Schedule
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][tc] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

On 10/11/2018 01:08 PM, Zane Bitter wrote:

On 10/10/18 1:35 PM, Jay Pipes wrote:

+tc topic

On 10/10/2018 11:49 AM, Fox, Kevin M wrote:
Sorry. Couldn't quite think of the name. I was meaning, openstack 
project tags.


I think having a tag that indicates the project is no longer using 
SELECT FOR UPDATE (and thus is safe to use multi-writer Galera) is an 
excellent idea, Kevin. ++


I would support such a tag, especially if it came with detailed 
instructions on how to audit your code to make sure you are not doing 
this with sqlalchemy. (Bonus points for a flake8 plugin that can be 
enabled in the gate.)


I can contribute to such a tag's documentation, but I don't currently 
have the bandwidth to start and shepherd it.


(One question for clarification: is this actually _required_ to use 
multi-writer Galera? My previous recollection was that it was possible, 
but inefficient, to use SELECT FOR UPDATE safely as long as you wrote a 
lot of boilerplate to restart the transaction if it failed.)


Certainly not. There is just a higher occurrence of the deadlock error 
in question when using SELECT FOR UPDATE versus using a compare-and-swap 
technique that does things like this:


UPDATE tbl SET field = value, generation = generation + 1
WHERE generation = $expected_generation;

The vast majority of cases I've seen where the deadlock occurred were 
during Rally tests, which were just brute-forcing breakage points and 
not particularly reflecting a real-world usage pattern.


So, in short, yes, it's perfectly safe and fine to use Galera in a 
multi-writer setup from the get-go with most OpenStack projects. It's 
just that *some* OpenStack projects of later releases have fewer code 
areas that aggravate the aforementioned deadlock conditions with Galera 
in multi-writer mode.


Best,
-jay


-jay



From: Jay Pipes [jaypi...@gmail.com]
Sent: Tuesday, October 09, 2018 12:22 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, 
vault, fabio and FQDN endpoints


On 10/09/2018 03:10 PM, Fox, Kevin M wrote:
Oh, this does raise an interesting question... Should such 
information be reported by the projects up to users through labels? 
Something like, "percona_multimaster=safe" Its really difficult for 
folks to know which projects can and can not be used that way 
currently.


Are you referring to k8s labels/selectors? or are you referring to
project tags (you know, part of that whole Big Tent thing...)?

-jay

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][taskflow] Thoughts on moving taskflow out of openstack/oslo

[Jay Pipes]

On 10/10/2018 01:41 PM, Greg Hill wrote:
I've been out of the openstack loop for a few years, so I hope this 
reaches the right folks.


Josh Harlow (original author of taskflow and related libraries) and I 
have been discussing the option of moving taskflow out of the openstack 
umbrella recently. This move would likely also include the futurist and 
automaton libraries that are primarily used by taskflow. The idea would 
be to just host them on github and use the regular Github features for 
Issues, PRs, wiki, etc, in the hopes that this would spur more 
development. Taskflow hasn't had any substantial contributions in 
several years and it doesn't really seem that the current openstack devs 
have a vested interest in moving it forward. I would like to move it 
forward, but I don't have an interest in being bound by the openstack 
workflow (this is why the project stagnated as core reviewers were 
pulled on to other projects and couldn't keep up with the review 
backlog, so contributions ground to a halt).


I'm not sure how using pull requests instead of Gerrit changesets would 
help "core reviewers being pulled on to other projects"?


Is this just about preferring not having a non-human gatekeeper like 
Gerrit+Zuul and being able to just have a couple people merge whatever 
they want to the master HEAD without needing to talk about +2/+W rights?


If it's just about preferring the pull request workflow versus the 
Gerrit rebase workflow, just say so. Same for just preferring the Github 
UI versus Gerrit's UI (which I agree is awful).


Anyway, it's cool with me to "free" taskflow from the onerous yoke of 
OpenStack development if that's what the contributors to it want.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][tc] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

+tc topic

On 10/10/2018 11:49 AM, Fox, Kevin M wrote:

Sorry. Couldn't quite think of the name. I was meaning, openstack project tags.


I think having a tag that indicates the project is no longer using 
SELECT FOR UPDATE (and thus is safe to use multi-writer Galera) is an 
excellent idea, Kevin. ++


-jay



From: Jay Pipes [jaypi...@gmail.com]
Sent: Tuesday, October 09, 2018 12:22 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

On 10/09/2018 03:10 PM, Fox, Kevin M wrote:

Oh, this does raise an interesting question... Should such information be reported by the 
projects up to users through labels? Something like, "percona_multimaster=safe" 
Its really difficult for folks to know which projects can and can not be used that way 
currently.


Are you referring to k8s labels/selectors? or are you referring to
project tags (you know, part of that whole Big Tent thing...)?

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python3] Enabling py37 unit tests

[Jay Pipes]

On 10/10/2018 09:42 AM, Corey Bryant wrote:
On Wed, Oct 10, 2018 at 9:26 AM Andreas Jaeger > wrote:


On 10/10/2018 14.45, Corey Bryant wrote:
 > [...]
 > == Enabling py37 unit tests ==
 >
 > Ubuntu Bionic (18.04 LTS) has the 3.7.0 interpreter and I have
reviews
 > up to define the py37 zuul job and templates here:
 > https://review.openstack.org/#/c/609066
 >
 > I'd like to start submitting reviews to projects to enable
 > openstack-python37-jobs (or variant) for projects that already have
 > openstack-python36-jobs in their .zuul.yaml, zuul.yaml,
 > .zuul.d/project.yaml.

We have projects testing python 3.5 and 3.6 already. Adding 3.7 to
it is
a lot of wasted VMs. Can we limit testing and not test all three,
please?

Well, I wouldn't call any of them wasted if they're testing against a 
supported Python version.


++

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Supporting force live-migrate and force evacuate with nested allocations

[Jay Pipes]

On 10/10/2018 06:32 AM, Balázs Gibizer wrote:

Hi,

Thanks for all the feedback. I feel the following consensus is forming:

1) remove the force flag in a new microversion. I've proposed a spec
about that API change [1]


+1


2) in the old microversions change the blind allocation copy to gather
every resource from a nested source RPs too and try to allocate that
from the destination root RP. In nested allocation cases putting this
allocation to placement will fail and nova will fail the migration /
evacuation. However it will succeed if the server does not need nested
allocation neither on the source nor on the destination host (a.k.a the
legacy case). Or if the server has nested allocation on the source host
but does not need nested allocation on the destination host (for
example the dest host does not have nested RP tree yet).


I disagree on this. I'd rather just do a simple check for >1 provider in 
the allocations on the source and if True, fail hard.


The reverse (going from a non-nested source to a nested destination) 
will hard fail anyway on the destination because the POST /allocations 
won't work due to capacity exceeded (or failure to have any inventory at 
all for certain resource classes on the destination's root compute node).


-jay


I will start implementing #2) as part of the
use-nested-allocation-candidate bp soon and will continue with #1)
later in the cycle.

Nothing is set in stone yet so feedback is still very appreciated.

Cheers,
gibi

[1] https://review.openstack.org/#/c/609330/

On Tue, Oct 9, 2018 at 11:40 AM, Balázs Gibizer
 wrote:

Hi,

Setup
-

nested allocation: an allocation that contains resources from one or
more nested RPs. (if you have better term for this then please
suggest).

If an instance has nested allocation it means that the compute, it
allocates from, has a nested RP tree. BUT if a compute has a nested
RP tree it does not automatically means that the instance, allocating
from that compute, has a nested allocation (e.g. bandwidth inventory
will be on a nested RPs but not every instance will require bandwidth)

Afaiu, as soon as we have NUMA modelling in place the most trivial
servers will have nested allocations as CPU and MEMORY inverntory
will be moved to the nested NUMA RPs. But NUMA is still in the future.

Sidenote: there is an edge case reported by bauzas when an instance
allocates _only_ from nested RPs. This was discussed on last Friday
and it resulted in a new patch[0] but I would like to keep that
discussion separate from this if possible.

Sidenote: the current problem somewhat related to not just nested PRs
but to sharing RPs as well. However I'm not aiming to implement
sharing support in Nova right now so I also try to keep the sharing
disscussion separated if possible.

There was already some discussion on the Monday's scheduler meeting
but I could not attend.
http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-10-08-14.00.log.html#l-20


The meat


Both live-migrate[1] and evacuate[2] has an optional force flag on
the nova REST API. The documentation says: "Force  by not
verifying the provided destination host by the scheduler."

Nova implements this statement by not calling the scheduler if
force=True BUT still try to manage allocations in placement.

To have allocation on the destination host Nova blindly copies the
instance allocation from the source host to the destination host
during these operations. Nova can do that as 1) the whole allocation
is against a single RP (the compute RP) and 2) Nova knows both the
source compute RP and the destination compute RP.

However as soon as we bring nested allocations into the picture that
blind copy will not be feasible. Possible cases
0) The instance has non-nested allocation on the source and would
need non nested allocation on the destination. This works with blindy
copy today.
1) The instance has a nested allocation on the source and would need
a nested allocation on the destination as well.
2) The instance has a non-nested allocation on the source and would
need a nested allocation on the destination.
3) The instance has a nested allocation on the source and would need
a non nested allocation on the destination.

Nova cannot generate nested allocations easily without reimplementing
some of the placement allocation candidate (a_c) code. However I
don't like the idea of duplicating some of the a_c code in Nova.

Nova cannot detect what kind of allocation (nested or non-nested) an
instance would need on the destination without calling placement a_c.
So knowing when to call placement is a chicken and egg problem.

Possible solutions:
A) fail fast

0) Nova can detect that the source allocatioin is non-nested and try
the blindy copy and it will succeed.
1) Nova can detect that the source allocaton is nested and fail the
operation
2) Nova only sees a non nested source allocation. Even if the dest RP
tree is nested it does not mean that the 

Re: [openstack-dev] [nova] Supporting force live-migrate and force evacuate with nested allocations

[Jay Pipes]

On 10/09/2018 05:01 PM, Eric Fried wrote:

On 10/09/2018 02:20 PM, Jay Pipes wrote:

On 10/09/2018 11:04 AM, Balázs Gibizer wrote:

If you do the force flag removal in a nw microversion that also means
(at least to me) that you should not change the behavior of the force
flag in the old microversions.


Agreed.

Keep the old, buggy and unsafe behaviour for the old microversion and in
a new microversion remove the --force flag entirely and always call GET
/a_c, followed by a claim_resources() on the destination host.

For the old microversion behaviour, continue to do the "blind copy" of
allocations from the source compute node provider to the destination
compute node provider.


TBC, for nested/sharing source, we should consolidate all the resources
into a single allocation against the destination's root provider?


No. If there's >1 provider in the allocation for the source, just fail.


That "blind copy" will still fail if there isn't
capacity for the new allocations on the destination host anyway, because
the blind copy is just issuing a POST /allocations, and that code path
still checks capacity on the target resource providers.


What happens when the migration fails, either because of that POST
/allocations, or afterwards? Do we still have the old allocation around
to restore? Cause we can't re-figure it from the now-monolithic
destination allocation.


Again, just hard fail if there's >1 provider in the allocation on the 
source.



There isn't a
code path in the placement API that allows a provider's inventory
capacity to be exceeded by new allocations.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

On 10/09/2018 03:10 PM, Fox, Kevin M wrote:

Oh, this does raise an interesting question... Should such information be reported by the 
projects up to users through labels? Something like, "percona_multimaster=safe" 
Its really difficult for folks to know which projects can and can not be used that way 
currently.


Are you referring to k8s labels/selectors? or are you referring to 
project tags (you know, part of that whole Big Tent thing...)?


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Supporting force live-migrate and force evacuate with nested allocations

[Jay Pipes]

On 10/09/2018 11:04 AM, Balázs Gibizer wrote:

If you do the force flag removal in a nw microversion that also means
(at least to me) that you should not change the behavior of the force
flag in the old microversions.


Agreed.

Keep the old, buggy and unsafe behaviour for the old microversion and in 
a new microversion remove the --force flag entirely and always call GET 
/a_c, followed by a claim_resources() on the destination host.


For the old microversion behaviour, continue to do the "blind copy" of 
allocations from the source compute node provider to the destination 
compute node provider. That "blind copy" will still fail if there isn't 
capacity for the new allocations on the destination host anyway, because 
the blind copy is just issuing a POST /allocations, and that code path 
still checks capacity on the target resource providers. There isn't a 
code path in the placement API that allows a provider's inventory 
capacity to be exceeded by new allocations.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

On 10/09/2018 06:34 AM, Florian Engelmann wrote:

Am 10/9/18 um 11:41 AM schrieb Jay Pipes:

On 10/09/2018 04:34 AM, Christian Berendt wrote:




On 8. Oct 2018, at 19:48, Jay Pipes  wrote:

Why not send all read and all write traffic to a single haproxy 
endpoint and just have haproxy spread all traffic across each Galera 
node?


Galera, after all, is multi-master synchronous replication... so it 
shouldn't matter which node in the Galera cluster you send traffic to.


Probably because of MySQL deadlocks in Galera:

—snip—
Galera cluster has known limitations, one of them is that it uses 
cluster-wide optimistic locking. This may cause some transactions to 
rollback. With an increasing number of writeable masters, the 
transaction rollback rate may increase, especially if there is write 
contention on the same dataset. It is of course possible to retry the 
transaction and perhaps it will COMMIT in the retries, but this will 
add to the transaction latency. However, some designs are deadlock 
prone, e.g sequence tables.

—snap—

Source: 
https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial 



Have you seen the above in production?


Yes of course. Just depends on the application and how high the workload 
gets.


Please read about deadloks and nova in the following report by Intel:

http://galeracluster.com/wp-content/uploads/2017/06/performance_analysis_and_tuning_in_china_mobiles_openstack_production_cloud_2.pdf 


I have read the above. It's a synthetic workload analysis, which is why 
I asked if you'd seen this in production.


For the record, we addressed much of the contention/races mentioned in 
the above around scheduler resource consumption in the Ocata and Pike 
releases of Nova.


I'm aware that the report above identifies the quota handling code in 
Nova as the primary culprit of the deadlock issues but again, it's a 
synthetic workload that is designed to find breaking points. It doesn't 
represent a realistic production workload.


You can read about the deadlock issue in depth on my blog here:

http://www.joinfu.com/2015/01/understanding-reservations-concurrency-locking-in-nova/

That explains where the source of the problem comes from (it's the use 
of SELECT FOR UPDATE, which has been removed from Nova's quota-handling 
code in the Rocky release).


If just Nova is affected we could also create an additional HAProxy 
listener using all Galera nodes with round-robin for all other services?


I fail to see the point of using Galera with a single writer. At that 
point, why bother with Galera at all? Just use a single database node 
with a single slave for backup purposes.



Anyway - proxySQL would be a great extension.


I don't disagree that proxySQL is a good extension. However, it adds yet 
another services to the mesh that needs to be deployed, configured and 
maintained.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

On 10/09/2018 04:34 AM, Christian Berendt wrote:




On 8. Oct 2018, at 19:48, Jay Pipes  wrote:

Why not send all read and all write traffic to a single haproxy endpoint and 
just have haproxy spread all traffic across each Galera node?

Galera, after all, is multi-master synchronous replication... so it shouldn't 
matter which node in the Galera cluster you send traffic to.


Probably because of MySQL deadlocks in Galera:

—snip—
Galera cluster has known limitations, one of them is that it uses cluster-wide 
optimistic locking. This may cause some transactions to rollback. With an 
increasing number of writeable masters, the transaction rollback rate may 
increase, especially if there is write contention on the same dataset. It is of 
course possible to retry the transaction and perhaps it will COMMIT in the 
retries, but this will add to the transaction latency. However, some designs 
are deadlock prone, e.g sequence tables.
—snap—

Source: 
https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial


Have you seen the above in production?

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

[Jay Pipes]

On 10/08/2018 06:14 AM, Florian Engelmann wrote:
3. HAProxy is not capable to handle "read/write" split with Galera. I 
would like to introduce ProxySQL to be able to scale Galera.


Why not send all read and all write traffic to a single haproxy endpoint 
and just have haproxy spread all traffic across each Galera node?


Galera, after all, is multi-master synchronous replication... so it 
shouldn't matter which node in the Galera cluster you send traffic to.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [diskimage-builder] Element pip-and-virtualenv failed to install pip

[Jay Pipes]

On 09/07/2018 03:46 PM, Hang Yang wrote:

Hi there,

I'm new to the DIB tool and ran into an issue when used 2.16.0 DIB tool 
to build a CentOS based image with pip-and-virtualenv element. It failed 
at 
https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#L78 
due to cannot find pip command.


I found the /tmp/get_pip.py was there but totally empty. I have to 
manually add a wget step to retreat the get_pip.py right before the 
failed step then it worked. But should the get_pip.py be downloaded 
automatically by this 
https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/pip-and-virtualenv/source-repository-pip-and-virtualenv 
? Does anyone know how could this issue happen? Thanks in advance for 
any help.


Hi Hang,

Are you using a package or a source-based installation for your dib? The 
reason I ask is because from the docs it seems that the installation 
procedure for pip is quite different depending on whether you're using a 
package or source-based install.


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova] [ironic] agreement on how to specify options that impact scheduling and configuration

[Jay Pipes]

Added [ironic] topic.

On 10/04/2018 06:06 PM, Chris Friesen wrote:
While discussing the "Add HPET timer support for x86 guests" 
blueprint[1] one of the items that came up was how to represent what are 
essentially flags that impact both scheduling and configuration.  Eric 
Fried posted a spec to start a discussion[2], and a number of nova 
developers met on a hangout to hash it out.  This is the result.


In this specific scenario the goal was to allow the user to specify that 
their image required a virtual HPET.  For efficient scheduling we wanted 
this to map to a placement trait, and the virt driver also needed to 
enable the feature when booting the instance.  (This can be generalized 
to other similar problems, including how to specify scheduling and 
configuration information for Ironic.)


We discussed two primary approaches:

The first approach was to specify an arbitrary "key=val" in flavor 
extra-specs or image properties, which nova would automatically 
translate into the appropriate placement trait before passing it to 
placement.  Once scheduled to a compute node, the virt driver would look 
for "key=val" in the flavor/image to determine how to proceed.


The second approach was to directly specify the placement trait in the 
flavor extra-specs or image properties.  Once scheduled to a compute 
node, the virt driver would look for the placement trait in the 
flavor/image to determine how to proceed.


Ultimately, the decision was made to go with the second approach.  The 
result is that it is officially acceptable for virt drivers to key off 
placement traits specified in the image/flavor in order to turn on/off 
configuration options for the instance.  If we do get down to the virt 
driver and the trait is set, and the driver for whatever reason 
determines it's not capable of flipping the switch, it should fail.


Ironicers, pay attention to the above! :) It's a green light from Nova 
to use the traits list contained in the flavor extra specs and image 
metadata when (pre-)configuring an instance.


It should be noted that it only makes sense to use placement traits for 
things that affect scheduling.  If it doesn't affect scheduling, then it 
can be stored in the flavor extra-specs or image properties separate 
from the placement traits.  Also, this approach only makes sense for 
simple booleans.  Anything requiring more complex configuration will 
likely need additional extra-spec and/or config and/or unicorn dust.


Ironicers, also pay close attention to the advice above. Things that are 
not "scheduleable" -- in other words, things that don't filter the list 
of hosts that a workload can land on -- should not go in traits.


Finally, here's the HPET os-traits patch. Reviews welcome (it's tiny patch):

https://review.openstack.org/608258

Best,
-jay


Chris

[1] https://blueprints.launchpad.net/nova/+spec/support-hpet-on-guest
[2] 
https://review.openstack.org/#/c/607989/1/specs/stein/approved/support-hpet-on-guest.rst 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Tenks

[Jay Pipes]

On 10/02/2018 08:58 AM, Mark Goddard wrote:

Hi,

In the most recent Ironic meeting we discussed [1] tenks, and the 
possibility of adding the project under Ironic governance. We agreed to 
move the discussion to the mailing list. I'll introduce the project here 
and give everyone a chance to ask questions. If things appear to move in 
the right direction, I'll propose a vote for inclusion under Ironic's 
governance.


Tenks is a project for managing 'virtual bare metal clusters'. It aims 
to be a drop-in replacement for the various scripts and templates that 
exist in the Ironic devstack plugin for creating VMs to act as bare 
metal nodes in development and test environments. Similar code exists in 
Bifrost and TripleO, and probably other places too. By focusing on one 
project, we can ensure that it works well, and provides all the features 
necessary as support for bare metal in the cloud evolves.


That's tenks the concept. Tenks in reality today is a working version 
1.0, written in Ansible, built by Will Miller (w-miller) during his 
summer placement. Will has returned to his studies, and Will Szumski 
(jovial) has picked it up. You don't have to be called Will to work on 
Tenks, but it helps.


There are various resources available for anyone wishing to find out more:

* Ironic spec review: https://review.openstack.org/#/c/579583
* Documentation: https://tenks.readthedocs.io/en/latest/
* Source code: https://github.com/stackhpc/tenks
* Blog: https://stackhpc.com/tenks.html
* IRC: mgoddard or jovial in #openstack-ironic

What does everyone think? Is this something that the ironic community 
could or should take ownership of?


How does Tenks relate to OVB?

https://openstack-virtual-baremetal.readthedocs.io/en/latest/introduction.html

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter

[Jay Pipes]

On 10/01/2018 06:04 PM, Julia Kreger wrote:

On Mon, Oct 1, 2018 at 2:41 PM Eric Fried  wrote:


 > So say the user requests a node that supports UEFI because their
image
 > needs UEFI. Which workflow would you want here?
 >
 > 1) The operator (or ironic?) has already configured the node to
boot in
 > UEFI mode. Only pre-configured nodes advertise the "supports
UEFI" trait.
 >
 > 2) Any node that supports UEFI mode advertises the trait. Ironic
ensures
 > that UEFI mode is enabled before provisioning the machine.
 >
 > I imagine doing #2 by passing the traits which were specifically
 > requested by the user, from Nova to Ironic, so that Ironic can do the
 > right thing for the user.
 >
 > Your proposal suggests that the user request the "supports UEFI"
trait,
 > and *also* pass some glance UUID which the user understands will make
 > sure the node actually boots in UEFI mode. Something like:
 >
 > openstack server create --flavor METAL_12CPU_128G --trait
SUPPORTS_UEFI
 > --config-data $TURN_ON_UEFI_UUID
 >
 > Note that I pass --trait because I hope that will one day be
supported
 > and we can slow down the flavor explosion.

IMO --trait would be making things worse (but see below). I think UEFI
with Jay's model would be more like:

   openstack server create --flavor METAL_12CPU_128G --config-data $UEFI

where the UEFI profile would be pretty trivial, consisting of
placement.traits.required = ["BOOT_MODE_UEFI"] and object.boot_mode =
"uefi".

I agree that this seems kind of heavy, and that it would be nice to be
able to say "boot mode is UEFI" just once. OTOH I get Jay's point that
we need to separate the placement decision from the instance
configuration.

That said, what if it was:

  openstack config-profile create --name BOOT_MODE_UEFI --json -
  {
   "type": "boot_mode_scheme",
   "version": 123,
   "object": {
       "boot_mode": "uefi"
   },
   "placement": {
    "traits": {
     "required": [
      "BOOT_MODE_UEFI"
     ]
    }
   }
  }
  ^D

And now you could in fact say

  openstack server create --flavor foo --config-profile BOOT_MODE_UEFI

using the profile name, which happens to be the same as the trait name
because you made it so. Does that satisfy the yen for saying it once? (I
mean, despite the fact that you first had to say it three times to get
it set up.)



I do want to zoom out a bit and point out that we're talking about
implementing a new framework of substantial size and impact when the
original proposal - using the trait for both - would just work out of
the box today with no changes in either API. Is it really worth it?


+1000. Reading both of these threads, it feels like we're basically 
trying to make something perfect. I think that is a fine goal, except it 
is unrealistic because the enemy of good is perfection.




By the way, with Jim's --trait suggestion, this:

 > ...dozens of flavors that look like this:
 > - 12CPU_128G_RAID10_DRIVE_LAYOUT_X
 > - 12CPU_128G_RAID5_DRIVE_LAYOUT_X
 > - 12CPU_128G_RAID01_DRIVE_LAYOUT_X
 > - 12CPU_128G_RAID10_DRIVE_LAYOUT_Y
 > - 12CPU_128G_RAID5_DRIVE_LAYOUT_Y
 > - 12CPU_128G_RAID01_DRIVE_LAYOUT_Y

...could actually become:

  openstack server create --flavor 12CPU_128G --trait $WHICH_RAID
--trait
$WHICH_LAYOUT

No flavor explosion.


++ I believe this was where this discussion kind of ended up in.. ?Dublin?

The desire and discussion that led us into complex configuration 
templates and profiles being submitted were for highly complex scenarios 
where users wanted to assert detailed raid configurations to disk. 
Naturally, there are many issues there. The ability to provide such 
detail would be awesome for those 10% of operators that need such 
functionality. Of course, if that is the only path forward, then we 
delay the 90% from getting the minimum viable feature they need.



(Maybe if we called it something other than --trait, like maybe
--config-option, it would let us pretend we're not really overloading a
trait to do config - it's just a coincidence that the config option has
the same name as the trait it causes to be required.)


I feel like it might be confusing, but totally +1 to matching required 
trait name being a thing. That way scheduling is completely decoupled 
and if everything was correct then the request should already be 
scheduled properly.


I guess I'll just drop the idea of doing this properly then. It's true 
that the placement traits concept can be hacked up and the virt driver 
can just pass a list of trait strings to the Ironic API and that's the 
most expedient way to get what the 90% of people apparently want. It's 
also true that it will add a bunch of 

Re: [openstack-dev] [placement] The "intended purpose" of traits

[Jay Pipes]

On 10/01/2018 01:20 PM, Eric Fried wrote:

I agree that we should not overload placement as a mechanism to pass
configuration information ("set up RAID5 on my storage, please") to the
driver. So let's put that aside. (Looking forward to that spec.)


ack.


I still want to use something like "Is capable of RAID5" and/or "Has
RAID5 already configured" as part of a scheduling and placement
decision. Being able to have the GET /a_c response filtered down to
providers with those, ahem, traits is the exact purpose of that operation.


And yep, I have zero problem with this either, as I've noted. This is 
precisely what placement and traits were designed for.



While we're in the neighborhood, we agreed in Denver to use a trait to
indicate which service "owns" a provider [1], so we can eventually
coordinate a smooth handoff of e.g. a device provider from nova to
cyborg. This is certainly not a capability (but it is a trait), and it
can certainly be construed as a key/value (owning_service=cyborg). Are
we rescinding that decision?


Unfortunately I have zero recollection of a conversation about using 
traits for indicating who "owns" a provider. :(


I don't think I would support such a thing -- rather, I would support 
adding an attribute to the provider model itself for an owning service 
or such thing.


That's a great example of where the attribute has specific conceptual 
meaning to placement (the concept of ownership) and should definitely 
not be tucked away, encoded into a trait string.


OK, I'll get back to that spec now... :)

Best,
-jay


[1] https://review.openstack.org/#/c/602160/


I'm working on a spec that will describe a way for the user to instruct
Nova to pass configuration data to the virt driver (or device manager)
before instance spawn. This will have nothing to do with placement or
traits, since this configuration data is not modeling scheduling and
placement decisions.

I hope to have that spec done by Monday so we can discuss on the spec.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter

[Jay Pipes]

On 10/01/2018 09:01 AM, Jim Rollenhagen wrote:
On Mon, Oct 1, 2018 at 8:03 AM Jay Pipes <mailto:jaypi...@gmail.com>> wrote:


On 10/01/2018 04:36 AM, John Garbutt wrote:
 > On Fri, 28 Sep 2018 at 00:46, Jay Pipes mailto:jaypi...@gmail.com>
 > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>>> wrote:
 >
 >     On 09/27/2018 06:23 PM, Matt Riedemann wrote:
 >      > On 9/27/2018 3:02 PM, Jay Pipes wrote:
 >      >> A great example of this would be the proposed "deploy
template"
 >     from
 >      >> [2]. This is nothing more than abusing the placement
traits API in
 >      >> order to allow passthrough of instance configuration data
from the
 >      >> nova flavor extra spec directly into the nodes.instance_info
 >     field in
 >      >> the Ironic database. It's a hack that is abusing the entire
 >     concept of
 >      >> the placement traits concept, IMHO.
 >      >>
 >      >> We should have a way *in Nova* of allowing instance
configuration
 >      >> key/value information to be passed through to the virt
driver's
 >      >> spawn() method, much the same way we provide for
user_data that
 >     gets
 >      >> exposed after boot to the guest instance via configdrive
or the
 >      >> metadata service API. What this deploy template thing is
is just a
 >      >> hack to get around the fact that nova doesn't have a
basic way of
 >      >> passing through some collated instance configuration
key/value
 >      >> information, which is a darn shame and I'm really kind of
 >     annoyed with
 >      >> myself for not noticing this sooner. :(
 >      >
 >      > We talked about this in Dublin through right? We said a good
 >     thing to do
 >      > would be to have some kind of template/profile/config/whatever
 >     stored
 >      > off in glare where schema could be registered on that
thing, and
 >     then
 >      > you pass a handle (ID reference) to that to nova when
creating the
 >      > (baremetal) server, nova pulls it down from glare and hands it
 >     off to
 >      > the virt driver. It's just that no one is doing that work.
 >
 >     No, nobody is doing that work.
 >
 >     I will if need be if it means not hacking the placement API
to serve
 >     this purpose (for which it wasn't intended).
 >
 >
 > Going back to the point Mark Goddard made, there are two things here:
 >
 > 1) Picking the correct resource provider
 > 2) Telling Ironic to transform the picked node in some way
 >
 > Today we allow the use Capabilities for both.
 >
 > I am suggesting we move to using Traits only for (1), leaving (2) in
 > place for now, while we decide what to do (i.e. future of "deploy
 > template" concept).
 >
 > It feels like Ironic's plan to define the "deploy templates" in
Ironic
 > should replace the dependency on Glare for this use case, largely
 > because the definition of the deploy template (in my mind) is very
 > heavily related to inspector and driver properties, etc. Mark is
looking
 > at moving that forward at the moment.

That won't do anything about the flavor explosion problem, though,
right
John?


Does nova still plan to allow passing additional desired traits into the 
server create request?

I (we?) was kind of banking on that to solve the Baskin Robbins thing.


That's precisely what I've been looking into. From what I can tell, 
Ironic was planning on using these CUSTOM_DEPLOY_TEMPLATE_XXX traits in 
two ways:


1) To tell Nova what scheduling constraints the instance needed -- e.g. 
"hey Nova, make sure I land on a node that supports UEFI boot mode 
because my boot image relies on that".


2) As a convenient (because it would require no changes to Nova) way of 
passing instance pre-spawn configuration data to the Ironic virt driver 
-- e.g. pass the entire set of traits that are in the RequestSpec's 
flavor and image extra specs to Ironic before calling the Ironic node 
provision API.


#1 is fine IMHO, since it (mostly) represents a "capability" that the 
resource provider (the Ironic baremetal node) must support in order for 
the instance to successfully boot.


#2 is a problem, though, because it *doesn't* represent a capability. In 
fact, it can represent any and all sorts of key/value, JSON/dict or 
other information and this information is not intended to be passed to 
the placement/scheduler

Re: [openstack-dev] [Openstack-operators] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter

[Jay Pipes]

On 10/01/2018 04:36 AM, John Garbutt wrote:
On Fri, 28 Sep 2018 at 00:46, Jay Pipes <mailto:jaypi...@gmail.com>> wrote:


On 09/27/2018 06:23 PM, Matt Riedemann wrote:
 > On 9/27/2018 3:02 PM, Jay Pipes wrote:
 >> A great example of this would be the proposed "deploy template"
from
 >> [2]. This is nothing more than abusing the placement traits API in
 >> order to allow passthrough of instance configuration data from the
 >> nova flavor extra spec directly into the nodes.instance_info
field in
 >> the Ironic database. It's a hack that is abusing the entire
concept of
 >> the placement traits concept, IMHO.
 >>
 >> We should have a way *in Nova* of allowing instance configuration
 >> key/value information to be passed through to the virt driver's
 >> spawn() method, much the same way we provide for user_data that
gets
 >> exposed after boot to the guest instance via configdrive or the
 >> metadata service API. What this deploy template thing is is just a
 >> hack to get around the fact that nova doesn't have a basic way of
 >> passing through some collated instance configuration key/value
 >> information, which is a darn shame and I'm really kind of
annoyed with
 >> myself for not noticing this sooner. :(
 >
 > We talked about this in Dublin through right? We said a good
thing to do
 > would be to have some kind of template/profile/config/whatever
stored
 > off in glare where schema could be registered on that thing, and
then
 > you pass a handle (ID reference) to that to nova when creating the
 > (baremetal) server, nova pulls it down from glare and hands it
off to
 > the virt driver. It's just that no one is doing that work.

No, nobody is doing that work.

I will if need be if it means not hacking the placement API to serve
this purpose (for which it wasn't intended).


Going back to the point Mark Goddard made, there are two things here:

1) Picking the correct resource provider
2) Telling Ironic to transform the picked node in some way

Today we allow the use Capabilities for both.

I am suggesting we move to using Traits only for (1), leaving (2) in 
place for now, while we decide what to do (i.e. future of "deploy 
template" concept).


It feels like Ironic's plan to define the "deploy templates" in Ironic 
should replace the dependency on Glare for this use case, largely 
because the definition of the deploy template (in my mind) is very 
heavily related to inspector and driver properties, etc. Mark is looking 
at moving that forward at the moment.


That won't do anything about the flavor explosion problem, though, right 
John?


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [placement] The "intended purpose" of traits

[Jay Pipes]

On 09/28/2018 04:36 PM, Eric Fried wrote:

So here it is. Two of the top influencers in placement, one saying we
shouldn't overload traits, the other saying we shouldn't add a primitive
that would obviate the need for that. Historically, this kind of
disagreement seems to result in an impasse: neither thing happens and
those who would benefit are forced to find a workaround or punt.
Frankly, I don't particularly care which way we go; I just want to be
able to do the things.


I don't think that's a fair statement. You absolutely *do* care which 
way we go. You want to encode multiple bits of information into a trait 
string -- such as "PCI_ADDRESS_01_AB_23_CD" -- and leave it up to the 
caller to have to understand that this trait string has multiple bits of 
information encoded in it (the fact that it's a PCI device and that the 
PCI device is at 01_AB_23_CD).


You don't see a problem encoding these variants inside a string. Chris 
doesn't either.


I *do* see a problem with it, based on my experience in Nova where this 
kind of thing leads to ugly, unmaintainable, and incomprehensible code 
as I have pointed to in previous responses.


Furthermore, your point isn't that "you just want to be able to do the 
things". Your point (and the point of others, from Cyborg and Ironic) is 
that you want to be able to use placement to pass various bits of 
information to an instance, and placement wasn't designed for that 
purpose. Nova was.


So, instead of working out a solution with the Nova team for passing 
configuration data about an instance, the proposed solution is instead 
to hack/encode multiple bits of information into a trait string. This 
proposed solution is seen as a way around having to work out a more 
appropriate solution that has Nova pass that configuration data (as is 
appropriate, since nova is the project that manages instances) to the 
virt driver or generic device manager (i.e. Cyborg) before the instance 
spawns.


I'm working on a spec that will describe a way for the user to instruct 
Nova to pass configuration data to the virt driver (or device manager) 
before instance spawn. This will have nothing to do with placement or 
traits, since this configuration data is not modeling scheduling and 
placement decisions.


I hope to have that spec done by Monday so we can discuss on the spec.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [placement] The "intended purpose" of traits

[Jay Pipes]

On 09/28/2018 04:42 PM, Eric Fried wrote:

On 09/28/2018 09:41 AM, Balázs Gibizer wrote:

On Fri, Sep 28, 2018 at 3:25 PM, Eric Fried  wrote:

It's time somebody said this.

Every time we turn a corner or look under a rug, we find another use
case for provider traits in placement. But every time we have to have
the argument about whether that use case satisfies the original
"intended purpose" of traits.

That's only reason I've ever been able to glean: that it (whatever "it"
is) wasn't what the architects had in mind when they came up with the
idea of traits. We're not even talking about anything that would require
changes to the placement API. Just, "Oh, that's not a *capability* -
shut it down."

Bubble wrap was originally intended as a textured wallpaper and a
greenhouse insulator. Can we accept the fact that traits have (many,
many) uses beyond marking capabilities, and quit with the arbitrary
restrictions?


How far are we willing to go? Does an arbitrary (key: value) pair
encoded in a trait name like key_`str(value)` (e.g. CURRENT_TEMPERATURE:
85 encoded as CUSTOM_TEMPERATURE_85) something we would be OK to see in
placement?


Great question. Perhaps TEMPERATURE_DANGEROUSLY_HIGH is okay, but
TEMPERATURE_ is not.


That's correct, because you're encoding >1 piece of information into the 
single string (the fact that it's a temperature *and* the value of that 
temperature are the two pieces of information encoded into the single 
string).


Now that there's multiple pieces of information encoded in the string 
the reader of the trait string needs to know how to decode those bits of 
information, which is exactly what we're trying to avoid doing (because 
we can see from the ComputeCapabilitiesFilter, the extra_specs mess, and 
the giant hairball that is the NUMA and CPU pinning "metadata requests" 
how that turns out).



This thread isn't about setting these parameters; it's about getting
us to a point where we can discuss a question just like this one
without running up against: >
"That's a hard no, because you shouldn't encode key/value pairs in traits."

"Oh, why's that?"

"Because that's not what we intended when we created traits."

"But it would work, and the alternatives are way harder."

"-1"

"But..."

"-I


I believe I've articulated a number of times why traits should remain 
unary pieces of information, and not just said "because that's what we 
intended when we created traits".


I'm tough on this because I've seen the garbage code and unmaintainable 
mess that not having structurally sound data modeling concepts and 
information interpretation rules leads to in Nova and I don't want to 
encourage any more of it.


-jay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [placement] The "intended purpose" of traits

[Jay Pipes]

On 09/28/2018 09:25 AM, Eric Fried wrote:

It's time somebody said this.

Every time we turn a corner or look under a rug, we find another use
case for provider traits in placement. But every time we have to have
the argument about whether that use case satisfies the original
"intended purpose" of traits.

That's only reason I've ever been able to glean: that it (whatever "it"
is) wasn't what the architects had in mind when they came up with the
idea of traits.


Don't pussyfoot around things. It's me you're talking about, Eric. You 
could just ask me instead of passive-aggressively posting to the list 
like this.



We're not even talking about anything that would require changes to
the placement API. Just, "Oh, that's not a *capability* - shut it
down."
That's precisely the attitude that got the Nova scheduler into the 
unmaintainable and convoluted mess that it is now: "well, who cares if a 
concept was originally intended to describe X, it's just *easier* for us 
to re-use this random piece of data in ways it wasn't intended because 
that way we don't have to change anything about our docs or our API".


And *this* is the kind of stuff you end up with:

https://github.com/openstack/nova/blob/99bf62e42701397690fe2b4987ce4fd7879355b8/nova/scheduler/filters/compute_capabilities_filter.py#L35-L107

Which is a pile of unreadable, unintelligible garbage; nobody knows how 
it works, how it originally was intended to work, or how to really clean 
it up.



Bubble wrap was originally intended as a textured wallpaper and a
greenhouse insulator. Can we accept the fact that traits have (many,
many) uses beyond marking capabilities, and quit with the arbitrary
restrictions?


They aren't arbitrary. They are there for a reason: a trait is a boolean 
capability. It describes something that either a provider is capable of 
supporting or it isn't.


Conceptually, having boolean traits/capabilities is important because it 
allows the user to reason simply about how a provider meets the 
requested constraints for scheduling.


Currently, those constraints include the following:

* Does the provider have *capacity* for the requested resources?
* Does the provider have the required (or forbidden) *capabilities*?
* Does the provider belong to some group?

If we want to add further constraints to the placement allocation 
candidates request that ask things like:


* Does the provider have version 1.22.61821 of BIOS firmware from 
Marvell installed on it?
* Does the provider support an FPGA that has had an OVS program flashed 
to it in the last 20 days?
* Does the provider belong to physical network "corpnet" and also 
support creation of virtual NICs of type either "DIRECT" or "NORMAL"?


Then we should add a data model that allow providers to be decorated 
with key/value (or more complex than key/value) information where we can 
query for those kinds of constraints without needing to encode all sorts 
of non-binary bits of information into a capability string.


Propose such a thing and I'll gladly support it. But I won't support 
bastardizing the simple concept of a boolean capability just because we 
don't want to change the API or database schema.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter

[Jay Pipes]

On 09/27/2018 06:23 PM, Matt Riedemann wrote:

On 9/27/2018 3:02 PM, Jay Pipes wrote:
A great example of this would be the proposed "deploy template" from 
[2]. This is nothing more than abusing the placement traits API in 
order to allow passthrough of instance configuration data from the 
nova flavor extra spec directly into the nodes.instance_info field in 
the Ironic database. It's a hack that is abusing the entire concept of 
the placement traits concept, IMHO.


We should have a way *in Nova* of allowing instance configuration 
key/value information to be passed through to the virt driver's 
spawn() method, much the same way we provide for user_data that gets 
exposed after boot to the guest instance via configdrive or the 
metadata service API. What this deploy template thing is is just a 
hack to get around the fact that nova doesn't have a basic way of 
passing through some collated instance configuration key/value 
information, which is a darn shame and I'm really kind of annoyed with 
myself for not noticing this sooner. :(


We talked about this in Dublin through right? We said a good thing to do 
would be to have some kind of template/profile/config/whatever stored 
off in glare where schema could be registered on that thing, and then 
you pass a handle (ID reference) to that to nova when creating the 
(baremetal) server, nova pulls it down from glare and hands it off to 
the virt driver. It's just that no one is doing that work.


No, nobody is doing that work.

I will if need be if it means not hacking the placement API to serve 
this purpose (for which it wasn't intended).


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter

[Jay Pipes]

On 09/26/2018 05:48 PM, melanie witt wrote:

On Tue, 25 Sep 2018 12:08:03 -0500, Matt Riedemann wrote:

On 9/25/2018 8:36 AM, John Garbutt wrote:

 Another thing is about existing flavors configured for these
 capabilities-scoped specs. Are you saying during the deprecation 
we'd
 continue to use those even if the filter is disabled? In the 
review I

 had suggested that we add a pre-upgrade check which inspects the
 flavors
 and if any of these are found, we report a warning meaning those
 flavors
 need to be updated to use traits rather than capabilities. Would
 that be
 reasonable?


I like the idea of a warning, but there are features that have not yet
moved to traits:
https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html 



There is a more general plan that will help, but its not quite ready 
yet:

https://review.openstack.org/#/c/504952/

As such, I think we can't get pull the plug on flavors including
capabilities and passing them to Ironic, but (after a cycle of
deprecation) I think we can now stop pushing capabilities from Ironic
into Nova and using them for placement.


Forgive my ignorance, but if traits are not on par with capabilities,
why are we deprecating the capabilities filter?


I would like to know the answer to this as well.


In short, traits were never designed to be key/value pairs. They are 
simple strings indicating boolean capabilities.


Ironic "capabilities" are key/value metadata pairs. *Some* of those 
Ironic "capabilities" are possible to create as boolean traits.


For example, you can change the boot_mode=uefi and boot_mode=bios Ironic 
capabilities to be a trait called CUSTOM_BOOT_MODE_UEFI or 
CUSTOM_BOOT_MODE_BIOS [1].


Other Ironic "capabilities" are not, in fact, capabilities at all. 
Instead, they are just random key/value pairs that are not boolean in 
nature nor do they represent a capability of the baremetal hardware.


A great example of this would be the proposed "deploy template" from 
[2]. This is nothing more than abusing the placement traits API in order 
to allow passthrough of instance configuration data from the nova flavor 
extra spec directly into the nodes.instance_info field in the Ironic 
database. It's a hack that is abusing the entire concept of the 
placement traits concept, IMHO.


We should have a way *in Nova* of allowing instance configuration 
key/value information to be passed through to the virt driver's spawn() 
method, much the same way we provide for user_data that gets exposed 
after boot to the guest instance via configdrive or the metadata service 
API. What this deploy template thing is is just a hack to get around the 
fact that nova doesn't have a basic way of passing through some collated 
instance configuration key/value information, which is a darn shame and 
I'm really kind of annoyed with myself for not noticing this sooner. :(


-jay

[1] As I've asked for in the past, it would be great to have Ironic 
contributors push patches to the os-traits library for standardized 
baremetal capabilities like boot modes. Please do consider contributing 
there.


[2] 
https://review.openstack.org/#/c/504952/16/specs/approved/deploy-templates.rst


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] Stein PTG summary

[Jay Pipes]

On 09/27/2018 11:15 AM, Eric Fried wrote:

On 09/27/2018 07:37 AM, Matt Riedemann wrote:

On 9/27/2018 5:23 AM, Sylvain Bauza wrote:



On Thu, Sep 27, 2018 at 2:46 AM Matt Riedemann mailto:mriede...@gmail.com>> wrote:

     On 9/26/2018 5:30 PM, Sylvain Bauza wrote:
  > So, during this day, we also discussed about NUMA affinity and we
     said
  > that we could possibly use nested resource providers for NUMA
     cells in
  > Stein, but given we don't have yet a specific Placement API
     query, NUMA
  > affinity should still be using the NUMATopologyFilter.
  > That said, when looking about how to use this filter for vGPUs,
     it looks
  > to me that I'd need to provide a new version for the NUMACell
     object and
  > modify the virt.hardware module. Are we also accepting this
     (given it's
  > a temporary question), or should we need to wait for the
     Placement API
  > support ?
  >
  > Folks, what are you thoughts ?

     I'm pretty sure we've said several times already that modeling
NUMA in
     Placement is not something for which we're holding up the extraction.


It's not an extraction question. Just about knowing whether the Nova
folks would accept us to modify some o.vo object and module just for a
temporary time until Placement API has some new query parameter.
Whether Placement is extracted or not isn't really the problem, it's
more about the time it will take for this query parameter ("numbered
request groups to be in the same subtree") to be implemented in the
Placement API.
The real problem we have with vGPUs is that if we don't have NUMA
affinity, the performance would be around 10% less for vGPUs (if the
pGPU isn't on the same NUMA cell than the pCPU). Not sure large
operators would accept that :(

-Sylvain


I don't know how close we are to having whatever we need for modeling
NUMA in the placement API, but I'll go out on a limb and assume we're
not close.


True story. We've been talking about ways to do this since (at least)
the Queens PTG, but haven't even landed on a decent design, let alone
talked about getting it specced, prioritized, and implemented. Since
full NRP support was going to be a prerequisite in any case, and our
Stein plate is full, Train is the earliest we could reasonably expect to
get the placement support going, let alone the nova side. So yeah...


Given that, if we have to do something within nova for NUMA
affinity for vGPUs for the NUMATopologyFilter, then I'd be OK with that
since it's short term like you said (although our "short term"
workarounds tend to last for many releases). Anyone that cares about
NUMA today already has to enable the scheduler filter anyway.



+1 to this ^


Or, I don't know, maybe don't do anything and deal with the (maybe) 10% 
performance impact from the cross-NUMA main memory <-> CPU hit for 
post-processing of already parallel-processed GPU data.


In other words, like I've mentioned in numerous specs and in person, I 
really don't think this is a major problem and is mostly something we're 
making a big deal about for no real reason.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][glance][ironic][keystone][neutron][nova][edge] PTG summary on edge discussions

[Jay Pipes]

On 09/26/2018 05:10 AM, Colleen Murphy wrote:

Thanks for the summary, Ildiko. I have some questions inline.

On Tue, Sep 25, 2018, at 11:23 AM, Ildiko Vancsa wrote:





We agreed to prefer federation for Keystone and came up with two work
items to cover missing functionality:

* Keystone to trust a token from an ID Provider master and when the auth
method is called, perform an idempotent creation of the user, project
and role assignments according to the assertions made in the token


This sounds like it is based on the customizations done at Oath, which to my 
recollection did not use the actual federation implementation in keystone due 
to its reliance on Athenz (I think?) as an identity manager. Something similar 
can be accomplished in standard keystone with the mapping API in keystone which 
can cause dynamic generation of a shadow user, project and role assignments.


* Keystone should support the creation of users and projects with
predictable UUIDs (eg.: hash of the name of the users and projects).
This greatly simplifies Image federation and telemetry gathering


I was in and out of the room and don't recall this discussion exactly. We have 
historically pushed back hard against allowing setting a project ID via the 
API, though I can see predictable-but-not-settable as less problematic. One of 
the use cases from the past was being able to use the same token in different 
regions, which is problematic from a security perspective. Is that that idea 
here? Or could someone provide more details on why this is needed?


Hi Colleen,

I wasn't in the room for this conversation either, but I believe the 
"use case" wanted here is mostly a convenience one. If the edge 
deployment is composed of hundreds of small Keystone installations and 
you have a user (e.g. an NFV MANO user) which should have visibility 
across all of those Keystone installations, it becomes a hassle to need 
to remember (or in the case of headless users, store some lookup of) all 
the different tenant and user UUIDs for what is essentially the same 
user across all of those Keystone installations.


I'd argue that as long as it's possible to create a Keystone tenant and 
user with a unique name within a deployment, and as long as it's 
possible to authenticate using the tenant and user *name* (i.e. not the 
UUID), then this isn't too big of a problem. However, I do know that a 
bunch of scripts and external tools rely on setting the tenant and/or 
user via the UUID values and not the names, so that might be where this 
feature request is coming from.


Hope that makes sense?

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [openstack-dev] [penstack-dev]Discussion about the future of OpenStack in China

[Jay Pipes]

Fred,

I had a hard time understanding the articles. I'm not sure if you used 
Google Translate to do the translation from Chinese to English, but I 
personally found both of them difficult to follow.


There were a couple points that I did manage to decipher, though. One 
thing that both articles seemed to say was that OpenStack doesn't meet 
public (AWS-ish) cloud use cases and OpenStack doesn't compare favorably 
to VMWare either.


Is there a large contingent of Chinese OpenStack users that expect 
OpenStack to be a free (as in beer) version of VMware technology?


What are the 3 most important features that Chinese OpenStack users 
would like to see included in OpenStack projects?


Thanks,
-jay

On 09/24/2018 11:10 AM, Fred Li wrote:

Hi folks,

Recently there are several blogs which discussed about the future of 
OpenStack. If I was not wrong, the first one is 
"OpenStack-8-year-itch"[1], and you can find its English version 
attached. Thanks to google translation. The second one is 
"5-years-my-opinion-on-OpenStack" [2] with English version attached as 
well. Please translate the 3 to 6 and read them if you are interested.


I don't want to judge anything here. I just want to share as they are 
quite hot discussion and I think it is valuable for the whole community, 
not part of community to know.


[1] https://mp.weixin.qq.com/s/GM5cMOl0q3hb_6_eEiixzA
[2] https://mp.weixin.qq.com/s/qZkE4o_BHBPlbIjekjDRKw
[3] https://mp.weixin.qq.com/s/svX4z3JM5ArQ57A1jFoyLw
[4] https://mp.weixin.qq.com/s/Nyb0OxI2Z7LxDpofTTyWOg
[5] https://mp.weixin.qq.com/s/5GV4i8kyedHSbCxCO1VBRw
[6] https://mp.weixin.qq.com/s/yeBcMogumXKGQ0KyKrgbqA
--
Regards
Fred Li (李永乐)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [penstack-dev]Discussion about the future of OpenStack in China

[Jay Pipes]

Fred,

I had a hard time understanding the articles. I'm not sure if you used 
Google Translate to do the translation from Chinese to English, but I 
personally found both of them difficult to follow.


There were a couple points that I did manage to decipher, though. One 
thing that both articles seemed to say was that OpenStack doesn't meet 
public (AWS-ish) cloud use cases and OpenStack doesn't compare favorably 
to VMWare either.


Is there a large contingent of Chinese OpenStack users that expect 
OpenStack to be a free (as in beer) version of VMware technology?


What are the 3 most important features that Chinese OpenStack users 
would like to see included in OpenStack projects?


Thanks,
-jay

On 09/24/2018 11:10 AM, Fred Li wrote:

Hi folks,

Recently there are several blogs which discussed about the future of 
OpenStack. If I was not wrong, the first one is 
"OpenStack-8-year-itch"[1], and you can find its English version 
attached. Thanks to google translation. The second one is 
"5-years-my-opinion-on-OpenStack" [2] with English version attached as 
well. Please translate the 3 to 6 and read them if you are interested.


I don't want to judge anything here. I just want to share as they are 
quite hot discussion and I think it is valuable for the whole community, 
not part of community to know.


[1] https://mp.weixin.qq.com/s/GM5cMOl0q3hb_6_eEiixzA
[2] https://mp.weixin.qq.com/s/qZkE4o_BHBPlbIjekjDRKw
[3] https://mp.weixin.qq.com/s/svX4z3JM5ArQ57A1jFoyLw
[4] https://mp.weixin.qq.com/s/Nyb0OxI2Z7LxDpofTTyWOg
[5] https://mp.weixin.qq.com/s/5GV4i8kyedHSbCxCO1VBRw
[6] https://mp.weixin.qq.com/s/yeBcMogumXKGQ0KyKrgbqA
--
Regards
Fred Li (李永乐)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][edge] Notes from the PTG

[Jay Pipes]

On 09/19/2018 11:03 AM, Jim Rollenhagen wrote:
On Wed, Sep 19, 2018 at 8:49 AM, Jim Rollenhagen > wrote:


Tuesday: edge


Since cdent asked in IRC, when we talk about edge and far edge, we 
defined these roughly like this:

https://usercontent.irccloud-cdn.com/file/NunkkS2y/edge_architecture1.JPG


Far out, man.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Nominating Tetsuro Nakamura for placement-core

[Jay Pipes]

On 09/19/2018 11:25 AM, Chris Dent wrote:

I'd like to nominate Tetsuro Nakamura for membership in the
placement-core team. Throughout placement's development Tetsuro has
provided quality reviews; done the hard work of creating rigorous
functional tests, making them fail, and fixing them; and implemented
some of the complex functionality required at the persistence layer.
He's aware of and respects the overarching goals of placement and has
demonstrated pragmatism when balancing those goals against the
requirements of nova, blazar and other projects.

Please follow up with a +1/-1 to express your preference. No need to
be an existing placement core, everyone with an interest is welcome.


+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [nova][cinder] Migrate instances between regions or between clusters?

[Jay Pipes]

On 09/17/2018 09:39 AM, Peter Penchev wrote:

Hi,

So here's a possibly stupid question - or rather, a series of such :)
Let's say a company has two (or five, or a hundred) datacenters in
geographically different locations and wants to deploy OpenStack in both.
What would be a deployment scenario that would allow relatively easy
migration (cold, not live) of instances from one datacenter to another?

My understanding is that for servers located far away from one another
regions would be a better metaphor than availability zones, if only
because it would be faster for the various storage, compute, etc.
services to communicate with each other for the common case of doing
actions within the same datacenter.  Is this understanding wrong - is it
considered all right for groups of servers located in far away places to
be treated as different availability zones in the same cluster?

If the groups of servers are put in different regions, though, this
brings me to the real question: how can an instance be migrated across
regions?  Note that the instance will almost certainly have some
shared-storage volume attached, and assume (not quite the common case,
but still) that the underlying shared storage technology can be taught
about another storage cluster in another location and can transfer
volumes and snapshots to remote clusters.  From what I've found, there
are three basic ways:

- do it pretty much by hand: create snapshots of the volumes used in
   the underlying storage system, transfer them to the other storage
   cluster, then tell the Cinder volume driver to manage them, and spawn
   an instance with the newly-managed newly-transferred volumes


Yes, this is a perfectly reasonable solution. In fact, when I was at 
AT, this was basically how we allowed tenants to spin up instances in 
multiple regions: snapshot the instance, it gets stored in the Swift 
storage for the region, tenant starts the instance in a different 
region, and Nova pulls the image from the Swift storage in the other 
region. It's slow the first time it's launched in the new region, of 
course, since the bits need to be pulled from the other region's Swift 
storage, but after that, local image caching speeds things up quite a bit.


This isn't migration, though. Namely, the tenant doesn't keep their 
instance ID, their instance's IP addresses, or anything like that.


I've heard some users care about that stuff, unfortunately, which is why 
we have shelve [offload]. There's absolutely no way to perform a 
cross-region migration that keeps the instance ID and instance IP addresses.



- use Cinder to backup the volumes from one region, then restore them to
   the other; if this is combined with a storage-specific Cinder backup
   driver that knows that "backing up" is "creating a snapshot" and
   "restoring to the other region" is "transferring that snapshot to the
   remote storage cluster", it seems to be the easiest way forward (once
   the Cinder backup driver has been written)


Still won't have the same instance ID and IP address, which is what 
certain users tend to complain about needing with move operations.



- use Nova's "server image create" command, transfer the resulting
   Glance image somehow (possibly by downloading it from the Glance
   storage in one region and simulateneously uploading it to the Glance
   instance in the other), then spawn an instance off that image


Still won't have the same instance ID and IP address :)

Best,
-jay


The "server image create" approach seems to be the simplest one,
although it is a bit hard to imagine how it would work without
transferring data unnecessarily (the online articles I've seen
advocating it seem to imply that a Nova instance in a region cannot be
spawned off a Glance image in another region, so there will need to be
at least one set of "download the image and upload it to the other
side", even if the volume-to-image and image-to-volume transfers are
instantaneous, e.g. using glance-cinderclient).  However, when I tried
it with a Nova instance backed by a StorPool volume (no ephemeral image
at all), the Glance image was zero bytes in length and only its metadata
contained some information about a volume snapshot created at that
point, so this seems once again to go back to options 1 and 2 for the
different ways to transfer a Cinder volume or snapshot to the other
region.  Or have I missed something, is there a way to get the "server
image create / image download / image create" route to handle volumes
attached to the instance?

So... have I missed something else, too, or are these the options for
transferring a Nova instance between two distant locations?

Thanks for reading this far, and thanks in advance for your help!

Best regards,
Peter



___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : 

Re: [openstack-dev] [tc] notes from stein ptg meetings of the technical committee

[Jay Pipes]

On 09/17/2018 04:50 PM, Doug Hellmann wrote:

Excerpts from Zane Bitter's message of 2018-09-17 16:12:30 -0400:

On 17/09/18 3:06 PM, Jay Pipes wrote:

On 09/17/2018 01:31 PM, Doug Hellmann wrote:

New Project Application Process
===

We wrapped up Sunday with a discussion of of our process for reviewing
new project applications. Zane and Chris in particular felt the
process for Adjutant was too painful for the project team because
there was no way to know how long discussions might go on and now
way for them to anticipate some of the issues they encountered.

We talked about formalizing a "coach" position to have someone from
the TC (or broader community) work with the team to prepare their
application with sufficient detail, seek feedback before voting
starts, etc.

We also talked about adding a time limit to the process, so that
teams at least have a rejection with feedback in a reasonable amount
of time.  Some of the less contentious discussions have averaged
from 1-4 months with a few more contentious cases taking as long
as 10 months. We did not settle on a time frame during the meeting,
so I expect this to be a topic for us to work out during the next
term.


So, to summarize... the TC is back to almost exactly the same point it
was at right before the Project Structure Reform happened in 2014-2015
(that whole Big Tent thing).


I wouldn't go that far. There are more easy decisions than there were
before the reform, but there still exist hard decisions. This is perhaps
inevitable.


The Project Structure Reform occurred because the TC could not make
decisions on whether projects should join OpenStack using objective
criteria, and due to this, new project applicants were forced to endure
long waits and subjective "graduation" reviews that could change from
one TC election cycle to the next.

The solution to this was to make an objective set of application
criteria and remove the TC from the "Supreme Court of OpenStack" role
that new applicants needed to come before and submit to the court's
judgment.

Many people complained that the Project Structure Reform was the TC
simply abrogating responsibility for being a judgmental body.

It seems that although we've now gotten rid of those objective criteria
for project inclusion and gone back to the TC being a subjective
judgmental body, that the TC is still not actually willing to pass
judgment one way or the other on new project applicants.


No criteria have been gotten rid of, but even after the Project
Structure Reform there existed criteria that were subjective. Here is a
thread discussing them during the last TC election:

http://lists.openstack.org/pipermail/openstack-dev/2018-April/129622.html

(I actually think that the perception that the criteria should be
entirely objective might be a contributor to the problem: when faced
with a subjective decision and no documentation or precedent to guide
them, TC members can be reluctant to choose.)


I think turning the decision about which projects fit the mission
into an entirely mechanical one would be a mistake. I would prefer
us to use, and trust, our judgement in cases where the answer needs
some thought.

I don't remember the history quite the way Jay does, either. I
remember us trying to base the decision more about what the team
was doing than how the code looked or whether the implementation
met anyone's idea of "good". That's why we retained the requirement
that the project "aligns with the OpenStack Mission".


Hmm. I very specifically remember the incubation and graduation review 
of Zaqar and the fact that over a couple cycles of TC elections, the 
"advice" given by the TC about specific technical implementation details 
changed, often arbitrarily, depending on who was on the TC and what day 
of the week it was. In fact, I pretty vividly remember this arbitrary 
nature of the architectural review being one of the primary reasons we 
switched to a purely objective set of criteria.


Also, for the record, I actually wasn't referring to Adjutant 
specifically when I referred in my original post to "only tangentially 
related to cloud computing". I was referring to my recollection of 
fairly recent history. I remember the seemingly endless debates about 
whether some applicants "fit" the OpenStack ecosystem or whether the 
applicant was merely trying to jump on a hype bandwagon for marketing 
purposes. Again, I wasn't specifically referring to Adjutant here, so I 
apologize if my words came across that way.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] When can/should we change additionalProperties=False in GET /servers(/detail)?

[Jay Pipes]

On 09/17/2018 03:28 PM, Matt Riedemann wrote:
This is a question from a change [1] which adds a new changes-before 
filter to the servers, os-instance-actions and os-migrations APIs.


For context, the os-instance-actions API stopped accepting undefined 
query parameters in 2.58 when we added paging support.


The os-migrations API stopped allowing undefined query parameters in 
2.59 when we added paging support.


The open question on the review is if we should change GET /servers and 
GET /servers/detail to stop allowing undefined query parameters starting 
with microversion 2.66 [2]. Apparently when we added support for 2.5 and 
2.26 for listing servers we didn't think about this. It means that a 
user can specify a query parameter, documented in the API reference, but 
with an older microversion and it will be silently ignored. That is 
backward compatible but confusing from an end user perspective since it 
would appear to them that the filter is not being applied, when it fact 
it would be if they used the correct microversion.


So do we want to start enforcing query parameters when listing servers 
to our defined list with microversion 2.66 or just continue to silently 
ignore them if used incorrectly?


Note that starting in Rocky, the Neutron API will start rejecting 
unknown query parameteres [3] if the filter-validation extension is 
enabled (since Neutron doesn't use microversions). So there is some 
precedent in OpenStack for starting to enforce query parameters.


[1] https://review.openstack.org/#/c/599276/
[2] 
https://review.openstack.org/#/c/599276/23/nova/api/openstack/compute/schemas/servers.py 

[3] 
https://docs.openstack.org/releasenotes/neutron/rocky.html#upgrade-notes


My vote would be just change additionalProperties to False in the 599276 
patch and be done with it.


Add a release note about the change, of course.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] notes from stein ptg meetings of the technical committee

[Jay Pipes]

On 09/17/2018 01:31 PM, Doug Hellmann wrote:

New Project Application Process
===

We wrapped up Sunday with a discussion of of our process for reviewing
new project applications. Zane and Chris in particular felt the
process for Adjutant was too painful for the project team because
there was no way to know how long discussions might go on and now
way for them to anticipate some of the issues they encountered.

We talked about formalizing a "coach" position to have someone from
the TC (or broader community) work with the team to prepare their
application with sufficient detail, seek feedback before voting
starts, etc.

We also talked about adding a time limit to the process, so that
teams at least have a rejection with feedback in a reasonable amount
of time.  Some of the less contentious discussions have averaged
from 1-4 months with a few more contentious cases taking as long
as 10 months. We did not settle on a time frame during the meeting,
so I expect this to be a topic for us to work out during the next
term.


So, to summarize... the TC is back to almost exactly the same point it 
was at right before the Project Structure Reform happened in 2014-2015 
(that whole Big Tent thing).


The Project Structure Reform occurred because the TC could not make 
decisions on whether projects should join OpenStack using objective 
criteria, and due to this, new project applicants were forced to endure 
long waits and subjective "graduation" reviews that could change from 
one TC election cycle to the next.


The solution to this was to make an objective set of application 
criteria and remove the TC from the "Supreme Court of OpenStack" role 
that new applicants needed to come before and submit to the court's 
judgment.


Many people complained that the Project Structure Reform was the TC 
simply abrogating responsibility for being a judgmental body.


It seems that although we've now gotten rid of those objective criteria 
for project inclusion and gone back to the TC being a subjective 
judgmental body, that the TC is still not actually willing to pass 
judgment one way or the other on new project applicants.


Is this because it is still remarkably unclear what OpenStack actually 
*is* (the whole mission/scope thing)?


Or is this because TC members simply don't want to be the ones to say 
"No" to good-meaning people that may have an idea that is only 
tangentially related to cloud computing?


Everything old is new again.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] how nova should behave when placement returns consumer generation conflict

[Jay Pipes]

Thanks Giblet,

Will review this afternoon.

Best,
-jay

On 09/17/2018 09:10 AM, Balázs Gibizer wrote:


Hi,

Reworked and rebased the series based on this thread. The series starts 
here https://review.openstack.org/#/c/591597


Cheers,
gibi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] About microversion setting to enable nested resource provider

[Jay Pipes]

On 09/16/2018 09:28 PM, Naichuan Sun wrote:

Hi, Sylvain,

In truth I’m worrying about the old root rp which include the vgpu 
inventory. There is no field in the inventory which can display which 
GPU/GPUG it belong to, right? Anyway,  will discuss it after you come back.


As Sylvain mentions below, you will need to have some mechanism in the 
XenAPI virt driver which creates child resource providers under the 
existing root provider (which is the compute node resource provider). 
You will need to have the virt driver persist the mapping between your 
internal physical GPU group name and the UUID of the resource provider 
record that the virt driver creates for that PGPU group.


So, for example, let's say you have two PGPU groups on the host. They 
are named PGPU_A and PGPU_B. The XenAPI virt driver will need to ask the 
ProviderTree object it receives in the update_provider_tree() virt 
driver method whether there is a resource provider named "PGPU_A" in the 
tree. If not, the virt driver needs to create a new child resource 
provider with the name "PGPU_A" with a parent provider pointing to the 
root compute node provider. The ProviderTree.new_child() method is used 
to create new child providers:


https://github.com/openstack/nova/blob/82270cc261f6c1d9d2cc386f1fb445dd66023f75/nova/compute/provider_tree.py#L411

Hope that makes sense,
-jay


Thank very much.

BR.

Naichuan Sun

*From:*Sylvain Bauza [mailto:sba...@redhat.com]
*Sent:* Friday, September 14, 2018 9:34 PM
*To:* OpenStack Development Mailing List (not for usage questions) 

*Subject:* Re: [openstack-dev] About microversion setting to enable 
nested resource provider


Le jeu. 13 sept. 2018 à 19:29, Naichuan Sun <mailto:naichuan@citrix.com>> a écrit :


Hi, Sylvain,

Thank you very much for the information. It is pity that I can’t
attend the meeting.

I have a concern about reshaper in multi-type vgpu support.

In the old vgpu support, we only have one vgpu inventory in root
resource provider, which means we only support one vgpu type. When
do reshape, placement will send allocations(which include just one
vgpu resource allocation information) to the driver, if the host
have more than one pgpu/pgpug(which support different vgpu type),
how do we know which pgpu/pgpug own the allocation information? Do
we need to communicate with hypervisor the confirm that?

The reshape will actually move the existing allocations for a VGPU 
resource class to the inventory for this class that is on the child 
resource provider now with the reshape.


Since we agreed on keeping consistent naming, there is no need to guess 
which is which. That said, you raise a point that was discussed during 
the PTG and we all agreed there was an upgrade impact as multiple vGPUs 
shouldn't be allowed until the reshape is done.


Accordingly, see my spec I reproposed for Stein which describes the 
upgrade impact https://review.openstack.org/#/c/602474/


Since I'm at the PTG, we have huge time difference between you and me, 
but we can discuss on that point next week when I'm back (my mornings 
match then your afternoons)


-Sylvain

Thank you very much.

BR.

Naichuan Sun

*From:*Sylvain Bauza [mailto:sba...@redhat.com
<mailto:sba...@redhat.com>]
*Sent:* Thursday, September 13, 2018 11:47 PM
*To:* OpenStack Development Mailing List (not for usage questions)
mailto:openstack-dev@lists.openstack.org>>
*Subject:* Re: [openstack-dev] About microversion setting to enable
nested resource provider

Hey Naichuan,

FWIW, we discussed on the missing pieces for nested resource
providers. See the (currently-in-use) etherpad
https://etherpad.openstack.org/p/nova-ptg-stein and lookup for
"closing the gap on nested resource providers" (L144 while I speak)

The fact that we are not able to schedule yet is a critical piece
that we said we're going to work on it as soon as we can.

-Sylvain

On Thu, Sep 13, 2018 at 9:14 AM, Eric Fried mailto:openst...@fried.cc>> wrote:

There's a patch series in progress for this:

https://review.openstack.org/#/q/topic:use-nested-allocation-candidates

It needs some TLC. I'm sure gibi and tetsuro would welcome some
help...

efried


On 09/13/2018 08:31 AM, Naichuan Sun wrote:
 > Thank you very much, Jay.
 > Is there somewhere I could set microversion(some configure
file?), Or just modify the source code to set it?
 >
 > BR.
 > Naichuan Sun
     >
 > -Original Message-
 > From: Jay Pipes [mailto:jaypi...@gmail.com
<mailto:jaypi...@gmail.com>]
 > Sent: Thursday, September 13, 2018 9:19 PM
 > To: Naichuan Sun mailto:naichuan@citrix.com>>; OpenStack Development Mailing
List (not 

Re: [openstack-dev] About microversion setting to enable nested resource provider

[Jay Pipes]

On 09/13/2018 06:39 AM, Naichuan Sun wrote:

Hi, guys,

Looks n-rp is disabled by default because microversion matches 1.29 : 
https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/handlers/allocation_candidate.py#L252


Anyone know how to set the microversion to enable n-rp in placement?


It is the client which must send the 1.29+ placement API microversion 
header to indicate to the placement API server that the client wants to 
receive nested provider information in the allocation candidates response.


Currently, nova-scheduler calls the scheduler reportclient's 
get_allocation_candidates() method:


https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/manager.py#L138

The scheduler reportclient's get_allocation_candidates() method 
currently passes the 1.25 placement API microversion header:


https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/client/report.py#L353

https://github.com/openstack/nova/blob/0ba34a818414823eda5e693dc2127a534410b5df/nova/scheduler/client/report.py#L53

In order to get the nested information returned in the allocation 
candidates response, that would need to be upped to 1.29.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg][cinder][placement] etherpad for this afternoon's meeting

[Jay Pipes]

Hi Jay, where is this discussion taking place?

On Tue, Sep 11, 2018, 11:10 AM Jay S Bryant  wrote:

> All,
>
> I have created an etherpad to take notes during our meeting this
> afternoon:
> https://etherpad.openstack.org/p/cinder-placement-denver-ptg-2018
>
> If you have information you want to get in there before the meeting I
> would appreciate you pre-populating the pad.
>
> Jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][placement][upgrade][qa] Some upgrade-specific news on extraction

[Jay Pipes]

On 09/07/2018 11:17 AM, Dan Smith wrote:

The other obvious thing is the database. The placement repo code as-is
today still has the check for whether or not it should use the
placement database but falls back to using the nova_api database
[5]. So technically you could point the extracted placement at the
same nova_api database and it should work. However, at some point
deployers will clearly need to copy the placement-related tables out
of the nova_api DB to a new placement DB and make sure the
'migrate_version' table is dropped so that placement DB schema
versions can reset to 1.


I think it's wrong to act like placement and nova-api schemas are the
same. One is a clone of the other at a point in time, and technically it
will work today. However the placement db sync tool won't do the right
thing, and I think we run the major risk of operators not fully grokking
what is going on here, seeing that pointing placement at nova-api
"works" and move on. Later, when we add the next placement db migration
(which could technically happen in stein), they will either screw their
nova-api schema, or mess up their versioning, or be unable to apply the
placement change.


With respect to grenade and making this work in our own upgrade CI
testing, we have I think two options (which might not be mutually
exclusive):

1. Make placement support using nova.conf if placement.conf isn't
found for Stein with lots of big warnings that it's going away in
T. Then Rocky nova.conf with the nova_api database configuration just
continues to work for placement in Stein. I don't think we then have
any grenade changes to make, at least in Stein for upgrading *from*
Rocky. Assuming fresh devstack installs in Stein use placement.conf
and a placement-specific database, then upgrades from Stein to T
should also be OK with respect to grenade, but likely punts the
cut-over issue for all other deployment projects (because we don't CI
with grenade doing Rocky->Stein->T, or FFU in other words).


As I have said above and in the review, I really think this is the wrong
approach. At the current point of time, the placement schema is a clone
of the nova-api schema, and technically they will work. At the first point
that placement evolves its schema, that will no longer be a workable
solution, unless we also evolve nova-api's database in lockstep.


2. If placement doesn't support nova.conf in Stein, then grenade will
require an (exceptional) [6] from-rocky upgrade script which will (a)
write out placement.conf fresh and (b) run a DB migration script,
likely housed in the placement repo, to create the placement database
and copy the placement-specific tables out of the nova_api
database. Any script like this is likely needed regardless of what we
do in grenade because deployers will need to eventually do this once
placement would drop support for using nova.conf (if we went with
option 1).


Yep, and I'm asserting that we should write that script, make grenade do
that step, and confirm that it works. I think operators should do that
step during the stein upgrade because that's where the fork/split of
history and schema is happening. I'll volunteer to do the grenade side
at least.

Maybe it would help to call out specifically that, IMHO, this can not
and should not follow the typical config deprecation process. It's not a
simple case of just making sure we "find" the nova-api database in the
various configs. The problem is that _after_ the split, they are _not_
the same thing and should not be considered as the same. Thus, I think
to avoid major disaster and major time sink for operators later, we need
to impose the minor effort now to make sure that they don't take the
process of deploying a new service lightly.

Jay's original relatively small concern was that deploying a new
placement service and failing to properly configure it would result in a
placement running with the default, empty, sqlite database. That's a
valid concern, and I think all we need to do is make sure we fail in
that case, explaining the situation.

We just had a hangout on the topic and I think we've come around to the
consensus that just removing the default-to-empty-sqlite behavior is the
right thing to do. Placement won't magically find nova.conf if it exists
and jump into its database, and it also won't do the silly thing of
starting up with an empty database if the very important config step is
missed in the process of deploying placement itself. Operators will have
to deploy the new package and do the database surgery (which we will
provide instructions and a script for) as part of that process, but
there's really no other sane alternative without changing the current
agreed-to plan regarding the split.

Is everyone okay with the above summary of the outcome?


Yes from my perspective.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/05/2018 11:48 AM, Jeremy Stanley wrote:

On 2018-09-05 17:01:49 +0200 (+0200), Thomas Goirand wrote:
[...]

In a distro, no 2 package can hold the same file. That's
forbidden. This has nothing to do if someone has to "import
placemement" or not.

Just saying this, but *not* that we should rename (I didn't spot
any conflict yet and I understand the pain it would induce). This
command returns nothing:

apt-file search placement | grep python3/dist-packages/placement


Well, also since the Placement maintainers have expressed that they
aren't interested in making Python API contracts for it to be usable
as an importable library, there's probably no need to install its
modules into the global Python search path anyway. They could just
go into a private module path on the filesystem instead as long as
the placement service/entrypoint wrapper knows where to find them,
right?


Yep.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/04/2018 01:17 PM, Balázs Gibizer wrote:

On Tue, Sep 4, 2018 at 7:01 PM, Jay Pipes  wrote:

On 09/04/2018 12:59 PM, Balázs Gibizer wrote:

On Tue, Sep 4, 2018 at 6:25 PM, Jay Pipes  wrote:

On 09/04/2018 12:17 PM, Doug Hellmann wrote:

Excerpts from Jay Pipes's message of 2018-09-04 12:08:41 -0400:

On 09/04/2018 11:44 AM, Doug Hellmann wrote:

Excerpts from Chris Dent's message of 2018-09-04 15:32:12 +0100:

On Tue, 4 Sep 2018, Jay Pipes wrote:

Is there a reason we couldn't have 
openstack-placement be the package name?


I would hope we'd be able to do that, and probably should do that.
'openstack-placement' seems a find pypi package name for a think
from which you do 'import placement' to do some openstack stuff,
yeah?


That's still a pretty generic name for the top-level 
import, but I think
the only real risk is that the placement service couldn't be 
installed
at the same time as another package owned by someone 
else that used that

top-level name. I'm not sure how much of a risk that really is.


You mean if there was another Python package that used the package 
name

"placement"?

The alternative would be to make the top-level package something like
os_placement instead?


Either one works for me. Though I'm pretty sure that it isn't 
necessary. The reason it isn't necessary is because the stuff 
in the top-level placement package isn't meant to be imported by 
anything at all. It's the placement server code.


What about placement direct and the effort to allow cinder to 
import placement instead of running it as a separate service?


I don't know what placement direct is. Placement wasn't designed to be 
imported as a module. It was designed to be a (micro-)service with a 
REST API for interfacing.


In Vancouver we talked about allowing cinder to import placement as a 
library. See https://etherpad.openstack.org/p/YVR-cinder-placement L47


I wasn't in YVR, which explains why I's never heard of it. There's a 
number of misconceptions in the above document about the placement 
service that don't seem to have been addressed. I'm wondering if its 
worth revisiting the topic in Denver with the Cinder team or whether the 
Cinder team isn't interested in working with the placement service?


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/04/2018 12:59 PM, Balázs Gibizer wrote:

On Tue, Sep 4, 2018 at 6:25 PM, Jay Pipes  wrote:

On 09/04/2018 12:17 PM, Doug Hellmann wrote:

Excerpts from Jay Pipes's message of 2018-09-04 12:08:41 -0400:

On 09/04/2018 11:44 AM, Doug Hellmann wrote:

Excerpts from Chris Dent's message of 2018-09-04 15:32:12 +0100:

On Tue, 4 Sep 2018, Jay Pipes wrote:

Is there a reason we couldn't have openstack-placement be the 
package name?


I would hope we'd be able to do that, and probably should do that.
'openstack-placement' seems a find pypi package name for a think
from which you do 'import placement' to do some openstack stuff,
yeah?


That's still a pretty generic name for the top-level import, but I 
think

the only real risk is that the placement service couldn't be installed
at the same time as another package owned by someone else that used 
that

top-level name. I'm not sure how much of a risk that really is.


You mean if there was another Python package that used the package name
"placement"?

The alternative would be to make the top-level package something like
os_placement instead?


Either one works for me. Though I'm pretty sure that it isn't 
necessary. The reason it isn't necessary is because the stuff in the 
top-level placement package isn't meant to be imported by anything at 
all. It's the placement server code.


What about placement direct and the effort to allow cinder to import 
placement instead of running it as a separate service?


I don't know what placement direct is. Placement wasn't designed to be 
imported as a module. It was designed to be a (micro-)service with a 
REST API for interfacing.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/04/2018 12:17 PM, Doug Hellmann wrote:

Excerpts from Jay Pipes's message of 2018-09-04 12:08:41 -0400:

On 09/04/2018 11:44 AM, Doug Hellmann wrote:

Excerpts from Chris Dent's message of 2018-09-04 15:32:12 +0100:

On Tue, 4 Sep 2018, Jay Pipes wrote:


Is there a reason we couldn't have openstack-placement be the package name?


I would hope we'd be able to do that, and probably should do that.
'openstack-placement' seems a find pypi package name for a think
from which you do 'import placement' to do some openstack stuff,
yeah?


That's still a pretty generic name for the top-level import, but I think
the only real risk is that the placement service couldn't be installed
at the same time as another package owned by someone else that used that
top-level name. I'm not sure how much of a risk that really is.


You mean if there was another Python package that used the package name
"placement"?

The alternative would be to make the top-level package something like
os_placement instead?


Either one works for me. Though I'm pretty sure that it isn't necessary. 
The reason it isn't necessary is because the stuff in the top-level 
placement package isn't meant to be imported by anything at all. It's 
the placement server code.


Nothing is going to be adding openstack-placement into its 
requirements.txt file or doing:


 from placement import blah

If some part of the server repo is meant to be imported into some other 
system, say nova, then it will be pulled into a separate lib, ala 
ironiclib or neutronlib.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/04/2018 11:44 AM, Doug Hellmann wrote:

Excerpts from Chris Dent's message of 2018-09-04 15:32:12 +0100:

On Tue, 4 Sep 2018, Jay Pipes wrote:


Is there a reason we couldn't have openstack-placement be the package name?


I would hope we'd be able to do that, and probably should do that.
'openstack-placement' seems a find pypi package name for a think
from which you do 'import placement' to do some openstack stuff,
yeah?


That's still a pretty generic name for the top-level import, but I think
the only real risk is that the placement service couldn't be installed
at the same time as another package owned by someone else that used that
top-level name. I'm not sure how much of a risk that really is.


You mean if there was another Python package that used the package name 
"placement"?


The alternative would be to make the top-level package something like 
os_placement instead?


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] better name for placement

[Jay Pipes]

On 09/04/2018 09:37 AM, Jeremy Stanley wrote:

On 2018-09-04 09:32:20 +0100 (+0100), Chris Dent wrote:
[...]

it allowed for the possibility that there could be another project
which provided the same service-type. That hasn't really come to
pass

[...]

It still might make sense to attempt to look at this issue from
outside the limited scope of the OpenStack community. Is the
expectation that the project when packaged (on PyPI, in Linux
distributions and so on) will just be referred to as "placement"
with no further context?


I don't see any reason why the package name needs to be identical to the 
repo name.


Is there a reason we couldn't have openstack-placement be the package name?

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [nova] Nova-scheduler: when are filters applied?

[Jay Pipes]

On 09/03/2018 07:27 AM, Eugen Block wrote:

Hi,

To echo what cfriesen said, if you set your allocation ratio to 1.0, 
the system will not overcommit memory. Shut down instances consume 
memory from an inventory management perspective. If you don't want any 
danger of an instance causing an OOM, you must set you 
ram_allocation_ratio to 1.0.


let's forget about the scheduler, I'll try to make my question a bit 
clearer.


Let's say I have a ratio of 1.0 on my hypervisor, and let it have 24 GB 
of RAM available, ignoring the OS for a moment. Now I launch 6 
instances, each with a flavor requesting 4 GB of RAM, that would leave 
no space for further instances, right?
Then I shutdown two instances (freeing 8 GB RAM) and create a new one 
with 8 GB of RAM, the compute node is full again (assuming all instances 
actually consume all of their RAM).
Now I boot one of the shutdown instances again, the compute node would 
require additional 4 GB of RAM for that instance, and this would lead to 
OOM, isn't that correct? So a ratio of 1.0 would not prevent that from 
happening, would it?


I'm not entirely sure what you mean by "shut down an instance". Perhaps 
this is what is leading to confusion. I consider "shutting down an 
instance" to be stopping or suspending an instance.


As I mentioned below, shutdown instances consume memory from an 
inventory management perspective. If you stop or suspend an instance on 
your host, that instance is still consuming the same amount of memory in 
the placement service. You will *not* be able to launch a new instance 
on that same compute host *unless* your allocation ratio is >1.0.


Now, if by "shut down an instance", you actually mean "terminate an 
instance" or possibly "shelve and then offload an instance", then that 
is a different thing, and in both of *those* cases, resources are 
released on the compute host.


Best,
-jay


Zitat von Jay Pipes :


On 08/30/2018 10:54 AM, Eugen Block wrote:

Hi Jay,

You need to set your ram_allocation_ratio nova.CONF option to 1.0 if 
you're running into OOM issues. This will prevent overcommit of 
memory on your compute nodes.


I understand that, the overcommitment works quite well most of the time.

It just has been an issue twice when I booted an instance that had 
been shutdown a while ago. In the meantime there were new instances 
created on that hypervisor, and this old instance caused the OOM.


I would expect that with a ratio of 1.0 I would experience the same 
issue, wouldn't I? As far as I understand the scheduler only checks 
at instance creation, not when booting existing instances. Is that a 
correct assumption?


To echo what cfriesen said, if you set your allocation ratio to 1.0, 
the system will not overcommit memory. Shut down instances consume 
memory from an inventory management perspective. If you don't want any 
danger of an instance causing an OOM, you must set you 
ram_allocation_ratio to 1.0.


The scheduler doesn't really have anything to do with this.

Best,
-jay

___
Mailing list: 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to : openstack@lists.openstack.org
Unsubscribe : 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack




___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] Nominating Chris Dent for placement-core

[Jay Pipes]

On 08/31/2018 11:45 AM, Eric Fried wrote:

The openstack/placement project [1] and its core team [2] have been
established in gerrit.

I hereby nominate Chris Dent for membership in the placement-core team.
He has been instrumental in the design, implementation, and stewardship
of the placement API since its inception and has shown clear and
consistent leadership.

As we are effectively bootstrapping placement-core at this time, it
would seem appropriate to consider +1/-1 responses from heavy placement
contributors as well as existing cores (currently nova-core).

[1] https://review.openstack.org/#/admin/projects/openstack/placement
[2] https://review.openstack.org/#/admin/groups/1936,members


+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [chef] fog-openstack 0.2.0 breakage

[Jay Pipes]

Thanks for notifying about this, Samuel. Our most modern deployment is 
actually currently blocked on this and I'm glad to see a resolution.


Best,
-jay

On 08/31/2018 11:59 AM, Samuel Cassiba wrote:

Ohai!

fog-openstack 0.2.0 was recently released, which had less than optimal
effects on Chef OpenStack due to the client cookbook's lack of version
pinning on the gem.

The crucial change is that fog-openstack itself now determines
Identity API versions internally, in preparation for a versionless
Keystone endpoint. Chef OpenStack has carried code for Identity API
determination for years, to facilitate migrating from Identity v2.0 to
Identity v3. Unfortunately, those two methods became at odds with the
release of fog-openstack 0.2.

At the time of this writing, PR #421
(https://github.com/fog/fog-openstack/pull/421) has been merged, but
there is no new release on rubygems.org as of yet. That is likely to
happen Very Soon(tm).

On the home front, with the help of Roger Luethi and Christoph Albers,
we've introduced version constraints to the client cookbook to pin the
gem to 0.1.x. At present, we've merged constraints for master,
stable/queens and stable/pike.

The new release was primed to go into ChefDK 3.2 had it not been
brought up sooner. Thank you to everyone who gave a heads-up!

Best,

scas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] The problem of how to update resouce allocation ratio dynamically.

[Jay Pipes]

On 08/23/2018 11:01 PM, 余婷婷 wrote:

Hi:
    Sorry fo bothering everyone. Now I update my openstack to queen,and 
use the nova-placement-api to provider resource.
   When I use "/resource_providers/{uuid}/inventories/MEMORY_MB" to 
update memory_mb allocation_ratio, and it success.But after some 
minutes,it recove to old value automatically. Then I find it report the 
value from compute_node in nova-compute automatically. But the 
allocation_ratio of compute_node was came from the nova.conf.So that 
means，We can't update the allocation_ratio until we update the 
nova.conf? But I wish to update the allocation_ratio dynamically other 
to update the nova.conf. I don't known how  to update resouce allocation 
ratio dynamically.


We are attempting to determine what is going on with the allocation 
ratios being improperly set on the following bug:


https://bugs.launchpad.net/nova/+bug/1789654

Please bear with us as we try to fix it.

Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [nova] Nova-scheduler: when are filters applied?

[Jay Pipes]

On 08/30/2018 10:54 AM, Eugen Block wrote:

Hi Jay,

You need to set your ram_allocation_ratio nova.CONF option to 1.0 if 
you're running into OOM issues. This will prevent overcommit of memory 
on your compute nodes.


I understand that, the overcommitment works quite well most of the time.

It just has been an issue twice when I booted an instance that had been 
shutdown a while ago. In the meantime there were new instances created 
on that hypervisor, and this old instance caused the OOM.


I would expect that with a ratio of 1.0 I would experience the same 
issue, wouldn't I? As far as I understand the scheduler only checks at 
instance creation, not when booting existing instances. Is that a 
correct assumption?


To echo what cfriesen said, if you set your allocation ratio to 1.0, the 
system will not overcommit memory. Shut down instances consume memory 
from an inventory management perspective. If you don't want any danger 
of an instance causing an OOM, you must set you ram_allocation_ratio to 1.0.


The scheduler doesn't really have anything to do with this.

Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [nova]

[Jay Pipes]

On 08/30/2018 10:19 AM, Eugen Block wrote:

When does Nova apply its filters (Ram, CPU, etc.)?
Of course at instance creation and (live-)migration of existing 
instances. But what about existing instances that have been shutdown and 
in the meantime more instances on the same hypervisor have been launched?


When you start one of the pre-existing instances and even with RAM 
overcommitment you can end up with an OOM-Killer resulting in forceful 
shutdowns if you reach the limits. Is there something I've been missing 
or maybe a bad configuration of my scheduler filters? Or is it the 
admin's task to keep an eye on the load?


I'd appreciate any insights or pointers to something I've missed.


You need to set your ram_allocation_ratio nova.CONF option to 1.0 if 
you're running into OOM issues. This will prevent overcommit of memory 
on your compute nodes.


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [neutron][nova] Small bandwidth demo on the PTG

[Jay Pipes]

On 08/30/2018 04:55 AM, Balázs Gibizer wrote:

Hi,

Based on the Nova PTG planning etherpad [1] there is a need to talk 
about the current state of the bandwidth work [2][3]. Bence (rubasov) 
has already planned to show a small demo to Neutron folks about the 
current state of the implementation. So Bence and I are wondering about 
bringing that demo close to the nova - neutron cross project session. 
That session is currently planned to happen Thursday after lunch. So we 
are think about showing the demo right before that session starts. It 
would start 30 minutes before the nova - neutron cross project session.


Are Nova folks also interested in seeing such a demo?

If you are interested in seeing the demo please drop us a line or ping 
us in IRC so we know who should we wait for.


+1 from me. I'd be very interested in seeing it.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Jay Pipes]

On 08/29/2018 04:04 PM, Dan Smith wrote:

- The VMs to be migrated are not generally not expensive
configurations, just hardware lifecycles where boxes go out of
warranty or computer centre rack/cooling needs re-organising. For
CERN, this is a 6-12 month frequency of ~10,000 VMs per year (with a
~30% pet share)
- We make a cell from identical hardware at a single location, this
greatly simplifies working out hardware issues, provisioning and
management
- Some cases can be handled with the 'please delete and
re-create'. Many other cases need much user support/downtime (and
require significant effort or risk delaying retirements to get
agreement)


Yep, this is the "organizational use case" of cells I refer to. I assume
that if one aisle (cell) is being replaced, it makes sense to stand up
the new one as its own cell, migrate the pets from one to the other and
then decommission the old one. Being only an aisle away, it's reasonable
to think that *this* situation might not suffer from the complexity of
needing to worry about heavyweight migrate network and storage.


For this use case, why not just add the new hardware directly into the 
existing cell and migrate the workloads onto the new hardware, then 
disable the old hardware and retire it?


I mean, there might be a short period of time where the cell's DB and MQ 
would be congested due to lots of migration operations, but it seems a 
lot simpler to me than trying to do cross-cell migrations when cells 
have been designed pretty much from the beginning of cellsv2 to not talk 
to each other or allow any upcalls.


Thoughts?
-jay

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Jay Pipes]

On 08/29/2018 02:26 PM, Chris Friesen wrote:

On 08/29/2018 10:02 AM, Jay Pipes wrote:


Also, I'd love to hear from anyone in the real world who has successfully
migrated (live or otherwise) an instance that "owns" expensive hardware
(accelerators, SR-IOV PFs, GPUs or otherwise).


I thought cold migration of instances with such devices was supported 
upstream?


That's not what I asked. :)

-jay

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Jay Pipes]

On 08/29/2018 12:39 PM, Dan Smith wrote:

If we're going to discuss removing move operations from Nova, we should
do that in another thread. This one is about making existing operations
work :)


OK, understood. :)


The admin only "owns" the instance because we have no ability to
transfer ownership of the instance and a cell isn't a user-visible
thing. An external script that accomplishes this kind of orchestrated
move from one cell to another could easily update the ownership of
said instance in the DB.


So step 5 was "do surgery on the database"? :)


Yep. You'd be surprised how often that ends up being the case.

I'm currently sitting here looking at various integration tooling for 
doing just this kind of thing for our deployments of >150K baremetal 
compute nodes. The number of specific-to-an-environment variables that 
need to be considered and worked into the overall migration plan is 
breathtaking. And trying to do all of that inside of Nova just isn't 
feasible for the scale at which we run.


At least, that's my point of view. I won't drag this conversation out 
any further on tangents.


Best,
-jay

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Jay Pipes]

I respect your opinion but respectfully disagree that this is something 
we need to spend our time on. Comments inline.


On 08/29/2018 10:47 AM, Dan Smith wrote:

* Cells can shard across flavors (and hardware type) so operators
would like to move users off the old flavors/hardware (old cell) to
new flavors in a new cell.


So cell migrations are kind of the new release upgrade dance. Got it.


No, cell migrations are about moving instances between cells for
whatever reason. If you have small cells for organization, then it's
about not building arbitrary barriers between aisles. If you use it for
hardware refresh, then it might be related to long term lifecycle. I'm
not sure what any of this has to do with release upgrades or dancing.


A release upgrade dance involves coordination of multiple moving parts. 
It's about as similar to this scenario as I can imagine. And there's a 
reason release upgrades are not done entirely within Nova; clearly an 
external upgrade tool or script is needed to orchestrate the many steps 
and components involved in the upgrade process.


The similar dance for cross-cell migration is the coordination that 
needs to happen between Nova, Neutron and Cinder. It's called 
orchestration for a reason and is not what Nova is good at (as we've 
repeatedly seen)


 The thing that makes *this* particular scenario problematic is that 
cells aren't user-visible things. User-visible things could much more 
easily be orchestrated via external actors, as I still firmly believe 
this kind of thing should be done.



shelve was and continues to be a hack in order for users to keep an
IPv4 address while not consuming compute resources for some amount of
time. [1]


As we discussed in YVR most recently, it also may become an important
thing for operators and users where expensive accelerators are committed
to instances with part-time usage patterns.


I don't think that's a valid use case in respect to this scenario of 
cross-cell migration. If the target cell compute doesn't have the same 
expensive accelerators on them, nobody would want or permit a move to 
that target cell anyway.


Also, I'd love to hear from anyone in the real world who has 
successfully migrated (live or otherwise) an instance that "owns" 
expensive hardware (accelerators, SR-IOV PFs, GPUs or otherwise).


The patterns that I have seen are one of the following:

* Applications don't move. They are pets that stay on one or more VMs or 
baremetal nodes and they grow roots.


* Applications are designed to *utilize* the expensive hardware. They 
don't "own" the hardware itself.


In this latter case, the application is properly designed and stores its 
persistent data in a volume and doesn't keep state outside of the 
application volume. In these cases, the process of "migrating" an 
instance simply goes away. You just detach the application persistent 
volume, shut down the instance, start up a new one elsewhere (allowing 
the scheduler to select one that meets the resource constraints in the 
flavor/image), attach the volume again and off you go. No messing around 
with shelving, offloading, migrating, or any of that nonsense in Nova.


We should not pretend that what we're discussing here is anything other 
than hacking orchestration workarounds into Nova to handle 
poorly-designed applications that have grown roots on some hardware and 
think they "own" hardware resources in a Nova deployment.



It has also come up more than once in the realm of "but I need to
detach my root volume" scenarios. I love to hate on shelve as well,
but recently a few more legit (than merely keeping an IPv4 address)
use-cases have come out for it, and I don't think Matt is wrong that
cross-cell migration *might* be easier as a shelve operation under
the covers.


Matt may indeed be right, but I'm certainly allowed to express my 
opinion that I think shelve is a monstrosity that should be avoided at 
all costs and building additional orchestration functionality into Nova 
on top of an already-shaky foundation (shelve) isn't something I think 
is a long-term maintainable solution.



If cross-cell cold migration is similarly just about the user being
able to keep their instance's IPv4 address while allowing an admin to
move an instance's storage to another physical location, then my firm
belief is that this kind of activity needs to be coordinated
*externally to Nova*.


I'm not sure how you could make that jump, but no, I don't think that's
the case. In any sort of large cloud that uses cells to solve problems
of scale, I think it's quite likely to expect that your IPv4 address
physically can't be honored in the target cell, and/or requires some
less-than-ideal temporary tunneling for bridging the gap.


If that's the case, why are we discussing shelve at all? Just stop the 
instance, copy/migrate the volume data (if needed, again it completely 
depends on the deployment, network topology and block storage backend), 
to a new location (new 

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Jay Pipes]

Sorry for delayed response. Was on PTO when this came out. Comments 
inline...


On 08/22/2018 09:23 PM, Matt Riedemann wrote:

Hi everyone,

I have started an etherpad for cells topics at the Stein PTG [1]. The 
main issue in there right now is dealing with cross-cell cold migration 
in nova.


At a high level, I am going off these requirements:

* Cells can shard across flavors (and hardware type) so operators would 
like to move users off the old flavors/hardware (old cell) to new 
flavors in a new cell.


So cell migrations are kind of the new release upgrade dance. Got it.

* There is network isolation between compute hosts in different cells, 
so no ssh'ing the disk around like we do today. But the image service is 
global to all cells.


Based on this, for the initial support for cross-cell cold migration, I 
am proposing that we leverage something like shelve offload/unshelve 
masquerading as resize. We shelve offload from the source cell and 
unshelve in the target cell. This should work for both volume-backed and 
non-volume-backed servers (we use snapshots for shelved offloaded 
non-volume-backed servers).


shelve was and continues to be a hack in order for users to keep an IPv4 
address while not consuming compute resources for some amount of time. [1]


If cross-cell cold migration is similarly just about the user being able 
to keep their instance's IPv4 address while allowing an admin to move an 
instance's storage to another physical location, then my firm belief is 
that this kind of activity needs to be coordinated *externally to Nova*.


Each deployment is going to be different, and in all cases of cross-cell 
migration, the admins doing these move operations are going to need to 
understand various network, storage and failure domains that are 
particular to that deployment (and not something we have the ability to 
discover in any automated fashion).


Since we're not talking about live migration (thank all that is holy), I 
believe the safest and most effective way to perform such a cross-cell 
"migration" would be the following basic steps:


0. ensure that each compute node is associated with at least one nova 
host aggregate that is *only* in a single cell
1. shut down the instance (optionally snapshotting required local disk 
changes if the user is unfortunately using their root disk for 
application data)
2. "save" the instance's IP address by manually creating a port in 
Neutron and assigning the IP address manually to that port. this of 
course will be deployment-dependent since you will need to hope the 
saved IP address for the migrating instance is in a subnet range that is 
available in the target cell
3. migrate the volume manually. this will be entirely deployment and 
backend-dependent as smcginnis alluded to in a response to this thread
4. have the admin boot the instance in a host aggregate that is known to 
be in the target cell, passing --network port_id=$SAVED_PORT_WITH_IP and 
--volume $MIGRATED_VOLUME_UUID arguments as needed. the admin would need 
to do this because users don't know about host aggregates and, frankly, 
the user shouldn't know about host aggregates, cells, or any of this.


Best,
-jay

[1] ok, shelve also lets a user keep their instance ID. I don't care 
much about that.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [nova] [placement] XenServer CI failed frequently because of placement update

[Jay Pipes]

I think the immediate solution would be to just set cpu_allocation_ratio 
to 16.0 in the nova.CONF that your CI system is using.


Best,
-jay

On 08/29/2018 05:26 AM, Naichuan Sun wrote:

Hi, Eric and Jay,

The VCPU/Disk/RAM allocation ratio are set to 0.0 by default, and resource 
tracker would reset it to valid values in  
https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L199.
But it looks the value is set back to 0.0 by some function(I'm not sure who 
does it...), so xenserver CI is broken. Any suggestion about that?
Looks libvirt works well, do they set allocation ratio in the configure file?

Thank you very much.

BR.
Naichuan Sun

-Original Message-
From: Naichuan Sun
Sent: Wednesday, August 29, 2018 7:00 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: RE: [openstack-dev] [nova] [placement] XenServer CI failed frequently 
because of placement update

Thank you very much for the help, Bob, Jay and Eric.

Naichuan Sun

-Original Message-
From: Bob Ball [mailto:bob.b...@citrix.com]
Sent: Wednesday, August 29, 2018 12:22 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova] [placement] XenServer CI failed frequently 
because of placement update


Yeah, the nova.CONF cpu_allocation_ratio is being overridden to 0.0:


The default there is 0.0[1] - and the passing tempest-full from Zuul on 
https://review.openstack.org/#/c/590041/ has the same line when reading the 
config[2]:

We'll have a dig to see if we can figure out why it's not defaulting to 16 in 
the ComputeNode.

Thanks!

Bob

[1] https://git.openstack.org/cgit/openstack/nova/tree/nova/conf/compute.py#n386
[2] 
http://logs.openstack.org/41/590041/17/check/tempest-full/b3f9ddd/controller/logs/screen-n-cpu.txt.gz#_Aug_27_14_18_24_078058
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Open API 3.0 for OpenStack API

[Jay Pipes]

On 08/29/2018 02:36 AM, Edison Xiang wrote:
Based on Open API 3.0, it can bring lots of benefits for OpenStack 
Community and does not impact the current features the Community has.



3rd party developers can also do some self-defined development.


Hi Edison,

Would you mind expanding on what you are referring to with the above 
line about 3rd party developers doing self-defined development?


Thanks!
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [placement] XenServer CI failed frequently because of placement update

[Jay Pipes]

Yeah, the nova.CONF cpu_allocation_ratio is being overridden to 0.0:

Aug 27 07:43:02.179927 dsvm-devstack-citrix-lon-nodepool-1379254 
nova-compute[21125]: DEBUG oslo_service.service [None 
req-4bb236c4-54c3-42b7-aa4e-e5c8b1ece0c7 None None] cpu_allocation_ratio 
  = 0.0 {{(pid=21125) log_opt_values 
/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:3019}}


Best,
-jay

On 08/28/2018 10:01 AM, Bob Ball wrote:

We're not running with [1], however that did also fail the CI in the same way - 
see [2] for the full logs.

The first failing appeared to be around Aug 27 08:32:14:
Aug 27 08:32:14.502788 dsvm-devstack-citrix-lon-nodepool-1379254 
devstack@placement-api.service[13219]: DEBUG nova.api.openstack.placement.requestlog 
[req-94714f18-87f3-4ff5-9b17-f6e50131b3a9 req-fc47376d-cf04-4cd3-b69c-31ef4d5739a4 service 
placement] Starting request: 192.168.33.1 "GET 
/placement/allocation_candidates?limit=1000=MEMORY_MB%3A64%2CVCPU%3A1" 
{{(pid=13222) __call__ /opt/stack/new/nova/nova/api/openstack/placement/requestlog.py:38}}
Aug 27 08:32:14.583676 dsvm-devstack-citrix-lon-nodepool-1379254 
devstack@placement-api.service[13219]: DEBUG 
nova.api.openstack.placement.objects.resource_provider 
[req-94714f18-87f3-4ff5-9b17-f6e50131b3a9 
req-fc47376d-cf04-4cd3-b69c-31ef4d5739a4 service placement] found 0 providers 
with available 1 VCPU {{(pid=13222) _get_provider_ids_matching 
/opt/stack/new/nova/nova/api/openstack/placement/objects/resource_provider.py:2928}}

Just looking at Naichuan's output, I wonder if this is because allocation_ratio 
is registered as 0 in the inventory.

Bob

[2] 
http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/41/590041/17/check/dsvm-tempest-neutron-network/afadfe7/

-Original Message-
From: Eric Fried [mailto:openst...@fried.cc]
Sent: 28 August 2018 14:22
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova] [placement] XenServer CI failed frequently 
because of placement update

Naichuan-

Are you running with [1]? If you are, the placement logs (at debug
level) should be giving you some useful info. If you're not... perhaps you 
could pull that in :) Note that it refactors the _get_provider_ids_matching 
method completely, so it's possible your problem will magically go away when 
you do.

[1] https://review.openstack.org/#/c/590041/

On 08/28/2018 07:54 AM, Jay Pipes wrote:

On 08/28/2018 04:17 AM, Naichuan Sun wrote:

Hi, experts,

XenServer CI failed frequently with an error "No valid host was found.
" for more than a week. I think it is cause by placement update.


Hi Naichuan,

Can you give us a link to the logs a patchset's Citrix XenServer CI
that has failed? Also, a timestamp for the failure you refer to would
be useful so we can correlate across service logs.

Thanks,
-jay

__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [nova]Capacity discrepancy between command line and MySQL query

[Jay Pipes]

On 08/27/2018 09:40 AM, Risto Vaaraniemi wrote:

Hi,

I tried to migrate a guest to another host but it failed with a
message saying there's not enough capacity on the target host even
though the server should me nearly empty. The guest I'm trying to
move needs 4 cores, 4 GB of memory and 50 GB of disk. Each compute
node should have 20 cores, 128 GB RAM & 260 GB HD space.

When I check it with "openstack host show compute1" I see that there's
plenty of free resources. However, when I check it directly in MariaDB
nova_api or using Placement API calls I see different results i.e. not
enough cores & disk.

Is there a safe way to make the different registries / databases to
match? Can I just overwrite it using the Placement API?

I'm using Pike.

BR,
Risto

PS
I did make a few attempts to resize the guest that now runs on
compute1 but for some reason they failed and by default the resize
tries to restart the resized guest on a different host (compute1).
In the end I was able to do the resize on the same host (compute2).
I was wondering if the resize attempts messed up the compute1 resource
management.


Very likely, yes.

It's tough to say what exact sequence of resize and migrate commands 
have caused your inventory and allocation records in placement to become 
corrupted.


Have you tried restarting the nova-compute services on both compute 
nodes and seeing whether the placement service tries to adjust 
allocations upon restart?


Also, please check the logs on the nova-compute workers looking for any 
warnings or errors related to communication with placement.


Best,
-jay


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [nova] [placement] XenServer CI failed frequently because of placement update

[Jay Pipes]

On 08/28/2018 04:17 AM, Naichuan Sun wrote:

Hi, experts,

XenServer CI failed frequently with an error "No valid host was found. " 
for more than a week. I think it is cause by placement update.


Hi Naichuan,

Can you give us a link to the logs a patchset's Citrix XenServer CI that 
has failed? Also, a timestamp for the failure you refer to would be 
useful so we can correlate across service logs.


Thanks,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [placement] extraction (technical) update

[Jay Pipes]

On 08/27/2018 11:31 AM, Matt Riedemann wrote:

On 8/24/2018 7:36 AM, Chris Dent wrote:


Over the past few days a few of us have been experimenting with
extracting placement to its own repo, as has been discussed at
length on this list, and in some etherpads:

https://etherpad.openstack.org/p/placement-extract-stein
https://etherpad.openstack.org/p/placement-extraction-file-notes

As part of that, I've been doing some exploration to tease out the
issues we're going to hit as we do it. None of this is work that
will be merged, rather it is stuff to figure out what we need to
know to do the eventual merging correctly and efficiently.

Please note that doing that is just the near edge of a large
collection of changes that will cascade in many ways to many
projects, tools, distros, etc. The people doing this are aware of
that, and the relative simplicity (and fairly immediate success) of
these experiments is not misleading people into thinking "hey, no
big deal". It's a big deal.

There's a strategy now (described at the end of the first etherpad
listed above) for trimming the nova history to create a thing which
is placement. From the first run of that Ed created a github repo
and I branched that to eventually create:

https://github.com/EdLeafe/placement/pull/2

In that, all the placement unit and functional tests are now
passing, and my placecat [1] integration suite also passes.

That work has highlighted some gaps in the process for trimming
history which will be refined to create another interim repo. We'll
repeat this until the process is smooth, eventually resulting in an
openstack/placement.


We talked about the github strategy a bit in the placement meeting today 
[1]. Without being involved in this technical extraction work for the 
past few weeks, I came in with a different perspective on the end-game, 
and it was not aligned with what Chris/Ed thought as far as how we get 
to the official openstack/placement repo.


At a high level, Ed's repo [2] is a fork of nova with large changes on 
top using pull requests to do things like remove the non-placement nova 
files, update import paths (because the import structure changes from 
nova.api.openstack.placement to just placement), and then changes from 
Chris [3] to get tests working. Then the idea was to just use that to 
seed the openstack/placement repo and rather than review the changes 
along the way*, people that care about what changed (like myself) would 
see the tests passing and be happy enough.


However, I disagree with this approach since it bypasses our community 
code review system of using Gerrit and relying on a core team to approve 
changes at the sake of expediency.


What I would like to see are the changes that go into making the seed 
repo and what gets it to passing tests done in gerrit like we do for 
everything else. There are a couple of options on how this is done though:


1. Seed the openstack/placement repo with the filter_git_history.sh 
script output as Ed has done here [4]. This would include moving the 
placement files to the root of the tree and dropping nova-specific 
files. Then make incremental changes in gerrit like with [5] and the 
individual changes which make up Chris's big pull request [3]. I am 
primarily interested in making sure there are not content changes 
happening, only mechanical tree-restructuring type changes, stuff like 
that. I'm asking for more changes in gerrit so they can be sanely 
reviewed (per normal).


2. Eric took a slightly different tack in that he's OK with just a 
couple of large changes (or even large patch sets within a single 
change) in gerrit rather than ~30 individual changes. So that would be 
more like at most 3 changes in gerrit for [4][5][3].


3. The 3rd option is we just don't use gerrit at all and seed the 
official repo with the results of Chris and Ed's work in Ed's repo in 
github. Clearly this would be the fastest way to get us to a new repo 
(at the expense of bucking community code review and development process 
- is an exception worth it?).


Option 1 would clearly be a drain on at least 2 nova cores to go through 
the changes. I think Eric is on board for reviewing options 1 or 2 in 
either case, but he prefers option 2. Since I'm throwing a wrench in the 
works, I also need to stand up and review the changes if we go with 
option 1 or 2. Jay said he'd review them but consider these reviews 
lower priority. I expect we could get some help from some other nova 
cores though, maybe not on all changes, but at least some (thinking 
gibi, alex_xu, sfinucan).


Any CI jobs would be non-voting while going through options 1 or 2 until 
we get to a point that tests should finally be passing and we can make 
them voting (it should be possible to control this within the repo 
itself using zuul v3).


I would like to know from others (nova core or otherwise) what they 
would prefer, and if you are a nova core that wants option 1 (or 2) are 
you willing to help review 

Re: [openstack-dev] [nova] how nova should behave when placement returns consumer generation conflict

[Jay Pipes]

On 08/22/2018 08:55 AM, Balázs Gibizer wrote:

On Fri, Aug 17, 2018 at 5:40 PM, Eric Fried  wrote:

gibi-

 - On migration, when we transfer the allocations in either 
direction, a

 conflict means someone managed to resize (or otherwise change
 allocations?) since the last time we pulled data. Given the global 
lock

 in the report client, this should have been tough to do. If it does
 happen, I would think any retry would need to be done all the way back
 at the claim, which I imagine is higher up than we should go. So 
again,

 I think we should fail the migration and make the user retry.


 Do we want to fail the whole migration or just the migration step (e.g.
 confirm, revert)?
 The later means that failure during confirm or revert would put the
 instance back to VERIFY_RESIZE. While the former would mean that in 
case
 of conflict at confirm we try an automatic revert. But for a 
conflict at

 revert we can only put the instance to ERROR state.


This again should be "impossible" to come across. What would the
behavior be if we hit, say, ValueError in this spot?


I might not totally follow you. I see two options to choose from for the 
revert case:


a) Allocation manipulation error during revert of a migration causes 
that instance goes to ERROR. -> end user cannot retry the revert the 
instance needs to be deleted.


I would say this one is correct, but not because the user did anything 
wrong. Rather, *something inside Nova failed* because technically Nova 
shouldn't allow resource allocation to change while a server is in 
CONFIRMING_RESIZE task state. If we didn't make the server go to an 
ERROR state, I'm afraid we'd have no indication anywhere that this 
improper situation ever happened and we'd end up hiding some serious 
data corruption bugs.


b) Allocation manipulation error during revert of a migration causes 
that the instance goes back to VERIFY_RESIZE state. -> end user can 
retry the revert via the API.


I see three options to choose from for the confirm case:

a) Allocation manipulation error during confirm of a migration causes 
that instance goes to ERROR. -> end user cannot retry the confirm the 
instance needs to be deleted.


For the same reasons outlined above, I think this is the only safe option.

Best,
-jay

b) Allocation manipulation error during confirm of a migration causes 
that the instance goes back to VERIFY_RESIZE state. -> end user can 
retry the confirm via the API.


c) Allocation manipulation error during confirm of a migration causes 
that nova automatically tries to revert the migration. (For failure 
during this revert the same options available as for the generic revert 
case, see above)


We also need to consider live migration. It is similar in a sense that 
it also use move_allocations. But it is different as the end user 
doesn't explicitly confirm or revert a live migration.


I'm looking for opinions about which option we should take in each cases.

gibi



-efried

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] how nova should behave when placement returns consumer generation conflict

[Jay Pipes]

Sorry for the delay in responding to this, Gibi and Eric. Comments inline.

tl;dr: go with option a)

On 08/16/2018 11:34 AM, Eric Fried wrote:

Thanks for this, gibi.

TL;DR: a).

I didn't look, but I'm pretty sure we're not caching allocations in the
report client. Today, nobody outside of nova (specifically the resource
tracker via the report client) is supposed to be mucking with instance
allocations, right? And given the global lock in the resource tracker,
it should be pretty difficult to race e.g. a resize and a delete in any
meaningful way.


It's not a global (i.e. multi-node) lock. It's a semaphore for just that 
compute node. Migrations (mostly) involve more than one compute node, so 
the compute node semaphore is useless in that regard, thus the need to 
go with option a) and bail out if any change to a generation of any of 
the consumers involved in the migration operation.



So short term, IMO it is reasonable to treat any generation conflict
as an error. No retries. Possible wrinkle on delete, where it should
be a failure unless forced.


Agreed for all migration and deletion operations.


Long term, I also can't come up with any scenario where it would be
appropriate to do a narrowly-focused GET+merge/replace+retry. But
implementing the above short-term plan shouldn't prevent us from adding
retries for individual scenarios later if we do uncover places where it
makes sense.


Neither do I. Safety first, IMHO.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] UUID sentinel needs a home

[Jay Pipes]

On 08/24/2018 07:51 PM, Matt Riedemann wrote:

On 8/23/2018 2:05 PM, Chris Dent wrote:

On Thu, 23 Aug 2018, Dan Smith wrote:


...and it doesn't work like mock.sentinel does, which is part of the
value. I really think we should put this wherever it needs to be so that
it can continue to be as useful as is is today. Even if that means just
copying it into another project -- it's not that complicated of a thing.


Yeah, I agree. I had hoped that we could make something that was
generally useful, but its main value is its interface and if we
can't have that interface in a library, having it per codebase is no
biggie. For example it's been copied straight from nova into the
placement extractions experiments with no changes and, as one would
expect, works just fine.

Unless people are wed to doing something else, Dan's right, let's
just do that.


So just follow me here people, what if we had this common shared library 
where code could incubate and then we could write some tools to easily 
copy that common code into other projects...


Sounds masterful.

I'm pretty sure I could get said project approved as a top-level program 
under The Foundation and might even get a talk or two out of this idea. 
I can see the Intel money rolling in now...


Indeed, I'll open the commons bank account.

Ciao,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] UUID sentinel needs a home

[Jay Pipes]

On 08/23/2018 01:25 PM, Doug Hellmann wrote:

Excerpts from Eric Fried's message of 2018-08-23 09:51:21 -0500:

Do you mean an actual fixture, that would be used like:

  class MyTestCase(testtools.TestCase):
  def setUp(self):
  self.uuids = self.useFixture(oslofx.UUIDSentinelFixture()).uuids

  def test_foo(self):
  do_a_thing_with(self.uuids.foo)

?

That's... okay I guess, but the refactoring necessary to cut over to it
will now entail adding 'self.' to every reference. Is there any way
around that?


That is what I had envisioned, yes.  In the absence of a global,
which we do not want, what other API would you propose?


As dansmith mentioned, the niceness and simplicity of being able to do:

 import nova.tests.uuidsentinel as uuids

 ..

 def test_something(self):
 my_uuid = uuids.instance1

is remarkably powerful and is something I would want to keep.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jay Pipes]

Dan, thanks for the details and answers. Appreciated.

Best,
-jay

On 08/23/2018 10:50 AM, Dan Prince wrote:

On Wed, Aug 15, 2018 at 5:49 PM Jay Pipes  wrote:


On 08/15/2018 04:01 PM, Emilien Macchi wrote:

On Wed, Aug 15, 2018 at 5:31 PM Emilien Macchi mailto:emil...@redhat.com>> wrote:

 More seriously here: there is an ongoing effort to converge the
 tools around containerization within Red Hat, and we, TripleO are
 interested to continue the containerization of our services (which
 was initially done with Docker & Docker-Distribution).
 We're looking at how these containers could be managed by k8s one
 day but way before that we plan to swap out Docker and join CRI-O
 efforts, which seem to be using Podman + Buildah (among other things).

I guess my wording wasn't the best but Alex explained way better here:
http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-08-15.log.html#t2018-08-15T17:56:52

If I may have a chance to rephrase, I guess our current intention is to
continue our containerization and investigate how we can improve our
tooling to better orchestrate the containers.
We have a nice interface (openstack/paunch) that allows us to run
multiple container backends, and we're currently looking outside of
Docker to see how we could solve our current challenges with the new tools.
We're looking at CRI-O because it happens to be a project with a great
community, focusing on some problems that we, TripleO have been facing
since we containerized our services.

We're doing all of this in the open, so feel free to ask any question.


I appreciate your response, Emilien, thank you. Alex' responses to
Jeremy on the #openstack-tc channel were informative, thank you Alex.

For now, it *seems* to me that all of the chosen tooling is very Red Hat
centric. Which makes sense to me, considering Triple-O is a Red Hat product.


Perhaps a slight clarification here is needed. "Director" is a Red Hat
product. TripleO is an upstream project that is now largely driven by
Red Hat and is today marked as single vendor. We welcome others to
contribute to the project upstream just like anybody else.

And for those who don't know the history the TripleO project was once
multi-vendor as well. So a lot of the abstractions we have in place
could easily be extended to support distro specific implementation
details. (Kind of what I view podman as in the scope of this thread).



I don't know how much of the current reinvention of container runtimes
and various tooling around containers is the result of politics. I don't
know how much is the result of certain companies wanting to "own" the
container stack from top to bottom. Or how much is a result of technical
disagreements that simply cannot (or will not) be resolved among
contributors in the container development ecosystem.

Or is it some combination of the above? I don't know.

What I *do* know is that the current "NIH du jour" mentality currently
playing itself out in the container ecosystem -- reminding me very much
of the Javascript ecosystem -- makes it difficult for any potential
*consumers* of container libraries, runtimes or applications to be
confident that any choice they make towards one of the other will be the
*right* choice or even a *possible* choice next year -- or next week.
Perhaps this is why things like openstack/paunch exist -- to give you
options if something doesn't pan out.


This is exactly why paunch exists.

Re, the podman thing I look at it as an implementation detail. The
good news is that given it is almost a parity replacement for what we
already use we'll still contribute to the OpenStack community in
similar ways. Ultimately whether you run 'docker run' or 'podman run'
you end up with the same thing as far as the existing TripleO
architecture goes.

Dan



You have a tough job. I wish you all the luck in the world in making
these decisions and hope politics and internal corporate management
decisions play as little a role in them as possible.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack-operators] [glance] share image with domain

[Jay Pipes]

On 08/22/2018 11:05 AM, Brian Rosmaita wrote:

On Tue, Jul 10, 2018 at 8:04 AM Christian Berendt
 wrote:


It is possible to add a domain as a member, however this is not taken in 
account. It should be mentioned that you can also add non-existing project ids 
as a member.


Yes, you can add any string as an image member.  Back when image
sharing was implemented, it was thought that making an extra call to
keystone to verify the member_id would be too expensive because people
would be sharing so many images all the time, but AFAIK, that hasn't
turned out to be the case.  So it may be worth revisiting this issue.


It's not just that. It's that if someone then deletes the project from 
Keystone, it's not like Glance listens for that project delete 
notification and takes some remediation.


Instead, Glance would just keep a now-orphaned project ID in its 
membership data store. Which is perfectly fine IMHO. Is it ideal? No, 
but does it hurt anything? Not really... You could make some cleanup 
out-of-band process that looked for orphaned information and cleaned it 
up later on.


Such is the life of highly distributed services with no single data 
store that uses referential integrity... :)


Best,
-jay


For me it looks like it is not possible to share a image with visibility 
“shared” with a domain.


That is correct, the items in the member-list are treated as project
IDs.  (Actually, that's not entirely true, but will be completely true
very early in the Stein development cycle.[0])


Are there known workarounds or scripts for that use case?


I'm not aware of any.  You could write a script that took a domain ID,
got the list of projects in that domain, and shared the image with all
those projects, but then you'd have a synchronization problem when
projects were added to/removed from the domain, so that's probably not
a good idea.

If this is an important use case, please consider proposing a Glance
spec [1] or proposing it as a topic for the upcoming PTG [2].

cheers,
brian

[0] 
https://specs.openstack.org/openstack/glance-specs/specs/rocky/implemented/glance/spec-lite-deprecate-owner_is_tenant.html
[1] http://git.openstack.org/cgit/openstack/glance-specs
[2] https://etherpad.openstack.org/p/stein-ptg-glance-planning


Christian.

--
Christian Berendt
Chief Executive Officer (CEO)

Mail: bere...@betacloud-solutions.de
Web: https://www.betacloud-solutions.de

Betacloud Solutions GmbH
Teckstrasse 62 / 70190 Stuttgart / Deutschland

Geschäftsführer: Christian Berendt
Unternehmenssitz: Stuttgart
Amtsgericht: Stuttgart, HRB 756139


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] [oslo] UUID sentinel needs a home

[Jay Pipes]

On 08/23/2018 08:06 AM, Doug Hellmann wrote:

Excerpts from Davanum Srinivas (dims)'s message of 2018-08-23 06:46:38 -0400:

Where exactly Eric? I can't seem to find the import:

http://codesearch.openstack.org/?q=(from%7Cimport).*oslotest=nope==oslo.utils

-- dims


oslo.utils depends on oslotest via test-requirements.txt and oslotest is
used within the test modules in oslo.utils.

As I've said on both reviews, I think we do not want a global
singleton instance of this sentinal class. We do want a formal test
fixture.  Either library can export a test fixture and olso.utils
already has oslo_utils.fixture.TimeFixture so there's precedent to
adding it there, so I have a slight preference for just doing that.

That said, oslo_utils.uuidutils.generate_uuid() is simply returning
str(uuid.uuid4()). We have it wrapped up as a function so we can
mock it out in other tests, but we hardly need to rely on that if
we're making a test fixture for oslotest.

My vote is to add a new fixture class to oslo_utils.fixture.


OK, thanks for the helpful explanation, Doug. Works for me.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] UUID sentinel needs a home

[Jay Pipes]

On Wed, Aug 22, 2018, 10:13 AM Eric Fried  wrote:

> For some time, nova has been using uuidsentinel [1] which conveniently
> allows you to get a random UUID in a single LOC with a readable name
> that's the same every time you reference it within that process (but not
> across processes). Example usage: [2].
>
> We would like other projects (notably the soon-to-be-split-out placement
> project) to be able to use uuidsentinel without duplicating the code. So
> we would like to stuff it in an oslo lib.
>
> The question is whether it should live in oslotest [3] or in
> oslo_utils.uuidutils [4]. The proposed patches are (almost) the same.
> The issues we've thought of so far:
>
> - If this thing is used only for test, oslotest makes sense. We haven't
> thought of a non-test use, but somebody surely will.
> - Conversely, if we put it in oslo_utils, we're kinda saying we support
> it for non-test too. (This is why the oslo_utils version does some extra
> work for thread safety and collision avoidance.)
> - In oslotest, awkwardness is necessary to avoid circular importing:
> uuidsentinel uses oslo_utils.uuidutils, which requires oslotest. In
> oslo_utils.uuidutils, everything is right there.
>

My preference is to put it in oslotest. Why does oslo_utils.uuidutils
import oslotest? That makes zero sense to me...

-jay

- It's a... UUID util. If I didn't know anything and I was looking for a
> UUID util like uuidsentinel, I would look in a module called uuidutils
> first.
>
> We hereby solicit your opinions, either by further discussion here or as
> votes on the respective patches.
>
> Thanks,
> efried
>
> [1]
>
> https://github.com/openstack/nova/blob/17b69575bc240ca1dd8b7a681de846d90f3b642c/nova/tests/uuidsentinel.py
> [2]
>
> https://github.com/openstack/nova/blob/17b69575bc240ca1dd8b7a681de846d90f3b642c/nova/tests/functional/api/openstack/placement/db/test_resource_provider.py#L109-L115
> [3] https://review.openstack.org/594068
> [4] https://review.openstack.org/594179
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [neutron] live migration with multiple port bindings.

[Jay Pipes]

On Tue, Aug 21, 2018, 8:29 AM Matt Riedemann  wrote:

> On 8/21/2018 7:28 AM, Matt Riedemann wrote:
> > On 8/20/2018 3:18 PM, Sean Mooney wrote:
> >> in both the ovs-dpdk tests, when the migration failed and the vm
> >> contiuned to run on the source node however
> >> it had no network connectivity. on hard reboot of the vm, it went to
> >> error state because the vif binding
> >> was set to none as the vif:bidning-details:host_id  was set to none so
> >> the vif_type was also set to none.
> >> i have opened a nova bug to track the fact that the vm is left in an
> >> invalid state even though the status is active.
> >> see bug 1788014
> >
> > I've got a nova patch for this here:
> >
> > https://review.openstack.org/#/c/594139/
> >
> > However, I'd like Miguel to look at that bug because I assumed that when
> > nova deletes the dest host port binding, the only remaining port binding
> > is the inactive one for the source host, and neutron would automatically
> > activate it, similar to how neutron will automatically deactivate all
> > other bindings for a port when one of the other bindings is activated
> > (like when nova activates the dest host port binding during live
> > migration, the source host port binding is automatically deactivated
> > because only one port binding can be active at any time). If there is a
> > good reason why neutron doesn't do this on port binding delete, then I
> > guess we go with fixing this in nova.
> >
>
> By the way, Sean, thanks a ton for doing all of this testing. It's super
> helpful and way above anything I could have gotten setup myself for the
> various neutron backend configurations.
>

Agreed, big +1 and thanks to Sean for doing this.

However, I'd like to point out that this highlights the unfortunate
situation we're in: only a select couple contributors actually are able to
understand the overly complex, ludicrously inconsistent, and all too often
incompatible networking technologies that OpenStack has come to rely on 

This reminds me of a recent conversation I had on Twitter with an old
coworker of mine who is now at booking. com who stated the frustrating
complexity of networking and SDN setup in OpenStack was the reason he
switched to Kubernetes and hasn't looked back since.

-jay


> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Jay Pipes]

On 08/18/2018 08:25 AM, Chris Dent wrote:

So my hope is that (in no particular order) Jay Pipes, Eric Fried,
Takashi Natsume, Tetsuro Nakamura, Matt Riedemann, Andrey Volkov,
Alex Xu, Balazs Gibizer, Ed Leafe, and any other contributor to
placement whom I'm forgetting [1] would express their preference on
what they'd like to see happen.

At the same time, if people from neutron, cinder, blazar, zun,
mogan, ironic, and cyborg could express their preferences, we can get
through this by acclaim and get on with getting things done.


I am not opposed to extracting the placement service into its own repo. 
I also do not view it as a priority that should take precedence over the 
completion of other items, including the reshaper effort and the 
integration of placement calls into Nova (nested providers, sharing 
providers, etc).


The remaining items are Nova-centric. We need Nova-focused contributors 
to make placement more useful to Nova, and I fail to see how extracting 
the placement service will meet that goal. In fact, one might argue, as 
Melanie implies, that extracting placement outside of the Compute 
project would increase the velocity of the placement project *at the 
expense of* getting things done in the Nova project.


We've shown we can get many things done in placement. We've shown we can 
evolve the API fairly quickly. The velocity of the placement project 
isn't the problem. The problem is the lag between features being written 
into placement (sometimes too hastily IMHO) and actually *using* those 
features in Nova.


As for the argument about other projects being able (or being more 
willing to) use placement, I think that's not actually true. The 
projects that might want to ditch their own custom resource tracking and 
management code (Cyborg, Neutron, Cinder, Ironic) have either already 
done so or would require minimal changes to do that. There are no 
projects other than Ironic that I'm aware of that are interested in 
using the allocation candidates functionality (and the allocation claim 
process that entails) for the rough scheduling functionality that 
provides. I'm not sure placement being extracted would change that.


Would extracting placement out into its own repo result in a couple more 
people being added to the new placement core contributor team? Possibly. 
Will that result in Nova getting the integration pieces written that 
make use of placement? No, I don't believe so.


So, I'm on the fence. I understand the desire for separation, and I'm 
fully aware of my bias as a current Nova core contributor. I even 
support the process of extracting placement. But do I think it will do 
much other than provide some minor measure of independence? No, not really.


Consider me +0.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jay Pipes]

On 08/15/2018 04:01 PM, Emilien Macchi wrote:
On Wed, Aug 15, 2018 at 5:31 PM Emilien Macchi > wrote:


More seriously here: there is an ongoing effort to converge the
tools around containerization within Red Hat, and we, TripleO are
interested to continue the containerization of our services (which
was initially done with Docker & Docker-Distribution).
We're looking at how these containers could be managed by k8s one
day but way before that we plan to swap out Docker and join CRI-O
efforts, which seem to be using Podman + Buildah (among other things).

I guess my wording wasn't the best but Alex explained way better here:
http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-08-15.log.html#t2018-08-15T17:56:52

If I may have a chance to rephrase, I guess our current intention is to 
continue our containerization and investigate how we can improve our 
tooling to better orchestrate the containers.
We have a nice interface (openstack/paunch) that allows us to run 
multiple container backends, and we're currently looking outside of 
Docker to see how we could solve our current challenges with the new tools.
We're looking at CRI-O because it happens to be a project with a great 
community, focusing on some problems that we, TripleO have been facing 
since we containerized our services.


We're doing all of this in the open, so feel free to ask any question.


I appreciate your response, Emilien, thank you. Alex' responses to 
Jeremy on the #openstack-tc channel were informative, thank you Alex.


For now, it *seems* to me that all of the chosen tooling is very Red Hat 
centric. Which makes sense to me, considering Triple-O is a Red Hat product.


I don't know how much of the current reinvention of container runtimes 
and various tooling around containers is the result of politics. I don't 
know how much is the result of certain companies wanting to "own" the 
container stack from top to bottom. Or how much is a result of technical 
disagreements that simply cannot (or will not) be resolved among 
contributors in the container development ecosystem.


Or is it some combination of the above? I don't know.

What I *do* know is that the current "NIH du jour" mentality currently 
playing itself out in the container ecosystem -- reminding me very much 
of the Javascript ecosystem -- makes it difficult for any potential 
*consumers* of container libraries, runtimes or applications to be 
confident that any choice they make towards one of the other will be the 
*right* choice or even a *possible* choice next year -- or next week. 
Perhaps this is why things like openstack/paunch exist -- to give you 
options if something doesn't pan out.


You have a tough job. I wish you all the luck in the world in making 
these decisions and hope politics and internal corporate management 
decisions play as little a role in them as possible.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jay Pipes]

On 08/15/2018 05:32 AM, Cédric Jeanneret wrote:

Dear Community,

As you may know, a move toward Podman as replacement of Docker is starting.


This was news to me. Is this just a triple-o thing?

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][tooz][etcd] need help debugging tooz test failure

[Jay Pipes]

On 08/13/2018 03:52 PM, Doug Hellmann wrote:

Excerpts from Jay Pipes's message of 2018-08-13 13:21:56 -0400:

On 08/12/2018 04:11 PM, Doug Hellmann wrote:

The tooz tests on master and stable/rocky are failing with an error:

  UnicodeDecodeError: 'utf8' codec can't decode byte 0xc4 in position 0:
  invalid continuation byte

This is unrelated to the change, which is simply importing test job
settings or updating the .gitreview file. I need someone familiar with
the library to help debug the issue.

Can we get a volunteer?


Looking into it. Seems to be related to this upstream patch to
python-etcd3gw:

https://github.com/dims/etcd3-gateway/commit/224f40972b42c4ff16234c0e78ea765e3fe1af95

Best,
-jay



Thanks, Jay!

I see that Dims says he pushed a release. Is that something we need to
update in the constraints list, then?


Yeah, likely. We'll need to blacklist the 0.2.3 release of etcd3-gateway 
library in the openstack/tooz requirements file.


I think? :)

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][tooz][etcd] need help debugging tooz test failure

[Jay Pipes]

On 08/12/2018 04:11 PM, Doug Hellmann wrote:

The tooz tests on master and stable/rocky are failing with an error:

 UnicodeDecodeError: 'utf8' codec can't decode byte 0xc4 in position 0:
 invalid continuation byte

This is unrelated to the change, which is simply importing test job
settings or updating the .gitreview file. I need someone familiar with
the library to help debug the issue.

Can we get a volunteer?


Looking into it. Seems to be related to this upstream patch to 
python-etcd3gw:


https://github.com/dims/etcd3-gateway/commit/224f40972b42c4ff16234c0e78ea765e3fe1af95

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we still want to lowercase metadata keys?

[Jay Pipes]

On 08/13/2018 11:56 AM, Chris Friesen wrote:

On 08/13/2018 08:26 AM, Jay Pipes wrote:

On 08/13/2018 10:10 AM, Matthew Booth wrote:



I suspect I've misunderstood, but I was arguing this is an anti-goal.
There's no reason to do this if the db is working correctly, and it
would violate the principal of least surprise in dbs with legacy
datasets (being all current dbs). These values have always been mixed
case, lets just leave them be and fix the db.


Do you want case-insensitive keys or do you not want case-insensitive 
keys?


It seems to me that people complain that MySQL is case-insensitive by 
default
but actually *like* the concept that a metadata key of "abc" should be 
"equal

to" a metadata key of "ABC".


How do we behave on PostgreSQL?  (I realize it's unsupported, but it 
still has users.)  It's case-sensitive by default, do we override that?


Personally, I've worked on case-sensitive systems long enough that I'd 
actually be surprised if "abc" matched "ABC". :)


You have worked with case-insensitive systems for as long or longer, 
maybe without realizing it: All URLs are case-insensitive.


If a user types in http://google.com they go to the same place as 
http://Google.com because DNS is case-insensitive [1] and has been since 
its beginning. Users -- of HTTP APIs in particular -- have tended to 
become accustomed to case-insensitivity in their HTTP API calls.


This case is no different, IMHO.

Best,
-jay

[1] https://tools.ietf.org/html/rfc4343#section-4

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we still want to lowercase metadata keys?

[Jay Pipes]

On 08/13/2018 10:10 AM, Matthew Booth wrote:

On Mon, 13 Aug 2018 at 14:05, Jay Pipes  wrote:


On 08/13/2018 06:06 AM, Matthew Booth wrote:

Thanks mriedem for answering my previous question, and also pointing
out the related previous spec around just forcing all metadata to be
lowercase:

(Spec: approved in Newton) https://review.openstack.org/#/c/311529/
(Online migration: not merged, abandoned)
https://review.openstack.org/#/c/329737/

There are other code patches, but the above is representative. What I
had read was the original bug:

https://bugs.launchpad.net/nova/+bug/1538011

The tl;dr is that the default collation used by MySQL results in a bug
when creating 2 metadata keys which differ only in case. The proposal
was obviously to simply make all metadata keys lower case. However, as
melwitt pointed out in the bug at the time that's a potentially user
hostile change. After some lost IRC discussion it seems that folks
believed at the time that to fix this properly would seriously
compromise the performance of these queries. The agreed way forward
was to allow existing keys to keep their case, but force new keys to
be lower case (so I wonder how the above online migration came
about?).

Anyway, as Rajesh's patch shows, it's actually very easy just to fix
the MySQL misconfiguration:

https://review.openstack.org/#/c/504885/

So my question is, given that the previous series remains potentially
user hostile, the fix isn't as complex as previously believed, and it
doesn't involve a performance penalty, are there any other reasons why
we might want to resurrect it rather than just go with Rajesh's patch?
Or should we ask Rajesh to expand his patch into a series covering
other metadata?


Keep in mind this patch is only related to *aggregate* metadata, AFAICT.


Right, but the original bug pointed out that the same problem applies
equally to a bunch of different metadata stores. I haven't verified,
but the provenance was good ;) There would have to be other patches
for the other metadata stores.


Yes, it is quite unfortunate that OpenStack has about 15 different ways 
of storing metadata key/value information.




Any patch series that tries to "fix" this issue needs to include all of
the following:

* input automatically lower-cased [1]
* inline (note: not online, inline) data migration inside the
InstanceMeta object's _from_db_object() method for existing
non-lowercased keys


I suspect I've misunderstood, but I was arguing this is an anti-goal.
There's no reason to do this if the db is working correctly, and it
would violate the principal of least surprise in dbs with legacy
datasets (being all current dbs). These values have always been mixed
case, lets just leave them be and fix the db.


Do you want case-insensitive keys or do you not want case-insensitive keys?

It seems to me that people complain that MySQL is case-insensitive by 
default but actually *like* the concept that a metadata key of "abc" 
should be "equal to" a metadata key of "ABC".


In other words, it seems to me that users actually expect that:

> nova aggregate-create agg1
> nova aggregate-set-metadata agg1 abc=1
> nova aggregate-set-metadata agg1 ABC=2

should result in the original "abc" metadata item getting its value set 
to "2".


If that isn't the case -- and I have a very different impression of what 
users *actually* expect from the CLI/UI -- then let me know.


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we still want to lowercase metadata keys?

[Jay Pipes]

On 08/13/2018 06:06 AM, Matthew Booth wrote:

Thanks mriedem for answering my previous question, and also pointing
out the related previous spec around just forcing all metadata to be
lowercase:

(Spec: approved in Newton) https://review.openstack.org/#/c/311529/
(Online migration: not merged, abandoned)
https://review.openstack.org/#/c/329737/

There are other code patches, but the above is representative. What I
had read was the original bug:

https://bugs.launchpad.net/nova/+bug/1538011

The tl;dr is that the default collation used by MySQL results in a bug
when creating 2 metadata keys which differ only in case. The proposal
was obviously to simply make all metadata keys lower case. However, as
melwitt pointed out in the bug at the time that's a potentially user
hostile change. After some lost IRC discussion it seems that folks
believed at the time that to fix this properly would seriously
compromise the performance of these queries. The agreed way forward
was to allow existing keys to keep their case, but force new keys to
be lower case (so I wonder how the above online migration came
about?).

Anyway, as Rajesh's patch shows, it's actually very easy just to fix
the MySQL misconfiguration:

https://review.openstack.org/#/c/504885/

So my question is, given that the previous series remains potentially
user hostile, the fix isn't as complex as previously believed, and it
doesn't involve a performance penalty, are there any other reasons why
we might want to resurrect it rather than just go with Rajesh's patch?
Or should we ask Rajesh to expand his patch into a series covering
other metadata?


Keep in mind this patch is only related to *aggregate* metadata, AFAICT.

Any patch series that tries to "fix" this issue needs to include all of 
the following:


* input automatically lower-cased [1]
* inline (note: not online, inline) data migration inside the 
InstanceMeta object's _from_db_object() method for existing 
non-lowercased keys
* change the collation of the aggregate_metadata.key column (note: this 
will require an entire rebuild of the table, since this column is part 
of a unique constraint [3]
* online data migration for migrating non-lowercased keys to their 
lowercased counterpars (essentially doing `UPDATE key = LOWER(key) WHERE 
LOWER(key) != key` once the collation has been changed)


None of the above touches the API layer. I suppose some might argue that 
the REST API should be microversion-bumped since the expected behaviour 
of the API will change (data will be transparently changed in one 
version of the API and not another). I don't personally think that's 
something I would require a microversion for, but who knows what others 
may say.


Best,
-jay

[1] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L295 
and 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L331 
and 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L356 



[2] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L248


[3] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/db/sqlalchemy/api_models.py#L64


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How to debug no valid host failures with placement

[Jay Pipes]

On Wed, Aug 1, 2018 at 11:15 AM, Ben Nemec  wrote:

> Hi,
>
> I'm having an issue with no valid host errors when starting instances and
> I'm struggling to figure out why.  I thought the problem was disk space,
> but I changed the disk_allocation_ratio and I'm still getting no valid
> host.  The host does have plenty of disk space free, so that shouldn't be a
> problem.
>
> However, I'm not even sure it's disk that's causing the failures because I
> can't find any information in the logs about why the no valid host is
> happening.  All I get from the scheduler is:
>
> "Got no allocation candidates from the Placement API. This may be a
> temporary occurrence as compute nodes start up and begin reporting
> inventory to the Placement service."
>
> While in placement I see:
>
> 2018-08-01 15:02:22.062 20 DEBUG nova.api.openstack.placement.requestlog
> [req-0a830ce9-e2af-413a-86cb-b47ae129b676 fc44fe5cefef43f4b921b9123c95e694
> b07e6dc2e6284b00ac7070aa3457c15e - default default] Starting request:
> 10.2.2.201 "GET /placement/allocation_candidat
> es?limit=1000=DISK_GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A1"
> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/placemen
> t/requestlog.py:38
> 2018-08-01 15:02:22.103 20 INFO nova.api.openstack.placement.requestlog
> [req-0a830ce9-e2af-413a-86cb-b47ae129b676 fc44fe5cefef43f4b921b9123c95e694
> b07e6dc2e6284b00ac7070aa3457c15e - default default] 10.2.2.201 "GET
> /placement/allocation_candidates?limit=1000=DISK_
> GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A1" status: 200 len: 53 microversion:
> 1.25
>
> Basically it just seems to be logging that it got a request, but there's
> no information about what it did with that request.
>
> So where do I go from here?  Is there somewhere else I can look to see why
> placement returned no candidates?
>
>
Hi again, Ben, hope you are enjoying your well-earned time off! :)

I've created a patch that (hopefully) will address some of the difficulty
that folks have had in diagnosing which parts of a request caused all
providers to be filtered out from the return of GET /allocation_candidates:

https://review.openstack.org/#/c/590041

This patch changes two primary things:

1) Query-splitting

The patch splits the existing monster SQL query that was being used for
querying for all providers that matched all requested resources, required
traits, forbidden traits and required aggregate associations into doing
multiple queries, one for each requested resource. While this does increase
the number of database queries executed for each call to GET
/allocation_candidates, the changes allow better visibility into what parts
of the request cause an exhaustion of matching providers. We've benchmarked
the new patch and have shown the performance impact of doing 3 queries
versus 1 (when there is a request for 3 resources -- VCPU, RAM and disk) is
minimal (a few extra milliseconds for execution against a DB with 1K
providers having inventory of all three resource classes).

2) Diagnostic logging output

The patch adds debug log output within each loop iteration, so there is no
logging output that shows how many matching providers were found for each
resource class involved in the request. The output looks like this in the
logs:

[req-2d30faa8-4190-4490-a91e-610045530140] inside VCPU request loop.
before applying trait and aggregate filters, found 12 matching
providers[req-2d30faa8-4190-4490-a91e-610045530140] found 12 providers
with capacity for the requested 1
VCPU.[req-2d30faa8-4190-4490-a91e-610045530140] inside MEMORY_MB
request loop. before applying trait and aggregate filters, found 9
matching providers[req-2d30faa8-4190-4490-a91e-610045530140] found 9
providers with capacity for the requested 64 MEMORY_MB. before loop
iteration we had 12 matches.[req-2d30faa8-4190-4490-a91e-610045530140]
RequestGroup(use_same_provider=False, resources={MEMORY_MB:64,
VCPU:1}, traits=[], aggregates=[]) (suffix '') returned 9 matches

If a request includes required traits, forbidden traits or required
aggregate associations, there are additional log messages showing how many
matching providers were found after applying the trait or aggregate
filtering set operation (in other words, the log output shows the impact of
the trait filter or aggregate filter in much the same way that the existing
FilterScheduler logging shows the "before and after" impact that a
particular filter had on a request process.

Have a look at the patch in question and please feel free to add your
feedback and comments on ways this can be improved to meet your needs.

Best,
-jay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][placement] Excessive WARNING level log messages in placement-api

[Jay Pipes]

For evidence, see:

http://logs.openstack.org/41/590041/1/check/tempest-full-py3/db08dec/controller/logs/screen-placement-api.txt.gz?level=WARNING

thousands of these are filling the logs with WARNING-level log messages, 
making it difficult to find anything:


Aug 08 22:17:30.837557 ubuntu-xenial-inap-mtl01-0001226060 
devstack@placement-api.service[14403]: WARNING py.warnings 
[req-a809b022-59af-4628-be73-488cfec3187d 
req-d46cb1f0-431f-490f-955b-b9c2cd9f6437 service placement] 
/usr/local/lib/python3.5/dist-packages/oslo_policy/policy.py:896: 
UserWarning: Policy placement:resource_providers:list failed scope 
check. The token used to make the request was project scoped but the 
policy requires ['system'] scope. This behavior may change in the future 
where using the intended scope is required
Aug 08 22:17:30.837800 ubuntu-xenial-inap-mtl01-0001226060 
devstack@placement-api.service[14403]:   warnings.warn(msg)
Aug 08 22:17:30.838067 ubuntu-xenial-inap-mtl01-0001226060 
devstack@placement-api.service[14403]:


Is there any way we can get rid of these?

Thanks,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] [nova] Log files on exceeding cpu allocation limit

[Jay Pipes]

py", line
139, in select_destinations
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager raise
exception.NoValidHost(reason="")
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager NoValidHost:
No valid host was found.
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:35.974 1648 ERROR nova.conductor.manager
2018-08-08 09:28:36.328 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] Failed to compute_task_build_instances: No valid host
was found.
Traceback (most recent call last):

   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 226, in inner
 return func(*args, **kwargs)

   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 139, in select_destinations
 raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
: NoValidHost_Remote: No valid host was found.
2018-08-08 09:28:36.331 1648 WARNING nova.scheduler.utils
[req-ef0d8ea1-e801-483e-b913-9148a6ac5d90
2499343cbc7a4ca5a7f14c43f9d9c229 3850596606b7459d8802a72516991a19 -
default default] [instance: b466a974-06ba-459b-bc04-2ccb2b3ee720]
Setting instance to ERROR state.: NoValidHost_Remote: No valid host
was found.
### END ###
On Wed, Aug 8, 2018 at 9:45 AM Jay Pipes  wrote:


On 08/08/2018 09:37 AM, Cody wrote:

On 08/08/2018 07:19 AM, Bernd Bausch wrote:

I would think you don't even reach the scheduling stage. Why bother
looking for a suitable compute node if you exceeded your quota anyway?

The message is in the conductor log because it's the conductor that does
most of the work. The others are just slackers (like nova-api) or wait
for instructions from the conductor.

The above is my guess, of course, but IMHO a very educated one.

Bernd.


Thank you, Bernd. I didn't know the inner workflow in this case.
Initially, I thought it was for the scheduler to discover that no more
resource was left available, hence I expected to see something from
the scheduler log. My understanding now is that the quota get checked
in the database prior to the deployment. That would explain why the
clue was in the nova-conductor.log, not the nova-scheduler.log.


Quota is checked in the nova-api node, not the nova-conductor.

As I said in my previous message, unless you paste what the logs are
that you are referring to, it's not possible to know what you are
referring to.

Best,
-jay


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [nova] Log files on exceeding cpu allocation limit

[Jay Pipes]

On 08/08/2018 09:37 AM, Cody wrote:

On 08/08/2018 07:19 AM, Bernd Bausch wrote:

I would think you don't even reach the scheduling stage. Why bother
looking for a suitable compute node if you exceeded your quota anyway?

The message is in the conductor log because it's the conductor that does
most of the work. The others are just slackers (like nova-api) or wait
for instructions from the conductor.

The above is my guess, of course, but IMHO a very educated one.

Bernd.


Thank you, Bernd. I didn't know the inner workflow in this case.
Initially, I thought it was for the scheduler to discover that no more
resource was left available, hence I expected to see something from
the scheduler log. My understanding now is that the quota get checked
in the database prior to the deployment. That would explain why the
clue was in the nova-conductor.log, not the nova-scheduler.log.


Quota is checked in the nova-api node, not the nova-conductor.

As I said in my previous message, unless you paste what the logs are 
that you are referring to, it's not possible to know what you are 
referring to.


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] Stepping down as coordinator for the Outreachy internships

[Jay Pipes]

On 08/08/2018 09:32 AM, Doug Hellmann wrote:

Excerpts from Victoria Martínez de la Cruz's message of 2018-08-07 20:47:28 
-0300:

Hi all,

I'm reaching you out to let you know that I'll be stepping down as
coordinator for OpenStack next round. I had been contributing to this
effort for several rounds now and I believe is a good moment for somebody
else to take the lead. You all know how important is Outreachy to me and
I'm grateful for all the amazing things I've done as part of the Outreachy
program and all the great people I've met in the way. I plan to keep
involved with the internships but leave the coordination tasks to somebody
else.

If you are interested in becoming an Outreachy coordinator, let me know and
I can share my experience and provide some guidance.

Thanks,

Victoria


Thank you, Victoria. Mentoring new developers is an important
responsibility, and your patient service in working with the Outreachy
program has set a good example.


Big +1.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev