Re: [openstack-dev] [nova] [placement] placement update 18-27

[Jay Pipes]

On 07/06/2018 10:09 AM, Chris Dent wrote:

# Questions

* Will consumer id, project and user id always be a UUID? We've
   established for certain that user id will not, but things are
   less clear for the other two. This issue is compounded by the
   fact that these two strings are different but the same UUID:
   5eb033fd-c550-420e-a31c-3ec2703a403c,
   5eb033fdc550420ea31c3ec2703a403c (bug 1758057 mentioned above) but
   we treat them differently in our code.


As mentioned by a couple people on IRC, a consumer's external project 
identifier and external user identifier come directly from Keystone. 
Since Keystone has no rule about these values being UUIDs or even 
UUID-like, we clearly cannot treat them as UUIDs in the placement service.


Our backend data storage for these attributes is suitably a String(255) 
column and there is no validation done on these values. In fact, the 
project and user external identifiers are taken directly from the 
nova.context WSGI environ when sending from the placement client [1].


So, really, the only thing we're discussing is whether consumer_id is 
always a UUID.


I believe it should be, and the fact that it's referred to as 
consumer_uuid in so many places should be indicative of its purpose. I 
know originally the field in the DB was a String(64), but it's since 
been changed to String(36), further evidence that consumer_id was 
intended to be a UUID.


I believe we should validate it as such at the placement API layer. The 
only current consumers in the placement service are instances and 
migrations, both of which use a UUID identifier. I don't think it's too 
onerous to require future consumers to be identified with a UUID, and it 
would be nice to be able to rely on a structured, agreed format for 
unique identification of consumers across services.


As noted the project_id and user_id are not required to be UUIDs and I 
don't believe we should add any validation for those fields.


Best,
-jay

[1] For those curious, nova-scheduler calls 
scheduler.utils.claim_resources(...):


https://github.com/openstack/nova/blob/8469fa70dafa83cb068538679100bede7679edc3/nova/scheduler/filter_scheduler.py#L219

which itself calls reportclient.claim_resources(...) with the 
instance.user_id and instance.project_id values:


https://github.com/openstack/nova/blob/8469fa70dafa83cb068538679100bede7679edc3/nova/scheduler/utils.py#L500

The instance.project_id and instance.user_id values are populated from 
the WSGI environ here:


https://github.com/openstack/nova/blob/8469fa70dafa83cb068538679100bede7679edc3/nova/compute/api.py#L831-L832

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [placement] placement update 18-27

[Jay Pipes]

On 07/09/2018 02:52 PM, Chris Dent wrote:

On Fri, 6 Jul 2018, Chris Dent wrote:


This is placement update 18-27, a weekly update of ongoing
development related to the [OpenStack](https://www.openstack.org/)
[placement
service](https://developer.openstack.org/api-ref/placement/). This
is a contract version.


Forgot to mention: There won't be an 18-28 this Friday, I'll be out
and about. If someone else would like to do one, that would be
great.


On it.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [placement] placement update 18-27

[Chris Dent]

On Fri, 6 Jul 2018, Chris Dent wrote:


This is placement update 18-27, a weekly update of ongoing
development related to the [OpenStack](https://www.openstack.org/)
[placement
service](https://developer.openstack.org/api-ref/placement/). This
is a contract version.


Forgot to mention: There won't be an 18-28 this Friday, I'll be out
and about. If someone else would like to do one, that would be
great.

--
Chris Dent   ٩◔̯◔۶   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [placement] placement update 18-27

[Chris Dent]

HTML: https://anticdent.org/placement-update-18-27.html

This is placement update 18-27, a weekly update of ongoing
development related to the [OpenStack](https://www.openstack.org/)
[placement
service](https://developer.openstack.org/api-ref/placement/). This
is a contract version.

# Most Important

In the past week or so we've found a two suites of bugs that are
holding up other work. One set is related to consumers and the
handling of consumer generations (linked in that theme, below). The
other is related to various ways in which managing parents of nested
providers is not correct. Those are:

* 
* 

The first is already fixed, the second was discovered as a result of
thinking about the first.

We also have some open questions about which of consumer id, project
id, and user id are definitely going to be a valid UUID and what
that means in relation to enforcement and our definition of "what's
a good uuid":

* 

As usual, this is more support for the fact that we need to be
doing increased manual testing to find where our automated tests
have gaps, and fill them.

On themes, we have several things rushing for attention before the
end of the cycle (reminder: Feature Freeze is the end of this
month). We need to get the various facets of consumers fixed up in a
way that we all agree is correct. We need to get the Reshaped
Providers implemented. And there's some hope (maybe vain?) that we
can get the report client and virt drivers talking in a more nested
and shared form.

# What's Changed

The microversion for nested allocation candidates has merged as
1.29.

The huge pile of osc-placement changes at

has merged. Yay!

# Bugs

* Placement related [bugs not yet in
  progress](https://goo.gl/TgiPXb): 16, -3 on last week.
* [In progress placement bugs](https://goo.gl/vzGGDQ) 17, +7 on last
  week.

# Questions

* Will consumer id, project and user id always be a UUID? We've
  established for certain that user id will not, but things are
  less clear for the other two. This issue is compounded by the
  fact that these two strings are different but the same UUID:
  5eb033fd-c550-420e-a31c-3ec2703a403c,
  5eb033fdc550420ea31c3ec2703a403c (bug 1758057 mentioned above) but
  we treat them differently in our code.

# Main Themes

## Documentation

This is a section for reminding us to document all the fun stuff we
are enabling. Open areas include:

* Documenting optional placement database. A bug,
  [1778227](https://bugs.launchpad.net/nova/+bug/1778227) has been
  created to track this. This has started, for the install docs, but
  there are more places that need to be touched.

* "How to deploy / model shared disk. Seems fairly straight-forward,
  and we could even maybe create a multi-node ceph job that does
  this - wouldn't that be awesome?!?!", says an enthusiastic Matt
  Riedemann.

* The whens and wheres of re-shaping and VGPUs.

## Nested providers in allocation candidates

The main code of this has merged. What's left are dealing with
things like the parenting bugs mentioned above, and actually
reporting any nested providers and inventory so we can make use of
them.

## Consumer Generations

A fair bit of open bugs fixes and debate on this stuff.

* 
  No ability to update consumer's project and/or user external ID
* 
  Consumers never get deleted 
* 

  Add UUID validation for consumer_uuid
  (This one has some of the discussion about whether consumers are
  always going to be UUIDs)
* 
  move lookup of provider from _new_allocations()
* 
  return 404 when no consumer found in allocs

Note that once this is correct we'll still have work to do in the
report client to handle consumer generations before nova can do
anything with it.

## Reshape Provider Trees

This allows moving inventory and allocations that were on resource
provider A to resource provider B in an atomic fashion. The
blueprint topic is:

* 

There are WIPs for the HTTP parts and the resource tracker parts, on
that topic.

## Mirror Host Aggregates

This needs a command line tool:

* 

## Extraction

Extraction is mostly taking a back seat at the moment while we
find and fix bugs in existing features. We've also done quite a lot
of the preparatory work. The main things to be thinking about are:

* os-resource-classes
* infra and co-gating issues that are going to come up
* copying whatever nova-based test fixture we might like

# Other

24 entries last week. 20 now (this is a contract week, there's
plenty of new