Re: [openstack-dev] [nova] os-capabilities library created

[Chris Dent]

On Tue, 16 Aug 2016, Alex Xu wrote:


There are two solutions in my mind
1. There aren't different dependence for traits, that is easy. The
validation API in the placement engine API is enough.
2. There are different dependence for traits. We need API for compute node
report those dependence. This isn't easy, we need model the dependence in
the placement engine. This sounds like Chris will hate this way :)


If you're talking about me here, "hate" is probably a bit too strong
of a term. Maybe "get sad"?

In any case my dislike is not with the idea of putting the capabilities
behind an API that can be accessed by a multitude of clients. That
seems like a good idea. It removes some dependencies and allows some
dynamism in the collection and management of traits. That is important
to allow.

What I dislike is the idea of including it in the placement service.
We should have a standalone traits service (for which the placement
service is one of the multitude of clients). I know people get
stressed about having yet another service running but if we continue
with the standard service evolution trend the placement service is
going to end up being yet another monolith in the OpenStack stack.
It should do one thing well and compose with other services to
satisfy use cases.

We should have a traits service. It should be very simple. It should
be amenable to simple caching for performance boosting and should be
easy to scale horizontally. If people want it to use the same
database as the placement engine, that's fine but it should be
its own process and ideally it's own code repo.

--
Chris Dent   ┬─┬ノ( º _ ºノ)https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Alex Xu]

2016-08-16 7:31 GMT+08:00 Jay Pipes :

> On 08/15/2016 03:57 AM, Alex Xu wrote:
>
>> 2016-08-15 12:56 GMT+08:00 Yingxin Cheng > >:
>>
>>
>> Hi,
>>
>> I'm concerned with the dependencies between "os-capabilities"
>> library and all the other OpenStack services such as Nova,
>> Placement, Ironic, etc.
>>
>> Rather than embedding the universal "os-capabilities" in Nova,
>> Cinder, Glance, Ironic services that will introduce complexities if
>> the library versions are different, I'd prefer to hide this library
>> behind the placement service and expose consistent interfaces as
>> well as caps to all the other services. But the drawback here is
>> also obvious: for example, when nova wants to support a new
>> capability, the development will require os-capabilities updates,
>> and related lib version bumps, which is inconvenient and seems
>> unnecessary.
>>
>>
>> +1 for the concern, this is good point. +1 for the hide os-capability
>> behind the placement service and expose consistent interfaces.
>>
>>
>>
>> So IMHO, the possible solution is:
>> * Let each services (Nova, Ironic ...) themselves manage their
>> capabilities under proper namespaces such as "compute", "ironic" or
>> "storage";
>>
>>
>> Actually I think we won't catalog the capabilities by services, we will
>> catalog the capabilities by resource types. Like nova and ironic may
>> share some capabilities, due to they provider compute resource.
>>
>>
>> * Let os-capabilities define as few as caps possible that are only
>> cross-project;
>> * And also use os-capabilities to convert service-defined or
>> user-defined caps to a standardized and distinct form that can be
>> understood by the placement engine.
>>
>>
>> This solution sounds complex for me, you need define the capabilities
>> many different places(in os-capabilities for cross-project caps, in each
>> service for project-only caps), and this may leads to different service
>> define same caps but with own naming. And due to the dependence is a
>> library. When you upgrade the os-capabilities, you have to restart all
>> the services.
>>
>> I'm thinking of another solution:
>>
>> 1. os-capabilities just define the capabilities into the JSON/YAML files.
>> 2. We expose the capabilities through the REST API of placement engine,
>> this REST API will read the capabilities from the JSON/YAML files
>> defined by os-capabilities.
>>
>> This resolves some problems as below:
>> 1. your concern, hide the os-capabilities behind one single API
>> 2. when os-capabilities upgrade, you needn't update all the
>> os-capabilities library for all control nodes. You only need update the
>> os-capabilities for the node running placement engine. Due to the
>> capabilities in the JSON/YAML files, you even needn't restart the
>> placement engine service.
>>
>> But yes, this also can be done by glance metadata API.
>>
>
> Yeah, I think hiding the catalog of constant trait strings behind the
> placement API is probably a good thing to do.
>
> However, I envision that os-traits (I'm now calling it this...) will
> contain some modules for discovery of various traits on a resource provider
> as well as modules that can detect conflicts in a request for a list of
> traits. Those modules would clearly be something that various services
> would want to import, irrespective of the import of the constant trait
> string catalog.
>

If we hide the trait strings behind the placement API, there will be an API
endpoint for validate the request caps. In nova, we can access this API
directly without os-traits.

One thing I'm not sure yet, we have different virt dirvers for different
hypervisors, so whether all the virt drivers have same dependence between
traits. For example, hyperV have gen1 and gen2 which support different
caps, there are some caps are conflicts between gen1 and gen2. But maybe in
another hypervisor, those caps aren't conflicts ether other. But I didn't
find out a real use-case yet. I need dig into more.

There are two solutions in my mind
1. There aren't different dependence for traits, that is easy. The
validation API in the placement engine API is enough.
2. There are different dependence for traits. We need API for compute node
report those dependence. This isn't easy, we need model the dependence in
the placement engine. This sounds like Chris will hate this way :)

Anyway I need to dig into more.


>
> Thoughts?
> -jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/15/2016 03:57 AM, Alex Xu wrote:

2016-08-15 12:56 GMT+08:00 Yingxin Cheng >:

Hi,

I'm concerned with the dependencies between "os-capabilities"
library and all the other OpenStack services such as Nova,
Placement, Ironic, etc.

Rather than embedding the universal "os-capabilities" in Nova,
Cinder, Glance, Ironic services that will introduce complexities if
the library versions are different, I'd prefer to hide this library
behind the placement service and expose consistent interfaces as
well as caps to all the other services. But the drawback here is
also obvious: for example, when nova wants to support a new
capability, the development will require os-capabilities updates,
and related lib version bumps, which is inconvenient and seems
unnecessary.


+1 for the concern, this is good point. +1 for the hide os-capability
behind the placement service and expose consistent interfaces.



So IMHO, the possible solution is:
* Let each services (Nova, Ironic ...) themselves manage their
capabilities under proper namespaces such as "compute", "ironic" or
"storage";


Actually I think we won't catalog the capabilities by services, we will
catalog the capabilities by resource types. Like nova and ironic may
share some capabilities, due to they provider compute resource.


* Let os-capabilities define as few as caps possible that are only
cross-project;
* And also use os-capabilities to convert service-defined or
user-defined caps to a standardized and distinct form that can be
understood by the placement engine.


This solution sounds complex for me, you need define the capabilities
many different places(in os-capabilities for cross-project caps, in each
service for project-only caps), and this may leads to different service
define same caps but with own naming. And due to the dependence is a
library. When you upgrade the os-capabilities, you have to restart all
the services.

I'm thinking of another solution:

1. os-capabilities just define the capabilities into the JSON/YAML files.
2. We expose the capabilities through the REST API of placement engine,
this REST API will read the capabilities from the JSON/YAML files
defined by os-capabilities.

This resolves some problems as below:
1. your concern, hide the os-capabilities behind one single API
2. when os-capabilities upgrade, you needn't update all the
os-capabilities library for all control nodes. You only need update the
os-capabilities for the node running placement engine. Due to the
capabilities in the JSON/YAML files, you even needn't restart the
placement engine service.

But yes, this also can be done by glance metadata API.


Yeah, I think hiding the catalog of constant trait strings behind the 
placement API is probably a good thing to do.


However, I envision that os-traits (I'm now calling it this...) will 
contain some modules for discovery of various traits on a resource 
provider as well as modules that can detect conflicts in a request for a 
list of traits. Those modules would clearly be something that various 
services would want to import, irrespective of the import of the 
constant trait string catalog.


Thoughts?
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/15/2016 12:56 AM, Yingxin Cheng wrote:

Hi,

I'm concerned with the dependencies between "os-capabilities" library
and all the other OpenStack services such as Nova, Placement, Ironic, etc.

Rather than embedding the universal "os-capabilities" in Nova, Cinder,
Glance, Ironic services that will introduce complexities if the library
versions are different, I'd prefer to hide this library behind the
placement service and expose consistent interfaces as well as caps to
all the other services. But the drawback here is also obvious: for
example, when nova wants to support a new capability, the development
will require os-capabilities updates, and related lib version bumps,
which is inconvenient and seems unnecessary.


It may seem unnecessary and inconvenient, but I honestly don't 
anticipate it being particularly onerous for developers to keep track of.



So IMHO, the possible solution is:
* Let each services (Nova, Ironic ...) themselves manage their
capabilities under proper namespaces such as "compute", "ironic" or
"storage";


-1. What if Ironic wants to utilize a capability that is "compute" -- 
say, an x86 CPU instruction set extension feature? Clearly, we don't 
want a situation where Ironic needs to import Nova in order to get a 
list of these constants?



* Let os-capabilities define as few as caps possible that are only
cross-project;
* And also use os-capabilities to convert service-defined or
user-defined caps to a standardized and distinct form that can be
understood by the placement engine.


User-defined capabilities should be in the placement API only I think. 
There simply isn't anything to do for these in a shared library because, 
well, they aren't shared things. They are deployment-specific and belong 
as just data that is returned from the deployment's particular placement 
API after being added as a custom trait/capability string by an 
administrator.


Best,
-jay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Alex Xu]

2016-08-15 12:56 GMT+08:00 Yingxin Cheng :

> Hi,
>
> I'm concerned with the dependencies between "os-capabilities" library and
> all the other OpenStack services such as Nova, Placement, Ironic, etc.
>
> Rather than embedding the universal "os-capabilities" in Nova, Cinder,
> Glance, Ironic services that will introduce complexities if the library
> versions are different, I'd prefer to hide this library behind the
> placement service and expose consistent interfaces as well as caps to all
> the other services. But the drawback here is also obvious: for example,
> when nova wants to support a new capability, the development will require
> os-capabilities updates, and related lib version bumps, which is
> inconvenient and seems unnecessary.
>

+1 for the concern, this is good point. +1 for the hide os-capability
behind the placement service and expose consistent interfaces.


>
> So IMHO, the possible solution is:
> * Let each services (Nova, Ironic ...) themselves manage their
> capabilities under proper namespaces such as "compute", "ironic" or
> "storage";
>

Actually I think we won't catalog the capabilities by services, we will
catalog the capabilities by resource types. Like nova and ironic may share
some capabilities, due to they provider compute resource.


> * Let os-capabilities define as few as caps possible that are only
> cross-project;
> * And also use os-capabilities to convert service-defined or user-defined
> caps to a standardized and distinct form that can be understood by the
> placement engine.
>
>
This solution sounds complex for me, you need define the capabilities many
different places(in os-capabilities for cross-project caps, in each service
for project-only caps), and this may leads to different service define same
caps but with own naming. And due to the dependence is a library. When you
upgrade the os-capabilities, you have to restart all the services.

I'm thinking of another solution:

1. os-capabilities just define the capabilities into the JSON/YAML files.
2. We expose the capabilities through the REST API of placement engine,
this REST API will read the capabilities from the JSON/YAML files defined
by os-capabilities.

This resolves some problems as below:
1. your concern, hide the os-capabilities behind one single API
2. when os-capabilities upgrade, you needn't update all the os-capabilities
library for all control nodes. You only need update the os-capabilities for
the node running placement engine. Due to the capabilities in the JSON/YAML
files, you even needn't restart the placement engine service.

But yes, this also can be done by glance metadata API.


> My two cents,
> Yingxin
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Yingxin Cheng]

Hi,

I'm concerned with the dependencies between "os-capabilities" library and
all the other OpenStack services such as Nova, Placement, Ironic, etc.

Rather than embedding the universal "os-capabilities" in Nova, Cinder,
Glance, Ironic services that will introduce complexities if the library
versions are different, I'd prefer to hide this library behind the
placement service and expose consistent interfaces as well as caps to all
the other services. But the drawback here is also obvious: for example,
when nova wants to support a new capability, the development will require
os-capabilities updates, and related lib version bumps, which is
inconvenient and seems unnecessary.

So IMHO, the possible solution is:
* Let each services (Nova, Ironic ...) themselves manage their capabilities
under proper namespaces such as "compute", "ironic" or "storage";
* Let os-capabilities define as few as caps possible that are only
cross-project;
* And also use os-capabilities to convert service-defined or user-defined
caps to a standardized and distinct form that can be understood by the
placement engine.


My two cents,
Yingxin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

Word up, Travis :) A few comments inline, but overall I'm looking 
forward to collaborating with you, Steve, and all the other Searchlight 
contributors on os-capabilities (or os-caps as Sean wants to rename it ;)


On 08/11/2016 08:00 PM, Tripp, Travis S wrote:

[Graffit] was originally co-sponsored by Intel
to help expose out all the CPU capabilities in Nova. The constants in
the metadef catalog all come from combing through the code in Nova
was a complete maze and were not available at the time from
Nova (or cinder or glance or …) See overview here [2]:

 [2] https://wiki.openstack.org/wiki/Graffiti


Yep, I'm thoroughly familiar with the maze-ishness of the code in Nova 
and os-capabilities was borne out of an attempt to curate/catalog a 
number of collections of string metadata, constants spread over modules, 
and various hardcoded feature flags/strings in the virt drivers and 
elsewhere.



2) It uses a custom JSON format instead of JSONSchema, so we now need to
figure out the schema for these metadef documents and keep up to date
with that schema as it changes.


It uses JSON schema, but surrounds it with a very lightweight envelope.
The envelope is called a namespace and is simply a container of JSON
schema, allowing us to manage it as a programmatic unit and as a way
for cloud deployers to share the capabilities across clouds very easily.

We did place a limitation on it that it cannot support nested objects. This
was primarily due to the extreme difficulty of representing that construct
to users in an easy to understand way:

http://docs.openstack.org/developer/glance/metadefs-concepts.html#catalog-terminology


I actually do not think there is a need for a JSON schema for any of the 
capability strings in os-caps. I wasn't planning on supporting anything 
more than simple strings with a string prefix for "namespaces" and a 
common delimiter (I chose ':'). I'd like to keep things as simple as 
possible.



3) It mixes qualitative things -- CPU model, features, etc -- with
quantitative things -- amount of cores, threads, etc. These two things
are precisely what we are trying to decouple from each other in the next
generation of Nova's "flavors".


I noticed you didn't respond to this part of my email (from a year ago). 
It's actually a really important point. The mixing of quantitative and 
qualitative things in the Nova flavor extra specs as well as the Glance 
metadefs stuff is a real problem we're trying to fix with the new 
placement API.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Mooney, Sean K]

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Friday, August 12, 2016 2:20 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova] os-capabilities library created
> 
> On 08/12/2016 04:05 AM, Daniel P. Berrange wrote:
> > On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:
> >> Hi Novas and anyone interested in how to represent capabilities in a
> >> consistent fashion.
> >>
> >> I spent an hour creating a new os-capabilities Python library this
> evening:
> >>
> >> http://github.com/jaypipes/os-capabilities
> >>
> >> Please see the README for examples of how the library works and how
> >> I'm thinking of structuring these capability strings and symbols. I
> >> intend os-capabilities to be the place where the OpenStack community
> >> catalogs and collates standardized features for hardware, devices,
> >> networks, storage, hypervisors, etc.
> >>
> >> Let me know what you think about the structure of the library and
> >> whether you would be interested in owning additions to the library
> of
> >> constants in your area of expertise.
> >
> > How are you expecting that these constants are used ? It seems
> > unlikely the, say nova code, code is going to be explicitly accessing
> > any of the individual CPU flag constants.
> 
> These capability strings are what deployers will associate with a
> flavor in Nova and they will be passed in the request to the placement
> API in either a "requirements" or a "preferences" list. In order to
> ensure that two OpenStack clouds refer to various capabilities (not
> just CPU flags, see below), we need a curated list of these
> standardized constants.
> 
>  > It should surely just be entirely metatadata
> > driven - eg libvirt driver would just parse libvirt capabilities XML
> > and extract all the CPU flag strings & simply export them.
> 
> You are just thinking in terms of (lib)virt/compute capabilities.
> os-capabilities intends to provide a standard set of capability
> constants for more than virt/compute, including storage, network
> devices and more.
> 
> But, yes, I imagine discovery code running on a compute node with the
> *libvirt* virt driver could indeed simply query the libvirt
> capabilities XML snippet and translate those capability codes into os-
> capabilities constants. Remember, VMWare and Hyper-V also need to do
> this discovery and translation to a standardized set of constants. So
> does ironic-inspector when it queries an IPMI interface of course.
> 
>  > It would be very
> > undesirable to have to add new code to os-capabilities every time
> that
> > Intel/AMD create new CPU flags for new features, and force users to
> > upgrade openstack to be able to express requirements on those CPU
> flags.
> 
> I don't see how we would be able to expose a particular new CPU flag
> *across disparate OpenStack clouds* unless we have some standardized
> set of constants that has been curated. Not all OpenStack clouds run
> libvirt. And again, think bigger than just virt/compute.
[Mooney, Sean K] just as an aside I think Libvirt actually gets its capability
Information from udev. Again that wont help you on windows but it's at least not
Requiring Libvirt.  os-capabilities could retrieve info via udev also 
potentially.

Ipmi will allow you to discover some capabilities of the system but
It might be worth considering redfish fit for capabilities discovery
http://www.dmtf.org/standards/redfish
https://www.brighttalk.com/webcast/9077/163783

On a personal note could we call os-capabilities, os-caps?
Its shorter and  I have misspelled capabilities 4 different ways in typing this 
Response which I have now fixed it.
> 
> Best,
> -jay
> 
> >> Next steps for the library include:
> >>
> >> * Bringing in other top-level namespaces like disk: or net: and
> >> working with contributors to fill in the capability strings and
> symbols.
> >> * Adding constraints functionality to the library. For instance,
> >> building in information to the os-capabilities interface that would
> >> allow a set of capabilities to be cross-checked for set violations.
> >> As an example, a resource provider having DISK_GB inventory cannot
> >> have *both* the disk:ssd
> >> *and* the disk:hdd capability strings associated with it -- clearly
> >> the disk storage is either SSD or spinning disk.
> >
> > Regards,
> > Daniel
> >
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/12/2016 04:05 AM, Daniel P. Berrange wrote:

On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:

Hi Novas and anyone interested in how to represent capabilities in a
consistent fashion.

I spent an hour creating a new os-capabilities Python library this evening:

http://github.com/jaypipes/os-capabilities

Please see the README for examples of how the library works and how I'm
thinking of structuring these capability strings and symbols. I intend
os-capabilities to be the place where the OpenStack community catalogs and
collates standardized features for hardware, devices, networks, storage,
hypervisors, etc.

Let me know what you think about the structure of the library and whether
you would be interested in owning additions to the library of constants in
your area of expertise.


How are you expecting that these constants are used ? It seems unlikely
the, say nova code, code is going to be explicitly accessing any of the
individual CPU flag constants.


These capability strings are what deployers will associate with a flavor 
in Nova and they will be passed in the request to the placement API in 
either a "requirements" or a "preferences" list. In order to ensure that 
two OpenStack clouds refer to various capabilities (not just CPU flags, 
see below), we need a curated list of these standardized constants.


> It should surely just be entirely metatadata

driven - eg libvirt driver would just parse libvirt capabilities XML and
extract all the CPU flag strings & simply export them.


You are just thinking in terms of (lib)virt/compute capabilities. 
os-capabilities intends to provide a standard set of capability 
constants for more than virt/compute, including storage, network devices 
and more.


But, yes, I imagine discovery code running on a compute node with the 
*libvirt* virt driver could indeed simply query the libvirt capabilities 
XML snippet and translate those capability codes into os-capabilities 
constants. Remember, VMWare and Hyper-V also need to do this discovery 
and translation to a standardized set of constants. So does 
ironic-inspector when it queries an IPMI interface of course.


> It would be very

undesirable to have to add new code to os-capabilities every time that
Intel/AMD create new CPU flags for new features, and force users to upgrade
openstack to be able to express requirements on those CPU flags.


I don't see how we would be able to expose a particular new CPU flag 
*across disparate OpenStack clouds* unless we have some standardized set 
of constants that has been curated. Not all OpenStack clouds run 
libvirt. And again, think bigger than just virt/compute.


Best,
-jay


Next steps for the library include:

* Bringing in other top-level namespaces like disk: or net: and working with
contributors to fill in the capability strings and symbols.
* Adding constraints functionality to the library. For instance, building in
information to the os-capabilities interface that would allow a set of
capabilities to be cross-checked for set violations. As an example, a
resource provider having DISK_GB inventory cannot have *both* the disk:ssd
*and* the disk:hdd capability strings associated with it -- clearly the disk
storage is either SSD or spinning disk.


Regards,
Daniel



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/12/2016 07:49 AM, Jim Rollenhagen wrote:

On Thu, Aug 11, 2016 at 9:03 PM, Jay Pipes  wrote:

On 08/11/2016 05:46 PM, Clay Gerrard wrote:


On Thu, Aug 11, 2016 at 2:25 PM, Ed Leafe > wrote:


Overall this looks good, although it seems a bit odd to have
ALL_CAPS_STRINGS to represent all:caps:strings throughout. The
example you gave:

>>> print os_caps.HW_CPU_X86_SSE42
hw:cpu:x86:sse42


Just to be clear, this project doesn't *do* anything right?  Like it
won't parse `/proc/cpuinfo` and actually figure out a machines cpu flags
that can then be broadcast as "capabilities"?

Like, TBH I think it took me longer than I would prefer to honestly
admit to find out about /sys/block//queue/rotational [1]

So if there was a library about standardizing how hardware capabilities
are discovered and reported - that maybe seems like a sane sort of thing
for a collection of related projects to agree on.  But I'm not sure if
this does that?



Hi Clay!

It does not currently do that, but I'm interested in adding this capability
(pun intended).


ironic-python-agent does some of this discovery. It isn't
comprehensive, but it's a good starting point if we want to
lift some of that code out. The classes are here, and the
discovery things are in the same file if you grep around. :)

https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L186-L251


Rock on. Duly noted :)

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jim Rollenhagen]

On Thu, Aug 11, 2016 at 9:03 PM, Jay Pipes  wrote:
> On 08/11/2016 05:46 PM, Clay Gerrard wrote:
>>
>> On Thu, Aug 11, 2016 at 2:25 PM, Ed Leafe > > wrote:
>>
>>
>> Overall this looks good, although it seems a bit odd to have
>> ALL_CAPS_STRINGS to represent all:caps:strings throughout. The
>> example you gave:
>>
>> >>> print os_caps.HW_CPU_X86_SSE42
>> hw:cpu:x86:sse42
>>
>>
>> Just to be clear, this project doesn't *do* anything right?  Like it
>> won't parse `/proc/cpuinfo` and actually figure out a machines cpu flags
>> that can then be broadcast as "capabilities"?
>>
>> Like, TBH I think it took me longer than I would prefer to honestly
>> admit to find out about /sys/block//queue/rotational [1]
>>
>> So if there was a library about standardizing how hardware capabilities
>> are discovered and reported - that maybe seems like a sane sort of thing
>> for a collection of related projects to agree on.  But I'm not sure if
>> this does that?
>
>
> Hi Clay!
>
> It does not currently do that, but I'm interested in adding this capability
> (pun intended).

ironic-python-agent does some of this discovery. It isn't
comprehensive, but it's a good starting point if we want to
lift some of that code out. The classes are here, and the
discovery things are in the same file if you grep around. :)

https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L186-L251

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Daniel P. Berrange]

On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:
> Hi Novas and anyone interested in how to represent capabilities in a
> consistent fashion.
> 
> I spent an hour creating a new os-capabilities Python library this evening:
> 
> http://github.com/jaypipes/os-capabilities
> 
> Please see the README for examples of how the library works and how I'm
> thinking of structuring these capability strings and symbols. I intend
> os-capabilities to be the place where the OpenStack community catalogs and
> collates standardized features for hardware, devices, networks, storage,
> hypervisors, etc.
> 
> Let me know what you think about the structure of the library and whether
> you would be interested in owning additions to the library of constants in
> your area of expertise.

How are you expecting that these constants are used ? It seems unlikely
the, say nova code, code is going to be explicitly accessing any of the
individual CPU flag constants. It should surely just be entirely metatadata
driven - eg libvirt driver would just parse libvirt capabilities XML and
extract all the CPU flag strings & simply export them. It would be very
undesirable to have to add new code to os-capabilities every time that
Intel/AMD create new CPU flags for new features, and force users to upgrade
openstack to be able to express requirements on those CPU flags.

> Next steps for the library include:
> 
> * Bringing in other top-level namespaces like disk: or net: and working with
> contributors to fill in the capability strings and symbols.
> * Adding constraints functionality to the library. For instance, building in
> information to the os-capabilities interface that would allow a set of
> capabilities to be cross-checked for set violations. As an example, a
> resource provider having DISK_GB inventory cannot have *both* the disk:ssd
> *and* the disk:hdd capability strings associated with it -- clearly the disk
> storage is either SSD or spinning disk.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/11/2016 05:50 PM, John Dickinson wrote:

On 3 Aug 2016, at 16:47, Jay Pipes wrote:

Hi Novas and anyone interested in how to represent capabilities in
a consistent fashion.

I spent an hour creating a new os-capabilities Python library this
evening:

http://github.com/jaypipes/os-capabilities

Please see the README for examples of how the library works and how
I'm thinking of structuring these capability strings and symbols. I
intend os-capabilities to be the place where the OpenStack
community catalogs and collates standardized features for hardware,
devices, networks, storage, hypervisors, etc.

Let me know what you think about the structure of the library and
whether you would be interested in owning additions to the library
of constants in your area of expertise.

Next steps for the library include:

* Bringing in other top-level namespaces like disk: or net: and
working with contributors to fill in the capability strings and
symbols. * Adding constraints functionality to the library. For
instance, building in information to the os-capabilities interface
that would allow a set of capabilities to be cross-checked for set
violations. As an example, a resource provider having DISK_GB
inventory cannot have *both* the disk:ssd *and* the disk:hdd
capability strings associated with it -- clearly the disk storage
is either SSD or spinning disk.

Anyway, lemme know your initial thoughts please.


Is this intended to be a cross-project thing? The message is tagged
"[nova]", so I'm kinda surprised I saw it,


Sorry about that, John. I've been focused on Nova usage at first, but 
for sure I'd like the library to contain cross-project, standardized 
string constants for various compute, storage, hardware, network, 
device, etc capabilities. It's literally just an hour of work and 
totally just spitballing ideas right now, so please don't see it as 
intentionally trying to exclude anyone.


> but the library seems to

be called openstack capabilities. So if this is going to be a big
thing for everyone, please update the ML subject tag (and help me
understand how it applies to more than just nova). And if it's just
for nova (err... "compute"), then naming it something that doesn't
imply every project will need to use it could help prevent future
misunderstanding.


As Ed mentioned, this library comes from use cases in the broken-out 
scheduler component (the placement API/engine) that we've been working 
on in Nova this release. Once the scheduler is broken out and being used 
by Nova for quantitative request matching, we will move on to supporting 
the qualitative side of the request -- and that's where the capabilities 
library comes into play. We are aiming to get this part of the placement 
API done in Ocata and I was just getting a head start.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/11/2016 05:25 PM, Ed Leafe wrote:

On Aug 3, 2016, at 6:47 PM, Jay Pipes  wrote:


Please see the README for examples of how the library works and how I'm 
thinking of structuring these capability strings and symbols. I intend 
os-capabilities to be the place where the OpenStack community catalogs and 
collates standardized features for hardware, devices, networks, storage, 
hypervisors, etc.


Overall this looks good, although it seems a bit odd to have ALL_CAPS_STRINGS 
to represent all:caps:strings throughout.


ALL_CAPS is generally used for constant symbols in many (most?) 
programming languages, AFAIK.



The example you gave:


print os_caps.HW_CPU_X86_SSE42

hw:cpu:x86:sse42

begs the question: is the rule going to be that capability for any constant 
will always be: CONST.lower().replace(“_”, “:”) ? If so, I’m not sure I see how 
the capabilities are not themselves constants.


Because you can make a typo with a string and it can go unnoticed. A 
typo in a constant means the code won't compile/import because of an 
unknown symbol.


That's the reason for having these strings represented as constant 
symbols in a shared library.



The namespacing is ugly, but I guess it’s necessary given that names are not 
simple tags, but nested data structures in string form.


Yeah, it's ugly-ish. I'm certainly open to suggestions for how to better 
nest/represent these groups of capability constants.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/11/2016 05:46 PM, Clay Gerrard wrote:

On Thu, Aug 11, 2016 at 2:25 PM, Ed Leafe > wrote:


Overall this looks good, although it seems a bit odd to have
ALL_CAPS_STRINGS to represent all:caps:strings throughout. The
example you gave:

>>> print os_caps.HW_CPU_X86_SSE42
hw:cpu:x86:sse42


Just to be clear, this project doesn't *do* anything right?  Like it
won't parse `/proc/cpuinfo` and actually figure out a machines cpu flags
that can then be broadcast as "capabilities"?

Like, TBH I think it took me longer than I would prefer to honestly
admit to find out about /sys/block//queue/rotational [1]

So if there was a library about standardizing how hardware capabilities
are discovered and reported - that maybe seems like a sane sort of thing
for a collection of related projects to agree on.  But I'm not sure if
this does that?


Hi Clay!

It does not currently do that, but I'm interested in adding this 
capability (pun intended).


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Tripp, Travis S]

Excerpts from Jay Pipes's message of 2016-08-03 19:47:37 -0400:
>>  Hi Novas and anyone interested in how to represent capabilities in a 
>>  consistent fashion.
>>
>>  I spent an hour creating a new os-capabilities Python library this 
evening:
> 
>>  http://github.com/jaypipes/os-capabilities
>> Anyway, lemme know your initial thoughts please.

   On 8/11/16, 3:52 PM, "Clint Byrum"  wrote:

> Did we ever resolve the similarities between this and Searchlight's
> similar goals of providing consistent metadata to the users?

> http://docs.openstack.org/developer/glance/metadefs-concepts.html

> I understand your library is for operators and developers to
> collaborate, but it seems like there should be some alignment with this
> UI that wants to do the same thing for the end user where appropriate.

The metadefs catalog wasn’t written just as a UI construct, it was actually
a derivative of an effort called graffiti [0] that was entirely about
capability and requirement matching and providing a way for portability
across clouds.

The Graffiti project was proposed at the Atlanta (Juno) OpenStack summit.
Since then quite a bit of the concepts have been adopted and are covered as
part of multiple different OpenStack Projects. 

[0] https://wiki.openstack.org/wiki/Graffiti/Architecture

A key concept is to both support defining standard metadata that can
be attached to various resources as well as providing a common service
for deployers to register their own metadata with visibility restrictions.
This can be anything from “Gold, Silver, Bronze” to “some hardware
capability”. Ultimately it is up to the deployer to activate, publish,
or discover the capabilities in their environment and enable them in the 
catalog.

Glance Metadata Definition Catalog (Box 1 in the workflow diagram)
 * http://docs.openstack.org/developer/glance/metadefs-concepts.html
* https://github.com/openstack/glance/tree/master/etc/metadefs
* https://youtu.be/zJpHXdBOoeM

Horizon features (Box 2 in the diagram – but also CLI)
* An admin UI for managing the catalog
* (Admin —> Metadata Definitions) (Kilo)
* A widget for associating metadata to different resources
* (Update Metadata action on each row item below)
* admin -> images (Juno)
* admin -> flavors (Kilo)
* admin —> Host Aggregates (Kilo)
* project —> images (Liberty)
* project —> instances (Mitaka)
* The ability to add metadata at launch time
* project —> Launch Instance (ng launch instance enabled) (Mitaka)

Searchlight (Box 3 in the workflow diagram)
* http://launchpad.net/searchlight
* https://wiki.openstack.org/wiki/Searchlight

The Searchlight project is primarily intended for high performance
searching across the cloud. Fulfilling the concepts of Graffiti is a side
effect, but did provide some of the original inspiration for the project.
We actually have blueprint we will do in “O” that will provide
data mapping from metadefs to the schemas in ElasticSearch [1].

* [1] 
https://blueprints.launchpad.net/searchlight/+spec/configurable-dynamic-properties

In addition, when this popped up in my mailbox today, a search revealed
This message from last August with a few points that I’d like to help clarify 
below:

On 8/10/15, 8:05 AM, "Jay Pipes"  wrote:

>  The Glance metadefs stuff is problematic in a number of ways:

> 1) It wasn't written with Nova in mind at all, but rather for UI needs.
> This means it introduces a bunch of constants that are different from
> the constants in Nova.

This is actually not the case. This was originally co-sponsored by Intel
to help expose out all the CPU capabilities in Nova. The constants in
the metadef catalog all come from combing through the code in Nova
was a complete maze and were not available at the time from
Nova (or cinder or glance or …) See overview here [2]:

 [2] https://wiki.openstack.org/wiki/Graffiti

> 2) It uses a custom JSON format instead of JSONSchema, so we now need to
> figure out the schema for these metadef documents and keep up to date
> with that schema as it changes.

It uses JSON schema, but surrounds it with a very lightweight envelope.
The envelope is called a namespace and is simply a container of JSON
schema, allowing us to manage it as a programmatic unit and as a way
for cloud deployers to share the capabilities across clouds very easily.

We did place a limitation on it that it cannot support nested objects. This
was primarily due to the extreme difficulty of representing that construct
to users in an easy to understand way:

http://docs.openstack.org/developer/glance/metadefs-concepts.html#catalog-terminology

> 3) It mixes qualitative things -- CPU model, features, etc -- with
> quantitative things -- amount of cores, threads, etc. These two things
> are precisely what we are trying to decouple from each other in the next
> generation of Nova's "flavors".
>
> 4) It is still missing the user-facing 

Re: [openstack-dev] [nova] os-capabilities library created

[Ed Leafe]

On Aug 11, 2016, at 5:29 PM, John Dickinson  wrote:

>> I will let Jay speak for himself (as if I could somehow prevent that!), but 
>> the intent here is that this won’t be for Nova specifically; it is targeted 
>> primarily for the forthcoming placement service (you might know it as the 
>> scheduler). The goal is to have a standard way of representing *qualitative* 
>> aspects of resources. So while we are not actively trying to make this a 
>> placement engine for all OpenStack services yet, the goal is to not be 
>> Nova-specific, so that once we have this up and running, we can offer it as 
>> a general placement service for any other project that has such needs.
> 
> Sounds great! How can I get involved with the new general purpose placement 
> scheduler engine and share Swift's requirements?

The main work being done now is on moving Nova from hard-coded CPU, disk, etc., 
to use Resource Provider classes instead. You can see the current work at 
https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/generic-resource-pools

You can also join the scheduler subteam meeting Mondays at 1400 UTC in 
@openstack-meeting-alt, where we discuss our current efforts. If you have a 
specific question or topic to bring up, add it to the meeting agenda at 
https://wiki.openstack.org/wiki/Meetings/NovaScheduler.


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[John Dickinson]

On 11 Aug 2016, at 15:14, Ed Leafe wrote:

> On Aug 11, 2016, at 4:50 PM, John Dickinson  wrote:
>
>> Is this intended to be a cross-project thing? The message is tagged 
>> "[nova]", so I'm kinda surprised I saw it, but the library seems to be 
>> called openstack capabilities. So if this is going to be a big thing for 
>> everyone, please update the ML subject tag (and help me understand how it 
>> applies to more than just nova). And if it's just for nova (err... 
>> "compute"), then naming it something that doesn't imply every project will 
>> need to use it could help prevent future misunderstanding.
>
> I will let Jay speak for himself (as if I could somehow prevent that!), but 
> the intent here is that this won’t be for Nova specifically; it is targeted 
> primarily for the forthcoming placement service (you might know it as the 
> scheduler). The goal is to have a standard way of representing *qualitative* 
> aspects of resources. So while we are not actively trying to make this a 
> placement engine for all OpenStack services yet, the goal is to not be 
> Nova-specific, so that once we have this up and running, we can offer it as a 
> general placement service for any other project that has such needs.


Sounds great! How can I get involved with the new general purpose placement 
scheduler engine and share Swift's requirements?

--john



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Ed Leafe]

On Aug 11, 2016, at 4:50 PM, John Dickinson  wrote:

> Is this intended to be a cross-project thing? The message is tagged "[nova]", 
> so I'm kinda surprised I saw it, but the library seems to be called openstack 
> capabilities. So if this is going to be a big thing for everyone, please 
> update the ML subject tag (and help me understand how it applies to more than 
> just nova). And if it's just for nova (err... "compute"), then naming it 
> something that doesn't imply every project will need to use it could help 
> prevent future misunderstanding.

I will let Jay speak for himself (as if I could somehow prevent that!), but the 
intent here is that this won’t be for Nova specifically; it is targeted 
primarily for the forthcoming placement service (you might know it as the 
scheduler). The goal is to have a standard way of representing *qualitative* 
aspects of resources. So while we are not actively trying to make this a 
placement engine for all OpenStack services yet, the goal is to not be 
Nova-specific, so that once we have this up and running, we can offer it as a 
general placement service for any other project that has such needs.


-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Clint Byrum]

Excerpts from Jay Pipes's message of 2016-08-03 19:47:37 -0400:
> Hi Novas and anyone interested in how to represent capabilities in a 
> consistent fashion.
> 
> I spent an hour creating a new os-capabilities Python library this evening:
> 
> http://github.com/jaypipes/os-capabilities
> 
> Please see the README for examples of how the library works and how I'm 
> thinking of structuring these capability strings and symbols. I intend 
> os-capabilities to be the place where the OpenStack community catalogs 
> and collates standardized features for hardware, devices, networks, 
> storage, hypervisors, etc.
> 
> Let me know what you think about the structure of the library and 
> whether you would be interested in owning additions to the library of 
> constants in your area of expertise.
> 
> Next steps for the library include:
> 
> * Bringing in other top-level namespaces like disk: or net: and working 
> with contributors to fill in the capability strings and symbols.
> * Adding constraints functionality to the library. For instance, 
> building in information to the os-capabilities interface that would 
> allow a set of capabilities to be cross-checked for set violations. As 
> an example, a resource provider having DISK_GB inventory cannot have 
> *both* the disk:ssd *and* the disk:hdd capability strings associated 
> with it -- clearly the disk storage is either SSD or spinning disk.
> 
> Anyway, lemme know your initial thoughts please.

Did we ever resolve the similarities between this and Searchlight's
similar goals of providing consistent metadata to the users?

http://docs.openstack.org/developer/glance/metadefs-concepts.html

I understand your library is for operators and developers to
collaborate, but it seems like there should be some alignment with this
UI that wants to do the same thing for the end user where appropriate.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[John Dickinson]

On 3 Aug 2016, at 16:47, Jay Pipes wrote:

> Hi Novas and anyone interested in how to represent capabilities in a 
> consistent fashion.
>
> I spent an hour creating a new os-capabilities Python library this evening:
>
> http://github.com/jaypipes/os-capabilities
>
> Please see the README for examples of how the library works and how I'm 
> thinking of structuring these capability strings and symbols. I intend 
> os-capabilities to be the place where the OpenStack community catalogs and 
> collates standardized features for hardware, devices, networks, storage, 
> hypervisors, etc.
>
> Let me know what you think about the structure of the library and whether you 
> would be interested in owning additions to the library of constants in your 
> area of expertise.
>
> Next steps for the library include:
>
> * Bringing in other top-level namespaces like disk: or net: and working with 
> contributors to fill in the capability strings and symbols.
> * Adding constraints functionality to the library. For instance, building in 
> information to the os-capabilities interface that would allow a set of 
> capabilities to be cross-checked for set violations. As an example, a 
> resource provider having DISK_GB inventory cannot have *both* the disk:ssd 
> *and* the disk:hdd capability strings associated with it -- clearly the disk 
> storage is either SSD or spinning disk.
>
> Anyway, lemme know your initial thoughts please.

Is this intended to be a cross-project thing? The message is tagged "[nova]", 
so I'm kinda surprised I saw it, but the library seems to be called openstack 
capabilities. So if this is going to be a big thing for everyone, please update 
the ML subject tag (and help me understand how it applies to more than just 
nova). And if it's just for nova (err... "compute"), then naming it something 
that doesn't imply every project will need to use it could help prevent future 
misunderstanding.

--john


>
> Best,
> -jay
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Clay Gerrard]

On Thu, Aug 11, 2016 at 2:25 PM, Ed Leafe  wrote:

>
> Overall this looks good, although it seems a bit odd to have
> ALL_CAPS_STRINGS to represent all:caps:strings throughout. The example you
> gave:
>
> >>> print os_caps.HW_CPU_X86_SSE42
> hw:cpu:x86:sse42
>
>
Just to be clear, this project doesn't *do* anything right?  Like it won't
parse `/proc/cpuinfo` and actually figure out a machines cpu flags that can
then be broadcast as "capabilities"?

Like, TBH I think it took me longer than I would prefer to honestly admit
to find out about /sys/block//queue/rotational [1]

So if there was a library about standardizing how hardware capabilities are
discovered and reported - that maybe seems like a sane sort of thing for a
collection of related projects to agree on.  But I'm not sure if this does
that?

-Clay

1. https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Ed Leafe]

On Aug 3, 2016, at 6:47 PM, Jay Pipes  wrote:

> Please see the README for examples of how the library works and how I'm 
> thinking of structuring these capability strings and symbols. I intend 
> os-capabilities to be the place where the OpenStack community catalogs and 
> collates standardized features for hardware, devices, networks, storage, 
> hypervisors, etc.

Overall this looks good, although it seems a bit odd to have ALL_CAPS_STRINGS 
to represent all:caps:strings throughout. The example you gave:

>>> print os_caps.HW_CPU_X86_SSE42
hw:cpu:x86:sse42

begs the question: is the rule going to be that capability for any constant 
will always be: CONST.lower().replace(“_”, “:”) ? If so, I’m not sure I see how 
the capabilities are not themselves constants.

The namespacing is ugly, but I guess it’s necessary given that names are not 
simple tags, but nested data structures in string form.

-- Ed Leafe






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Chris Dent]

On Wed, 3 Aug 2016, Jay Pipes wrote:

Let me know what you think about the structure of the library and whether you 
would be interested in owning additions to the library of constants in your 
area of expertise.


Seems mostly sane to me. I don't really care for the inspection of
sys.modules (or at least that it is inspecting the current module
rather than the one[s] where the constants actually live) but that's
a nit.

The main suggestion I would have is that you put the raw constants
(including the hierarchy) in something like YAML so that they are easy
to use without requiring Python (although most people will probably use
them that way).

It also seems a bit strange to me to have additional constant symbols to
represent data we are intending to be literally constant symbols.
But computers like that sort of stuff, so maybe that's just the way
it goes.

And then there's that naming thing. As discussed elsewhere we're
overloading "capabilities" throughout OpenStack.

* Adding constraints functionality to the library. For instance, building in 
information to the os-capabilities interface that would allow a set of 
capabilities to be cross-checked for set violations. As an example, a 
resource provider having DISK_GB inventory cannot have *both* the disk:ssd 
*and* the disk:hdd capability strings associated with it -- clearly the disk 
storage is either SSD or spinning disk.


That's going to be fun™.

Did you already have some ideas on how to do that? Annotate namespaces
to describe constraints on their contents? Annotate individual entries
to describe their friends or enemies?

--
Chris Dent   ┬─┬ノ( º _ ºノ) http://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jim Rollenhagen]

>> On Aug 4, 2016, at 10:48, Jay Pipes  wrote:
>> 
>>> On 08/04/2016 10:31 AM, Jim Rollenhagen wrote:
>>> On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:
>>> Hi Novas and anyone interested in how to represent capabilities in a
>>> consistent fashion.
>>> 
>>> I spent an hour creating a new os-capabilities Python library this evening:
>>> 
>>> http://github.com/jaypipes/os-capabilities
>>> 
>>> Please see the README for examples of how the library works and how I'm
>>> thinking of structuring these capability strings and symbols. I intend
>>> os-capabilities to be the place where the OpenStack community catalogs and
>>> collates standardized features for hardware, devices, networks, storage,
>>> hypervisors, etc.
>>> 
>>> Let me know what you think about the structure of the library and whether
>>> you would be interested in owning additions to the library of constants in
>>> your area of expertise.
>>> 
>>> Next steps for the library include:
>>> 
>>> * Bringing in other top-level namespaces like disk: or net: and working with
>>> contributors to fill in the capability strings and symbols.
>>> * Adding constraints functionality to the library. For instance, building in
>>> information to the os-capabilities interface that would allow a set of
>>> capabilities to be cross-checked for set violations. As an example, a
>>> resource provider having DISK_GB inventory cannot have *both* the disk:ssd
>>> *and* the disk:hdd capability strings associated with it -- clearly the disk
>>> storage is either SSD or spinning disk.
>> 
>> Well, if we constrain ourselves to VMs, yes. :)
> 
> I wasn't constraining ourselves to VMs :)
> 
>> One of the issues with running ironic behind nova is that there isn't
>> any way to express that a flavor (or instance) has (or should have)
>> multiple physical disks. It would certainly be possible to boot a
>> baremetal machine that does have SSD and spinning rust.
>> 
>> I don't have a solution in mind here, just wanted to point out that we
>> need to keep more than VMs in mind when talking about capabilities. :)
> 
> Note that in the above, I am explicit that the disk:hdd and disk:ssd 
> capabilities should not be provided by a resource provider **that has an 
> inventory of DISK_GB resources** :)
> 
> Ironic baremetal nodes do not have an inventory record of DISK_GB. Instead, 
> the resource class is dynamic -- e.g. IRON_SILVER. The constraint of not 
> having disk:hdd and disk:ssd wouldn't apply in that case.

Touché. I would like to be able to express that some baremetal resource class 
can have disk:ssd and disk:hdd capabilities, but it sounds like that's covered. 
Thanks for clearing that up for me. :)

// jim

> 
> Best,
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jay Pipes]

On 08/04/2016 10:31 AM, Jim Rollenhagen wrote:

On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:

Hi Novas and anyone interested in how to represent capabilities in a
consistent fashion.

I spent an hour creating a new os-capabilities Python library this evening:

http://github.com/jaypipes/os-capabilities

Please see the README for examples of how the library works and how I'm
thinking of structuring these capability strings and symbols. I intend
os-capabilities to be the place where the OpenStack community catalogs and
collates standardized features for hardware, devices, networks, storage,
hypervisors, etc.

Let me know what you think about the structure of the library and whether
you would be interested in owning additions to the library of constants in
your area of expertise.

Next steps for the library include:

* Bringing in other top-level namespaces like disk: or net: and working with
contributors to fill in the capability strings and symbols.
* Adding constraints functionality to the library. For instance, building in
information to the os-capabilities interface that would allow a set of
capabilities to be cross-checked for set violations. As an example, a
resource provider having DISK_GB inventory cannot have *both* the disk:ssd
*and* the disk:hdd capability strings associated with it -- clearly the disk
storage is either SSD or spinning disk.


Well, if we constrain ourselves to VMs, yes. :)


I wasn't constraining ourselves to VMs :)


One of the issues with running ironic behind nova is that there isn't
any way to express that a flavor (or instance) has (or should have)
multiple physical disks. It would certainly be possible to boot a
baremetal machine that does have SSD and spinning rust.

I don't have a solution in mind here, just wanted to point out that we
need to keep more than VMs in mind when talking about capabilities. :)


Note that in the above, I am explicit that the disk:hdd and disk:ssd 
capabilities should not be provided by a resource provider **that has an 
inventory of DISK_GB resources** :)


Ironic baremetal nodes do not have an inventory record of DISK_GB. 
Instead, the resource class is dynamic -- e.g. IRON_SILVER. The 
constraint of not having disk:hdd and disk:ssd wouldn't apply in that case.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] os-capabilities library created

[Jim Rollenhagen]

On Wed, Aug 03, 2016 at 07:47:37PM -0400, Jay Pipes wrote:
> Hi Novas and anyone interested in how to represent capabilities in a
> consistent fashion.
> 
> I spent an hour creating a new os-capabilities Python library this evening:
> 
> http://github.com/jaypipes/os-capabilities
> 
> Please see the README for examples of how the library works and how I'm
> thinking of structuring these capability strings and symbols. I intend
> os-capabilities to be the place where the OpenStack community catalogs and
> collates standardized features for hardware, devices, networks, storage,
> hypervisors, etc.
> 
> Let me know what you think about the structure of the library and whether
> you would be interested in owning additions to the library of constants in
> your area of expertise.
> 
> Next steps for the library include:
> 
> * Bringing in other top-level namespaces like disk: or net: and working with
> contributors to fill in the capability strings and symbols.
> * Adding constraints functionality to the library. For instance, building in
> information to the os-capabilities interface that would allow a set of
> capabilities to be cross-checked for set violations. As an example, a
> resource provider having DISK_GB inventory cannot have *both* the disk:ssd
> *and* the disk:hdd capability strings associated with it -- clearly the disk
> storage is either SSD or spinning disk.

Well, if we constrain ourselves to VMs, yes. :)

One of the issues with running ironic behind nova is that there isn't
any way to express that a flavor (or instance) has (or should have)
multiple physical disks. It would certainly be possible to boot a
baremetal machine that does have SSD and spinning rust.

I don't have a solution in mind here, just wanted to point out that we
need to keep more than VMs in mind when talking about capabilities. :)

// jim


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev